DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 3 and 5-16 are pending.
Claims 2 and 4 are canceled.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1, 3, 5 and 10-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Huang et al (US2022/0188554) in view of Adhikari et al (Iterative Bounding Box, 2020).
Regarding claims 1 and 15-16, Huang teaches a computer implemented method for validating annotations of objects, the method comprising:
receiving a plurality of spatial datapoints acquired by a sensor, wherein the spatial datapoints are related to an external environment of the sensor,
(Huang, Figs. 1-3, “the LiDAR sensors 208 and 3D data pipeline 214 (e.g., LiDAR pipeline 214) may capture and process, for example, one or more 3D point clouds, 3D BEV representations, depth maps, voxelizations, or other 3D models of the environment 104 surrounding the vehicle 102 and/or the agents 108 (e.g., vehicles 102, pedestrians, bicyclists, wildlife, vegetation, or any of various other moving and/or stationary objects) that the vehicle 102 may encounter”, [0024];
receiving annotation data of objects associated with the acquired spatial datapoints, the annotation data including an identification of each respective object, and
(Huang, Fig. 3, “the neural network 320 may extract feature vectors ... in the 3D point cloud 314 and provide the feature vectors to a 3D object-detector 326. ..., the 3D object detector 326 may include, for example, an ML model or neural network”, [0038]; “based on the 3D feature vectors received from the neural network 320 and the 2D feature vectors received from the neural network 318, the 3D object detector 326 may the generate an estimation (e.g., a prediction) of a 3D bounding box (e.g., cuboid) within the 3D point cloud 314 (or 3D BEV representation), indicating, for example, a detection and classification of the corresponding one or more agents 108 in 3D space”, [0039]; “an input of the point cloud class label 316 (e.g., 3D ground truth annotation)”, [0040]; estimated bounding box from 3D detector 326 for 3D point cloud 314 is given a label 316 for further inspection; “the neural network 320 may extract feature vectors encoding features of the corresponding one or more agents 108 (e.g., vehicles 102, pedestrians, bicyclists, wildlife, vegetation, or any of various other moving and/or stationary objects) in the 3D point cloud 314”, [0054]; the annotation system 304 may create different bounding boxes of different types of 3D objects for labeling)
validating, via a processing unit, the annotations of the objects by performing the steps of:
determining a target range for at least one property of the objects,
determining, from the acquired spatial datapoints and/or from the annotation data, a respective value of the at least one property for each respective object, and
for each object, identifying the object as an erroneous object if the respective value of the at least one property is outside the target range for the at least one property, the erroneous object being selected for review regarding erroneous annotation;
(Huang, Fig. 3, “Based on the input of the estimated 3D bounding box (e.g., cuboid) and an input of the point cloud class label 316 (e.g., 3D ground truth annotation), the 3D loss module 328 may then compare (e.g., position-wise, orientation-wise, size-wise, and so forth) the estimated 3D bounding box (e.g., cuboid) generated by the 3D object detector 326 to the point cloud class label 316 (e.g., 3D ground truth annotation) associated with the 3D point cloud 314. In certain embodiments, the 3D loss module 326 may generate, for example, a regression loss (e.g., MSE loss, MAE loss) as the result of the comparison between the estimated 3D bounding box (e.g., cuboid) and an input of the point cloud class label 316 (e.g., 3D ground truth annotation)”, [0040]; obviously, there should be a range for the regression loss for determining if the estimated 3D bounding box from 3D detector 326 belongs to the assigned label 316; when the regression loss is beyond a threshold of the loss range (or a similarity measure is greater than another threshold), it indicates a mismatch between the estimated 3D bounding box from 3D detector 326 and the assigned label 31, i.e., failing the annotation validation; the next action is to update/correct the model (NN 320 and 3D detector 326) (“the 3D regression loss may be then utilized in backpropagation to update parameters of the 3D object detector 326, the neural network 318, and the neural network 320”, [0040]) for a new 3D bounding box estimation from 3D detector 326 and to compare it with the label 316 again; the correction process for the object annotation, may also be performed by a human annotator as suggested by Adhikari, Fig. 1, after creating bounding boxes, the human annotator can correct the proposed bounding box to fit better to the object label, “The proposed annotations are inspected and manually corrected by a human annotator”, p3, c1; the method of Adhikari may be applied to correct 3D bounding box estimations)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Adhikari into the system or method of Huang in order for human annotators to focus on refining and correcting ML-generated 3D bounding boxes for those failed the validation or inspection. The time-consuming work of annotation validation and inspection is carried out automatically. Doing so would reduce human workload and speed up the object labeling process. The combination of Huang and Adhikari also teaches other enhanced capabilities.
wherein the identification of each respective object includes a classification and a predefined geometrical shape, the predefined geometrical shape being associated with a subset of the acquired spatial datapoints for each respective object; and
(Huang, “vehicles 102, pedestrians, bicyclists, wildlife, vegetation, or any of various other moving and/or stationary objects”, [0054]; “compare (e.g., position-wise, orientation-wise, size-wise, and so forth) the estimated 3D bounding box (e.g., cuboid) generated by the 3D object detector 326”, [0040]; “the 3D object detector 326 may the generate an estimation (e.g., a prediction) of a 3D bounding box (e.g., cuboid) within the 3D point cloud 314 (or 3D BEV representation), indicating, for example, a detection and classification of the corresponding one or more agents 108 in 3D space”, [0039]; identifying objects with a classification (e.g., vehicle) and a shape (cuboid) associated with the point cloud subset)
wherein the target range is determined by performing the steps of:
selecting a portion of the spatial datapoints which include a respective subset of the spatial datapoints for each of a plurality of sample objects,
(Huang, Fig. 3; “the 3D object detector 326 may the generate an estimation (e.g., a prediction) of a 3D bounding box (e.g., cuboid) within the 3D point cloud 314 (or 3D BEV representation), indicating, for example, a detection and classification of the corresponding one or more agents 108 in 3D space”, [0039]; the bounding box includes only portion of the 3D point cloud 314; “accessing a training sample including (1) an image of a scene, (2) depth measurements of the scene, and (3) a predetermined 3D position of an object in the scene”, [0063]; “determining a subset of the depth measurements of the scene that correspond to the object”, [0044]; processing a "training sample" (which contains a plurality of sample objects) and selecting the specific "subset" of points corresponding to each object to train the model)
determining a respective value of the at least one property for each sample object based on the predefined geometrical shape and/or the respective subset of the spatial datapoints,
(Huang, “the 3D object detector 326 may generate a 3D bounding box as a proposal within the 3D point cloud 314 to be compared (e.g., position-wise, orientation-wise, size-wise, and so forth) to a ground truth 3D bounding box (e.g., ground truth cuboid) for the corresponding one or more agents 108 to determine whether the estimated 3D bounding box generated by the 3D object detector 326 is accurate”, [0039]; comparing object properties with pre-defined ground truth values)
estimating at least one probability distribution for the property based on a statistical distribution for the values of the property for the sample objects, and
deriving the target range for the at least one property of the respective objects from the at least one probability distribution.
(Huang, Fig. 3; “3D object detector 326 may include, for example, an ML model or neural network that may be similar to the neural network 320”, [0038]; the output from 3D detector 326 is the result of a statistical process of a CNN (e.g., the statistical convolutional process); when compare the output with the ground truth 316, the threshold for identifying if the 3D bounding box is correctly labeled/annotated is usually derived from a similarity (i.e., a similarity measure of probability distributions) between the estimated 3D bounding boxes from 326 and the ground truth from 316); “the neural network 318 and the neural network 320 may each include ... a neural autoregressive distribution estimation (NADE) network ... suitable for extracting features”, [0034]; using a neural network—specifically citing a "distribution estimation (NADE)" network which inherently estimates a statistical probability distribution based on the sample objects (training data); the "target range" for validation is derived from this learned distribution (e.g., the decision boundary or loss threshold established by the statistical process of the network))
Regarding claim 3, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the predefined geometrical shape is a cuboid.
(Huang, “the 3D object detector 326 may the generate an estimation (e.g., a prediction) of a 3D bounding box (e.g., cuboid) within the 3D point cloud 314 (or 3D BEV representation), indicating, for example, a detection and classification of the corresponding one or more agents 108 in 3D space”, [0039])
Regarding claim 5, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according claim 1, wherein
a probability value is determined for the value of the at least one property based on at least one probability distribution, and
the respective value of the at least one property is outside the target range if the probability value is smaller than a predetermined threshold.
(Huang, see comments on claims 1 and 4; obviously, if the similarity is smaller than a predefined similarity threshold, the estimated 3D bounding box is not correctly labeled)
Regarding claim 10, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the at least one property is derived from a spatial distribution of the datapoints of the respective subset with respect to the predefined geometrical shape.
(Huang, Fig. 3; “position-wise, orientation-wise, size-wise”, [0040]; the shape of a bounding box is formed by different positions, orientations and sizes of the box edges)
Regarding claim 11, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the at least one property includes at least one statistical property of the datapoints of the respective subset.
(Huang, Fig. 3; “the detected features of the 3D data points may include, for example, a property, a density, a unique value, an intensity, or other similar feature of each of the 3D data points lying within the generated 3D viewing frustum that may indicate the 3D data point as corresponding the particular one or more agents 108 of interest”, [0051]; density distribution in 3D point cloud is a statistical property)
Regarding claim 12, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein the at least one property of the objects includes parameters of a spatial location of the objects.
(Huang, Fig. 3; “position-wise, orientation-wise, size-wise”, [0040])
Regarding claim 13, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein
the plurality of spatial datapoints is based on a sequence of Lidar scans for a predetermined time period, and
the at least one property includes a respective velocity of the objects with respect to the sensor, wherein the respective velocity is determined based on the sequence of Lidar scans.
(Huang, Fig. 3; “Huang, Figs. 1-3, “the LiDAR sensors 208 and 3D data pipeline 214 (e.g., LiDAR pipeline 214) may capture and process, for example, one or more 3D point clouds”, [0024]; a Lidar scans a target point by point and generates a 3D point sequence which forms a point cloud; the Lidar scan may be performed in motion; “the one or more agents 108 may include any potential objects the vehicle 102 may encounter along its drive trajectory”, [0022]; “perception module 202 may track the velocities, moving directions, accelerations, trajectories, relative distances, or relative positions of these agents 108”, [0027]; it’s well-known that Lidar can measure relative speed to a target by tracking target position change over time)
Regarding claim 14, the combination of Huang and Adhikari teaches its/their respective base claim(s).
The combination further teaches the method according to claim 1, wherein for each object being selected for the review regarding erroneous annotation, a potential annotation error is indicated.
(Huang, Adhikari, see comments on claim 1; Huang, Fig. 3, the annotations to the estimated 3D bounding boxes with high regression loss are all potential annotation errors; they can be the real errors if the updated ML model or a human annotator cannot find matches between the estimated bounding boxes and the available labels; or they can also be correctable errors, meaning that they can become valid annotations after the ML model is updated or after a human annotator corrects the 3D bounding boxes)
Allowable Subject Matter
Claim(s) 6-9 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 6 and 7 recite(s) limitation(s) related to iteratively selecting objects with lowest probability values until reaching a predefined percentage share for review; and multi-class classification where a separate probability distribution is estimated for each individual object class. There are no explicit teachings to the above limitation(s) found in the prior art cited in this office action and from the prior art search.
Claim(s) 8-9 depend on claim 7.
Response to Arguments
Applicant's arguments filed on 12/11/2026 with respect to one or more of the pending claims have been fully considered but they are not persuasive.
Regarding claim(s) 1, Applicant, in the remarks, argues that the combination of the cited reference(s) fails to teach the newly amended limitations in the claims.
The Examiner respectfully disagreed. The office action has been updated to address applicant’s argument. See the updated review comments for details.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 2/1/2026