Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Detailed Action
This action is in response to the claims filed 12/21/2023:
Claims 1 – 20 are pending.
Claims 1, 7, and 16 are independent.
Drawings
The drawings are objected to because FIG. 1A-C, 5, 6A-D,8, and 10A-D are low quality scans containing illegible elements. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 2, 7, 8, and 17 are objected to because of the following informalities:
Regarding claims 2, 8, and 17, "class wise" should read "class-wise" for consistency.
Regarding claim 7, "configured with a deep neural network" should read "configured with a deep neural network (DNN)".
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1, 7, and 16, "the object bounding box" lacks antecedent basis. "An object bounding box" is recommended.
Regarding claims 2, 8, and 17, "a certainty with accuracy" is indefinite. It's grammatically unclear what is intended by the phrase "a certainty with accuracy" and one of ordinary skill in the art would not readily be able to determine the scope of either "certainty" or "accuracy". It is similarly unclear what the difference is between in "determining the multi-class confidence calibration by determining a difference between a fused mean confidence and a certainty with accuracy". For example, a fused mean confidence, a certainty, and an accuracy all have unclear grammatical relationships between each other and "a difference between".
Regarding claims 2, 8, and 17, "the fused mean confidence is between a mean logits-based class-wise confidence and class wise certainty" is indefinite. "Between" is a relative metric and the instant claim does not provide any relative basis for comparison that would allow one of ordinary skill in the art to reasonably limit the scope of "between a mean logits-based class-wise confidence and class wise certainty". In the interest of further examination any relationship "between" a mean logits-based class-wise confidence and class wise certainty is interpreted as satisfying the fused mean confidence.
Regarding claim 3, "determining the localization calibration by determining a deviation between a predicted mean bounding box overlap and a predictive certainty of the bounding box" is replete with grammatical and idiomatic errors. For example, "predicted mean bounding box overlap" is not idiomatic technical English. It could mean a predicted overlap, a mean predicted bounding box's overlap, a mean overlap of predicted bounding boxes, some predicted mean value of overlap, or something else altogether. As each of these are contradictory interpretations the scope of the claim cannot be determined. Similarly, one of ordinary skill in the art would not know how to determine "a deviation between a predicted mean bounding box overlap and a predictive certainty of the bounding box": for example, is a deviation a difference, a squared error, a statistical deviation, etc. and how is it computed between an overlap and a certainty? Claim 3 recites a vague relation between two poorly defined quantities of different types. "Similarly, "determining the localization calibration by determining" is circular. In the interest of further examination the claim is interpreted as computing an error based on a bounding box overlap and a predictive certainty. Claims 9 and 18 are rejected for the same reasons.
Regarding claim 4, "the confidence of a predicted label and a non-predicted label" lacks antecedent basis. "A confidence" is recommended.
Regarding claim 14, "the braking system" lacks antecedent basis. "A braking system" is recommended.
Regarding claim 14, "the steering system" lacks antecedent basis. "A steering system" is recommended.
The remaining claims are rejected with respect to their dependence on the rejected claims.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under U.S.C. §102(a)(1) as being anticipated by Koivisto (US11210537B2).
Regarding claim 1, Koivisto teaches A method of training a deep neural network (DNN) for multi-class object detection using an object detection system, ([Col. 3 l. 1-6] "The present systems and methods for object detection and detection confidence suitable for autonomous driving is described in detail below with reference to the attached 5 drawing figures" [Col. 36 l. 30-33] "the model training 414 may train the object detector 306 using the ground truth data")
the object detection system including a camera and a controller having the DNN, the method comprising:([Col. 8 l. 31-35] "the object detection system 100 may be provided via multiple devices arranged in a distributed environment that collectively provide the functionality described herein, or may be embodied on a single device (e.g., the vehicle 1500). " [Col. 54 l. 36] "The vehicle may further include any number of camera types")
capturing an image by the camera; receiving, by the controller, the image;([Col. 1 l. 14-23] "an object detector may be used to accurately detect objects depicted in an image(s) in real-time (e.g., images captured using one or more sensors mounted on the autonomous vehicle)")
predicting, using the DNN, at least one bounding box and a class label with a confidence score for the image;([Col., 15 l. 6-56] "The object detector 306 includes a neural network(s), such as a CNN(s) used for object detection. The CNN(s) may use a gridbox or other architecture, as illustrated in FIG. 3, with N (e.g., 256, 512, or 1024) number of spatial elements [...] The location data 310 may represent any number of location coordinates defining a detected object region, such as four coordinates that correspond to the corners of a bounding box region" [Col. 16 l. 16-33] "The outputs of the object detector 106 (e.g., an output layer(s) 330 of the neural network(s)) may typically be provided for each spatial element (e.g., grid or spatial cell) and may be referred to as “raw” outputs or detections for a detected object. Any combination of the various raw outputs or detections may be what the detected object filter 116A uses to filter the detected objects provided by the object detector 306. For example, various thresholds may be used as cutoffs for any combination of the various outputs. Further, although not shown, the outputs may include class data representative of a class of the detected object, and different thresholds may be used for different classes" [Col. 19 l. 30-40] "The confidence score generator 112 may generate a confidence score for a cluster or aggregated detection based at least in part on features associated with at least the aggregated detection. The features may be provided using the feature determiner 110. The confidence score generator 112 may take a vector(s) (e.g., a single vector) of features as its input(s) and output a confidence score(s)" See also FIG. 2A)
calibrating the DNN by a multi-class confidence calibration, and([Col. 5 l. 25-30] "trained to determine the confidence score using features generated from detected objects" [Col. 48 l. 15-25] " a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections" [Col. 18 l. 55-63] "each of the spatial elements on this feature map may output for each of the C considered classes any of the various detected object data described herein with respect to the output layer(s) 330." Because Koivisto's object detector outputs data for C considered classes the confidence calibration operates in a multi-class object detection context)
a bounding box localization calibration; and([Col. 33 l. 14-25] "The ground truth values for bounding box outputs may include at least four values [B1, B2, B3, B4], obtained by applying functions to the label coordinates and spatial element center coordinates. These functions may be parametric, and may include parameters that are hyperparameters of a training procedure tuned and cross validated for output accuracy of the object detector 106")
outputting, by the controller, a calibrated image with the object bounding box, the corresponding class label, and a respective confidence score, ([Col. 3 l. 15-16] "FIG . 2A is an illustration of an image overlaid with visual
elements that correspond to detected objects" [Col. 9 l. 40-45] "The image 204 depicts regions of the environment 202, where the regions may include any number of objects, examples of which include objects 248A, 248B, 248C, and 248D, which are labeled in FIG. 2B")
wherein the confidence score is a probability associated with the predicted class label.([Col. 2 l. 20-28] "a confidence score (e.g., a scalar value) that may be directly interpreted as a confidence measure and may accurately indicate a probability that an aggregated detection corresponds to an actual object represented in image data.").
Regarding claim 2, Koivisto teaches The method of claim 1, further comprising determining the multi-class confidence calibration by determining a difference between a fused mean confidence and a certainty with accuracy, (Koivisto [Col. 20 l. 15-40] "During training, false positives may be assigned a label ‘0’ while true positives may be assigned a label ‘1’ for the confidence score. Using this approach, outputs from the confidence score generator 112 (e.g., corresponding to a single output node 142 in some examples) may be compared to the ground truth labels. The comparisons may be used to compute error terms and the error terms may be used by a backpropagation algorithm to adjust the weights of the machine learning model(s)." Koivisto determines confidence calibration by comparing confidence-score outputs to ground truth true/false positive labels and computing error terms)
wherein the fused mean confidence is between a mean logits-based class-wise confidence and class wise certainty.(Koivisto [Col. 21 l. 1-15] "the feature determiner 110 computes a statistic(s) (e.g., a statistical value(s)), and provides the statistic(s), or data derived therefrom as a feature. Examples of statistics include a sum, a mean, an average […] the statistic is based on an IOU between detected object regions, such as a maximum IOU between detected object regions (an example equation for computing an IOU is provided herein)").
Regarding claim 3, Koivisto teaches The method of claim 1, further comprising determining the localization calibration by determining a deviation between a predicted mean bounding box overlap and a predictive certainty of the bounding box.(Koivisto [Col. 21 l. 1-15] "the feature determiner 110 computes a statistic(s) (e.g., a statistical value(s)), and provides the statistic(s), or data derived therefrom as a feature. Examples of statistics include a sum, a mean, an average […] the statistic is based on an IOU between detected object regions, such as a maximum IOU between detected object regions (an example equation for computing an IOU is provided herein)").
Regarding claim 4, Koivisto teaches The method of claim 1, further comprising determining the multi-class confidence calibration by calibrating the confidence of a predicted label and a non-predicted label.(Koivisto [Col. 65 l. 9-13] "the neural network is trained to minimize the probability when the aggregated 10 detection is a false positive detection and maximize the probability when the aggregated detection is a true positive detection" [Col. 20 l. 15-30] "the feature determiner 110 may determine the features and provide the features as inputs to the confidence score generator 112. Ground truth labeling may be used to label the clusters as false positives or true positives. During training, false positives may be assigned a label ‘0’ while true positives may be assigned a label ‘1’ for the confidence score." Koivisto true positive interpreted as a predicted label and false positive interpreted as a non-predicted label).
Regarding claim 5, Koivisto teaches The method of claim 1, further comprising perceiving, by the DNN, multiple object classes in the received image.(Koivisto [Col. 5 l. 25-30] "trained to determine the confidence score using features generated from detected objects" [Col. 48 l. 15-25] " a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections" [Col. 18 l. 55-63] "each of the spatial elements on this feature map may output for each of the C considered classes any of the various detected object data described herein with respect to the output layer(s) 330." See also FIG 2A).
Regarding claim 6, Koivisto teaches The method of claim 5, further comprising outputting, by the controller, a control action based on the calibrated image with the object bounding box and the corresponding label.(Koivisto [Col. 48 l. 25-35] "the most confident detections should be considered as triggers for AEB.").
Regarding claims 7-12, claims 7-12 are directed towards a vehicle safety-critical control system for performing the method of claims 1-6. Therefore, the rejections applied to claims 1-6 also apply to claims 7-12.
Regarding claim 13, Koivisto teaches The system of claim 7, further comprising a braking system, (Koivisto [Col. 39 l. 58-Col. 40 l. 17] "Controller(s) 1536, which may include one or more system on chips (SoCs) 1504 (FIG. 15C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle 1500. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators 1548, to operate the steering system 1554 via one or more steering actuators 1556, to operate the propulsion system 1550 via one or more throttle/accelerators 1552")
wherein the controller is further configured to actuate the braking system based on an object predicted by the controller.(Koivisto [Col. 40 l. 18-35] "The controller(s) 1536 may provide the signals for controlling one or more components and/or systems of the vehicle 1500 in response to sensor data received from one or more sensors (e.g., sensor inputs)." [Col. 48 l. 14-48] "The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB").
Regarding claim 14, Koivisto teaches The system of claim 7, further comprising a steering system, the controller further configured to actuate the steering system in conjunction with the braking system based on an object predicted by the controller.(Koivisto [Col. 40 l. 18-35] "The controller(s) 1536 may provide the signals for controlling one or more components and/or systems of the vehicle 1500 in response to sensor data received from one or more sensors (e.g., sensor inputs)." [Col. 48 l. 14-48] "The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB").
Regarding claim 15, Koivisto teaches The system of claim 7, further comprising a transmission system, the controller further configured to actuate the transmission system, (Koivisto [Col. 39 l. 40-50] "The propulsion system 1550 may be connected to a drive train of the vehicle 1500, which may include a transmission, to enable the propulsion of the vehicle 1500. The propulsion system 1550 may be controlled in response to receiving signals from the throttle/accelerator 1552.")
the steering system, and the braking system based on an object predicted by the controller.(Koivisto [Col. 40 l. 18-35] "The controller(s) 1536 may provide the signals for controlling one or more components and/or systems of the vehicle 1500 in response to sensor data received from one or more sensors (e.g., sensor inputs)." [Col. 48 l. 14-48] "The DLA may be used to run any type of network to enhance control and driving safety, including for example, a neural network that outputs a measure of confidence for each object detection. Such a confidence value may be interpreted as a probability, or as providing a relative “weight” of each detection compared to other detections. This confidence value enables the system to make further decisions regarding which detections should be considered as true positive detections rather than false positive detections. For example, the system may set a threshold value for the confidence and consider only the detections exceeding the threshold value as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB").
Regarding claims 16-20, claims 16-20 are directed towards a computer readable media for performing the method of claims 1-5. Therefore, the rejections applied to claims 1-5 also apply to claims 16-20. Claims 16-20 recite additional elements A non-transitory computer-readable storage medium including computer executable instructions, wherein the instructions, when executed by a computer, cause the computer to perform a method comprising: (Koivisto [Col. 35 l. 14-22] “The method 1000A may also be embodied as computer-usable instructions stored on computer storage media.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Munir (“Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection”, 2023) is directed towards an object detection calibration system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights