DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 6/24/2026 has been entered. Claims 1, 3, and 5-9 have been amended. Claims 2 and 4 have been cancelled. Claims 11-14 are new. Claims 1, 3, 5-14 are pending in the application.
Specification
The specification is objected to as failing to provide proper antecedent basis for the claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP 608.01(o). Correction of the following is required: claim 12 recites "controlling the autonomous or semi-autonomous operating machine," but the specification does not use the term "controlling," or any cognate of "control," anywhere in the disclosure. The closest description is that the symbolic description "forms the basis for performing actions in the environment corresponding to the application or intended use of the machine" and that "the objects can be obstacles with which a collision must be avoided," at paragraph [0028] of the specification. Applicant should amend the specification to provide antecedent basis for the claimed controlling step, without introducing new matter.
Claim Objections
Claims 5, 6, 7, 12, 13 and 14 are objected to because of the following informalities:
a) Claim 5, element d), recites "correlations between metrics, which characterize the accuracy and/or a reliability of the primary recognition task and/or correlations between metrics, which characterize the accuracy and/or the reliability of the primary recognition task." The second alternative repeats the first word for word except for the article preceding "reliability." As filed, the second alternative closed with "and tags," which distinguished it from the first; the amendment of 2026-06-24 deleted "and tags," leaving the two alternatives coextensive. Applicant should delete the second alternative or restore a distinction between the two.
b) Claim 5, element f), recites "a recognition performance of certain sensors including the at least one sensor of an autonomous or semi-autonomous operating machine." Claim 1 already recites "at least one sensor of an autonomous or semi-autonomous operating machine," and element f) refers back to that sensor definitely, as "the at least one sensor." The article before "autonomous or semi-autonomous operating machine" should likewise be definite, so that element f) reads "the at least one sensor of the autonomous or semi-autonomous operating machine."
c) Claim 6 recites "by labeling the first partial sample with labels," while the second clause of the same claim recites "by labeling the further partial sample with the labels" and claim 1 recites "labels of the at least one labeled training sample." For consistent terminology, "with labels" should read "with the labels."
d) Claim 7 recites "generating labels for the first partial sample and/or the further partial sample." For consistency with claim 1, which recites "labels of the at least one labeled training sample," this should read "generating the labels."
e) Claim 12 recites "controlling the autonomous or semi-autonomous operating machine based on primary recognition task." The article is missing; this should read "based on the primary recognition task."
f) Claims 13 and 14 each recite "the recognition-accuracy requirement defined for the driving function." No recognition-accuracy requirement is set forth in claim 1, which recites only that "the accuracy is insufficient for a driving function of the autonomous or semi-autonomous operating machine." Applicant should recite "a recognition-accuracy requirement defined for the driving function," or amend claim 1 to set the requirement forth.
Appropriate correction is required.
Claim Rejections - 35 USC 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 5-12 are rejected under 35 U.S.C. 103 as being unpatentable over Frtunikj et al. (hereinafter Frtunikj), US 2022/0164602 A1, in view of Zhdanov et al. (hereinafter Zhdanov), US 11,048,979 B1.
Regarding independent claim 1, Frtunikj teaches an iterative method for determining training data for a primary model to solve a primary recognition task, the iterative method comprising (Frtunikj: [0005], "The methods then include using the function for assigning the importance score to each of the plurality of unlabeled sensor data logs, selecting a subset of the plurality of sensor data logs that have an importance score greater than a threshold, and using the subset of the plurality of sensor data logs for further training the machine learning model trained using the training dataset to generate an updated model"; the subset so selected is the body of data the object detection model (a primary model) is trained on, and the detection and labeling of objects that the model performs is a recognition task; [0022], "The method can be performed a predetermined number of times for a driving session, iteratively performed at a predetermined frequency for a driving session"; the selection is run again and again over the data a driving session produces, so the determination of that training data proceeds as repeated passes): providing sensor data as an unlabeled sample from at least one sensor of an autonomous or semi-autonomous operating machine (Frtunikj: [0023], "the system may receive raw data logs collected (e.g., recorded, obtained, etc.) by sensors mounted on a vehicle during operation (e.g., driving). The received data logs are unlabeled"; the raw data logs (sensor data) arrive from the sensors mounted on the vehicle and carry no labels when received, so they are the unlabeled sample the method works from; [0068], "it may be semi-autonomous in that a human operator may be required in certain conditions or for certain operations"; the vehicle (an autonomous or semi-autonomous operating machine) that carries those sensors is expressly one that may run without a human operator or with a human operator required only in certain conditions); generating pre-labels for the unlabeled sample using the primary model provided as a deep neural network (Frtunikj: [0025], "an example object detection model (e.g., a convolutional neural network (CNN), a mask R-CNN, etc.) may be used to detect and label one or more objects and/or events in each received raw data log"; the object detection model (the primary model) is a convolutional neural network, which is a network of many layers, and it assigns labels to the objects in the unlabeled logs before any human annotation is obtained; [0025], "The object detection model may output an image or point cloud that includes bounding boxes surrounding the detected objects and/or labels for the objects"; what that model emits for each unlabeled log is a set of boxes and labels (pre-labels) produced by the model itself); evaluating the pre-labels (Frtunikj: [0026], "the object detection model produces for each bounding box a confidence score which indicates a likelihood that the label assigned to the bounding box is correct"; each machine-produced label is scored for the likelihood that it is correct, which is an assessment of the label rather than of the raw log; [0032], "The importance function combines the confidence score for predicting a label in the data log using the machine learning mode with the trends/distributions of the existing training dataset, and assigns an importance score indicative of whether or not the data log will improve the training of the machine learning model"; that per-label confidence is the input to the importance function, so the labels for the objects (the pre-labels) that the model emitted are the thing being assessed); a) selecting a first partial sample of the unlabeled sample, based on the evaluation of the pre-labels, for generating at least one labeled training sample (Frtunikj: [0034], "If the importance score of the data log is greater than (or equal to) the threshold (110: YES), the system may use the data log for further annotation (e.g., manual labeling) and/or for building (i.e., training, testing, validating, updating, etc.) a machine learning model (112)"; the subset of the plurality of sensor data logs (a first partial sample of the unlabeled sample) whose importance score clears the threshold is what is routed for manual labeling and then used to build the model, and that routing turns on the score computed from the confidence of the labels the model itself assigned), wherein labels of the at least one labeled training sample characterize locations and/or attributes of objects represented in the sensor data (Frtunikj: [0025], "Each detected object may be represented by its location (centroid or center point), boundary, label class, or even size"; the label attached to each detected object states where the object is and what class and size it has, so the labels (labels of the at least one labeled training sample) give the locations and attributes of the objects appearing in the logs); b) training the primary model with the at least one labeled training sample to solve the primary recognition task by generating symbolic descriptions of the objects (Frtunikj: [0005], "using the subset of the plurality of sensor data logs for further training the machine learning model trained using the training dataset to generate an updated model"; the selected and annotated subset is fed back into the object detection model (the primary model), which is trained further on it; [0025], "The object detection model may output an image or point cloud that includes bounding boxes surrounding the detected objects and/or labels for the objects"; what the trained model then produces for each object is a bounding box together with a label class (symbolic descriptions of the objects)); c) selecting a further partial sample of the unlabeled sample, based on the evaluation of the pre-labels, for generating at least one labeled test sample (Frtunikj: [0022], "while the method describes selection of training data, the disclosure is not so limiting and similar principles can be used for selection of test data, validation data, or the like"; the same importance-based selection is applied a second time, for selection of test data, validation data (a further partial sample of the unlabeled sample), out of the same pool of unlabeled logs; [0020], "the selection of high-quality labeled training examples (and/or validation/test examples) amongst the virtually unlimited sensor data collected by vehicles available to the machine learning algorithm is typically costly"; the test and validation examples so drawn are themselves labeled examples (at least one labeled test sample), and their selection is the costly step the scoring is introduced to make efficient).
Frtunikj does not expressly teach d) evaluating a recognition performance of the primary model using the at least one labeled test sample on the primary recognition task, the evaluating comprising determining an accuracy with which the trained primary model generates the symbolic descriptions; and depending on a result of the evaluating the recognition performance either (i) re-performing parts a), b), c), and d) of the iterative method, or (ii) ending the iterative method, wherein when the result indicates that the accuracy is insufficient for a driving function of the autonomous or semi-autonomous operating machine, the re-performing includes selecting a second partial sample of the unlabeled sample and further training the primary model for improving the accuracy of the primary recognition task for the driving function, and wherein the pre-labels and the labels are a same type, such that the pre-labels are usable as the labels.
However, Zhdanov teaches d) evaluating a recognition performance of the primary model using the at least one labeled test sample on the primary recognition task, the evaluating comprising determining an accuracy with which the trained primary model generates the symbolic descriptions (Zhdanov: 11:20-23, "The validation dataset may be a portion of the input dataset (e.g., 5 or 10%). Which can be manually labeled to provide ground truth labels for the validation set"; the validation set (the at least one labeled test sample) is a portion taken out of the same input data and labeled by hand, its labels serving as ground truth; 11:23-26, "The model may be initially trained using this dataset and the accuracy of the results of inference on the input dataset using the incrementally trained model can be evaluated using the validation dataset"; the accuracy of what the incrementally trained model infers is then measured against that ground truth, which is a measurement of how well the model performs its task; 11:10-11, "an accuracy metric can be identified for each bounding box in each image in the dataset and stored"; the quantity measured is an accuracy figure for each bounding box the model emits, that is, for the descriptions the model produces of the objects it finds (comprising determining an accuracy with which the trained primary model generates the symbolic description)); and depending on a result of the evaluating the recognition performance either (i) re-performing parts a), b), c), and d) of the iterative method, or (ii) ending the iterative method, (Zhdanov: 6:45-51, "The above described process may then be repeated using the updated model. For example, if the updated model has converged, then the remainder of the input dataset can be accurately identified. If the updated model has not converged, then a new subset of the input dataset can be identified for further labeling according to the process described above"; the measured outcome decides the branch: a model that has reached the required level ends the process, and a model that has not sends the system back for a new subset of the same input data and another pass) wherein when the result indicates that the accuracy is insufficient for a driving function of the autonomous or semi-autonomous operating machine, the re-performing includes selecting a second partial sample of the unlabeled sample and further training the primary model for improving the accuracy of the primary recognition task for the driving function (Zhdanov: 11:50-53, "This active learning loop may then be repeated on new portions of the input dataset, with each iteration adding to the labeled dataset and further training the model, until the input dataset has been labeled"; the return branch identifies a new subset of the input dataset (a second partial sample of the unlabeled sample) and trains the model further on it, so the shortfall in measured accuracy is what the further training is directed at removing), and wherein the pre-labels and the labels are a same type, such that the pre-labels are usable as the labels (Zhdanov: 11:33-39, "This portion of the input dataset may include data items (e.g., images, videos, text portions, etc.) that have confidence scores above a threshold based on the current machine learning model, or a sampling of those data items. The auto labeling service 404 can use the current machine learning model to label the portion of the dataset and add it to the labeled dataset 412"; the auto labeling (the pre-labels) the current machine learning model performs on that portion is written straight into the labeled dataset and stands there as a label; 11:46-49, "The resulting labels can be consolidated 410 and added to the labeled dataset 412. Using the new labeled portions of the input dataset, the machine learning model can be further trained"; the annotator-supplied labels (the labels) enter that same labeled dataset and train the model alongside the machine-supplied ones, so the two are of one kind and serve interchangeably).
Because Frtunikj and Zhdanov are analogous art with both addressing the selection of data from a large unlabeled pool for annotation and for the training and evaluation of a model that detects and labels objects, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Zhdanov's ground-truth validation draw and its accuracy-gated repeat decision to the method of Frtunikj, with a reasonable expectation of success, by letting the importance-based selection of Frtunikj draw the validation portion as well as the training portion out of the same unlabeled sensor data logs, measuring the object detection model's accuracy on the annotated validation portion after each round of further training, and returning for a further draw and another round whenever the measured accuracy has not yet reached the level the autonomous vehicle's path planning and collision avoidance require, to teach d) evaluating a recognition performance of the primary model using the at least one labeled test sample on the primary recognition task, the evaluating comprising determining an accuracy with which the trained primary model generates the symbolic descriptions; and depending on a result of the evaluating the recognition performance either (i) re-performing parts a), b), c), and d) of the iterative method, or (ii) ending the iterative method, wherein when the result indicates that the accuracy is insufficient for a driving function of the autonomous or semi-autonomous operating machine, the re-performing includes selecting a second partial sample of the unlabeled sample and further training the primary model for improving the accuracy of the primary recognition task for the driving function, and wherein the pre-labels and the labels are a same type, such that the pre-labels are usable as the labels. This modification would have been motivated by the desire to increase the speed at which inference can be performed and to improve the accuracy of the model with each iteration (Zhdanov: col. 11 lines 1-4).
Regarding dependent claim 5, Frtunikj, in view of Zhdanov, teach the method of claim 1, wherein: at least one of the following elements is considered when evaluating: a) a similarity of the first partial sample and the further partial sample, b) a relevance of the first partial sample and the further partial sample for training the primary model, c) a proportion of conditions in the first partial sample and the further partial sample, d) correlations between metrics, which characterize the accuracy and/or a reliability of the primary recognition task[[ and/or correlations between metrics, which characterize the accuracy and/or the reliability of the primary recognition task]] (interpreted per Claim Objections set forth above), e) continuous and/or modified metrics of the primary recognition task, and f) a recognition performance of certain sensors including the at least one sensor of [[an]]the (interpreted per Claim Objections set forth above) autonomous or semi-autonomous operating machine (Frtunikj: [0033], "instance count by environmental conditions (e.g., rain, low light, occlusions, etc.), machine leaning model uncertainty"; the features the selection function scores include the instance count by environmental conditions (a proportion of conditions in the first partial sample and the further partial sample) in a data log, so the make-up of the conditions present in the data the score selects is one of the elements weighed in the evaluation). Elements a) through f) of claim 5 are recited as alternatives of which at least one is considered, and a claim reciting at least one member of such a listing is met when the prior art teaches one member; element c) is the member relied upon.
Regarding dependent claim 6, Frtunikj, in view of Zhdanov, teach the method of claim 1, further comprising: generating the at least one labeled training sample based on the first partial sample by labeling the first partial sample with the labels (interpreted per Claim Objections set forth above) (Frtunikj: [0034], "If the importance score of the data log is greater than (or equal to) the threshold (110: YES), the system may use the data log for further annotation (e.g., manual labeling) and/or for building (i.e., training, testing, validating, updating, etc.) a machine learning model (112)"; a data log the score keeps is sent out for annotation by hand and the annotated log is then used to build the model, so the training example is made by labeling the selected log itself); and generating the at least one labeled test sample based on the further partial sample by labeling the further partial sample with the labels (Zhdanov: 11:20-23, "The validation dataset may be a portion of the input dataset (e.g., 5 or 10%). Which can be manually labeled to provide ground truth labels for the validation set"; the validation portion drawn from the same input data is itself labeled, the labels being supplied by hand as ground truth, so the test example is made by labeling the drawn portion).
Regarding dependent claim 7, Frtunikj, in view of Zhdanov, teach the method of claim 6, further comprising: generating the labels (interpreted per Claim Objections set forth above) for the first partial sample and/or the further partial sample based on the pre-labels as a function of a confidence of the pre-labels (Zhdanov: 11:33-39, "This portion of the input dataset may include data items (e.g., images, videos, text portions, etc.) that have confidence scores above a threshold based on the current machine learning model, or a sampling of those data items. The auto labeling service 404 can use the current machine learning model to label the portion of the dataset and add it to the labeled dataset 412"; the confidence the model assigns to its own output decides which items are labeled from that output, and the label written for those items is the model's own prediction, so the label is produced from the machine-made label as a function of its confidence).
Regarding dependent claim 8, Frtunikj, in view of Zhdanov, teach the method of claim 1, wherein re-performing parts of the iterative method as a function of the result of evaluating the recognition performance comprises: a) providing the unlabeled sample (Zhdanov: 11:50-53, "This active learning loop may then be repeated on new portions of the input dataset, with each iteration adding to the labeled dataset and further training the model, until the input dataset has been labeled"; each further pass returns to the items of the input data that are still unlabeled and runs the same steps over them), b) generating the pre-labels and/or tags for the unlabeled sample (Zhdanov: 11:37-39, "The auto labeling service 404 can use the current machine learning model to label the portion of the dataset and add it to the labeled dataset 412"; on that pass the current machine learning model again produces labels for the items of that portion, which is the same generation of labels carried out afresh), c) evaluating the pre-labels and/or tags, and based on the evaluating, selecting (i) the second partial sample of the unlabeled sample to generate the at least one labeled training sample, and (ii) a third partial sample of the unlabeled sample to generate the at least one labeled test sample (Zhdanov: 11:33-37, "This portion of the input dataset may include data items (e.g., images, videos, text portions, etc.) that have confidence scores above a threshold based on the current machine learning model, or a sampling of those data items"; the confidence the current machine learning model attaches to its own labels again decides which items are taken for the labeled dataset, which is the same assessment carried out afresh; 11:62-64, "At each iteration of the active learning loop, the validation set for that iteration is different"; the validation set for that iteration (a third partial sample of the unlabeled sample) is a different one on every pass, so a fresh evaluation draw accompanies the fresh training draw), and d) generating the at least one labeled training sample based on the second partial sample and generating the at least one labeled test sample based on the third partial sample (Zhdanov: 11:40-42, "a second portion of the input dataset can be sent to manual labeling service 408 to be annotated by one or more human annotators"; the newly drawn training portion is sent to human annotators, who label it; 11:20-23, "The validation dataset may be a portion of the input dataset (e.g., 5 or 10%). Which can be manually labeled to provide ground truth labels for the validation set"; the newly drawn validation portion is likewise labeled by hand to serve as ground truth). The recitation of pre-labels and/or tags in claim 8 is an alternatives listing, and the pre-labels are the member relied upon.
Regarding dependent claim 9, Frtunikj, in view of Zhdanov, teach the method of claim 1, wherein the evaluation of the recognition performance of the primary model is based on metrics for characterizing a reliability and/or the accuracy of the primary recognition task of the primary model (Zhdanov: 11:10-11, "an accuracy metric can be identified for each bounding box in each image in the dataset and stored"; how well the model performs is settled through a stored accuracy figure computed for each box the model emits; 11:15-17, "In some embodiments the metric may be an intersection over union (IoU) metric for each bounding box"; that figure is an intersection over union value, which expresses how closely the model's box matches the annotated box and is therefore a measure of the accuracy of the model's output).
Regarding dependent claim 10, Frtunikj, in view of Zhdanov, teach the method of claim 1, further comprising: using the training data determined according to the iterative method to train the primary model to solve the primary recognition task (Frtunikj: [0005], "using the subset of the plurality of sensor data logs for further training the machine learning model trained using the training dataset to generate an updated model"; the subset the method determines is the training data, and that training data is then put to use training the object detection model (the primary model) so that it performs its detection and labeling task better).
Regarding dependent claim 11, Frtunikj, in view of Zhdanov, teach the method of claim 6, further comprising: manually labeling the first partial sample with the labels (Frtunikj: [0034], "If the importance score of the data log is greater than (or equal to) the threshold (110: YES), the system may use the data log for further annotation (e.g., manual labeling) and/or for building (i.e., training, testing, validating, updating, etc.) a machine learning model (112)"; the logs the selection keeps are routed for labeling by a person rather than by the model), and manually labeling the further partial sample with the labels (Zhdanov: 11:20-23, "The validation dataset may be a portion of the input dataset (e.g., 5 or 10%). Which can be manually labeled to provide ground truth labels for the validation set"; the validation portion drawn from the same input data is labeled by hand so that its labels can stand as ground truth).
Regarding dependent claim 12, Frtunikj, in view of Zhdanov, teach the method of claim 1, further comprising: controlling the autonomous or semi-autonomous operating machine based on the primary recognition task (interpreted per Claim Objections set forth above) to avoid the objects represented in the sensor data (Frtunikj: [0057], "The on-board computing device 612 may also assess the risk of a collision between a detected object and the autonomous vehicle 601. If the risk exceeds an acceptable threshold, it may determine whether the collision can be avoided"; the vehicle's on-board computer takes an object the model has detected and weighs the collision it presents, then settles whether that collision can be avoided; [0057], "If the collision can be avoided, then the on-board computing device 612 may execute one or more control instructions to perform a cautious maneuver (e.g., mildly slow down, accelerate, change lane, or swerve)"; on that finding the same on-board computer executes control instructions that slow, accelerate, change the lane of or swerve the vehicle (the autonomous or semi-autonomous operating machine), so the vehicle is controlled on the strength of the object the detection task returns and away from that object).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Frtunikj in view of Zhdanov, as applied in the rejection of claim 1 above, and further in view of Je et al. (hereinafter Je), US 2021/0334652 A1.
Regarding dependent claim 3, Frtunikj, in view of Zhdanov, teach the method of claim 1.
Frtunikj and Zhdanov do not expressly teach generating tags using a secondary model for the unlabeled sample, the secondary model provided as a deep neural network; evaluating the tags; selecting the first partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags; and selecting the further partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags.
However, Je teaches generating tags using a secondary model for the unlabeled sample, the secondary model provided as a deep neural network (Je: FIG. 5 and [0051], "a process of applying a learning operation to each of the frames and thus classifying each of the scenes of each of the frames into one of preset classes of driving environments and one of preset classes of driving roads, to thereby generate each of class codes of each of the frames, via a scene classifier 1210 based on deep learning"; each driving frame is run through the scene classifier 1210 based on deep learning (a secondary model), a classifier that works by deep learning (a deep neural network), and the classifier returns a class code for that frame; [0052], "the information on weather may include information on weather phenomena like sunshine, rain, snow, fog, etc. and the information on the time zone may include information like day, night, etc"; the classes a class code carries are the weather and the time of day the frame was taken in, so the code states a condition of the scene rather than an object in it; [0055], "a process of generating each of the scene codes of each of the frames by using each of the class codes of each of the frames and each of the event codes of each of the frames"; the scene codes (the tags) are built from those class codes, so they are generated using the classifier); evaluating the tags (Je: [0073], "in case the scene code corresponds to a rainy night, a frame where a pedestrian is detected may be determined as a hard example, that is, an example which has the degree of usefulness higher than a threshold usefulness value, to be used for training the perception network"; the scene codes (the tags) are themselves weighed: a rainy-night code makes a frame showing a detected pedestrian a hard example whose usefulness clears the threshold); selecting the first partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags (Je: [0064], "a process of selecting a first part of the frames, whose object detection information generated during the driving events satisfies a preset condition, as specific frames to be used for training the perception network of the autonomous vehicle, via a frame selecting module 1300, by using each of the scene codes of each of the frames and the object detection information"; a first part of the frames (a first partial sample of the unlabeled sample) is drawn only where the object detection information (the pre-labels) satisfies a preset condition, and the scene codes (the tags) enter the same draw, so the draw rests on an assessment of both); and selecting the further partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags (Je: [0065], "a process of selecting a second part of the frames, matching with a training policy of the perception network of the autonomous vehicle, as the specific frames among the frames by using the scene codes and the object detection information"; a second part of the frames (a further partial sample of the unlabeled sample) is drawn on the same scene codes and object detection information, so the same assessed pair of signals governs a further draw as well as the first).
Because Frtunikj, in view of Zhdanov, and Je are analogous art with all three addressing the selection of vehicle sensor data for training a network that recognizes objects, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to add Je's deep-learning scene classifier to the method of Frtunikj, in view of Zhdanov, with a reasonable expectation of success, by running that classifier over the same unlabeled frames the detection network pre-labels and folding the condition codes it returns into the selection signal before either the training draw or the evaluation draw is taken, to teach generating tags using a secondary model for the unlabeled sample, the secondary model provided as a deep neural network; evaluating the tags; selecting the first partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags; and selecting the further partial sample of the unlabeled sample, based on the evaluation of the pre-labels and the tags. This modification would have been motivated by the desire to improve the efficiency of training the perception network with new training data (Je: [0081]).
Claims 13 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Frtunikj in view of Zhdanov, as applied in the rejection of claim 1 above, and further in view of Farabet et al. (hereinafter Farabet), US 2019/0303759 A1.
Regarding dependent claim 13, Frtunikj, in view of Zhdanov, teach the method of claim 1.
Frtunikj and Zhdanov do not expressly teach wherein when the result indicates that the accuracy satisfies [[the]]a recognition-accuracy requirement (interpreted per Claim Objections set forth above) defined for the driving function, the trained primary model is validated for use by the driving function to generate the symbolic descriptions of the objects from the sensor data.
However, Farabet teaches wherein when the result indicates that the accuracy satisfies a recognition-accuracy requirement defined for the driving function, the trained primary model is validated for use by the driving function to generate the symbolic descriptions of the objects from the sensor data (Farabet: [0043], "Once the DNNs have been trained to an acceptable level of accuracy (e.g., 90%, 95%, 97%, etc.), the training and refinement process may move to a second workflow, such as workflow 300B"; an acceptable level of accuracy is fixed for the networks and the refinement process advances only once the networks have been trained to that level, so the level is settled in advance and the measured accuracy is judged against it; [0046], "The conditions or a combination of the conditions which the current DNNs are not considered to perform sufficiently well on (e.g., have an accuracy below a desired or required level) may be used to direct mining and labeling of data (e.g., additional data) that may increase the accuracy of the DNNs with reference to the conditions or combination of conditions"; the level so fixed is a desired or required level (a recognition-accuracy requirement) that the accuracy of the current DNNs is weighed against, an accuracy below it being what sends the system back to mine and label further data instead; [0031], "the AV perception DNNs may be used for detecting lanes and boundaries on driving surfaces, for detecting drivable free-space, for detecting traffic poles or signs, for detecting traffic lights, for detecting objects in the environment"; the networks that level is fixed for are the AV perception DNNs (the primary model), whose work is detecting objects in the environment (the objects), so the level is one fixed for the driving perception those networks carry out; [0032], "Once verified and/or validated, the validated AV perception DNNs and/or the validated IX perception DNNs may be incorporated into software stack(s) 116 (e.g., the IX software stack and/or the autonomous driving software stack)"; a network whose accuracy has been verified and validated is on that footing incorporated into the autonomous driving software stack (the driving function), so reaching the level is what passes the finished network to that software for its use; [0170], "3D location estimates of the object obtained from the neural network"; what such a network returns for each object it finds are 3D location estimates of the object (the symbolic descriptions of the objects), which is the reading of the object the network is kept for; [0029], "the vehicle hardware 104 may include the hardware of the vehicle 102 that is used to control the vehicle 102 through real-world environments based on the sensor data, one or more machine learning models (e.g., neural networks), and/or the like"; the hardware of the vehicle runs those neural networks together with the sensor data in controlling the vehicle through real-world environments, so the sensor data is what the networks are given and the readings they return from it are what the vehicle is controlled on).
Because Frtunikj, in view of Zhdanov, and Farabet are analogous art with all three addressing the training and the accuracy evaluation of a neural network that recognizes objects in the sensor data an autonomous vehicle collects, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Farabet's fixed accuracy level and its validation gate to the method of Frtunikj, in view of Zhdanov, with a reasonable expectation of success, by fixing in advance the accuracy the object detection model must reach for the vehicle's path planning and collision avoidance, weighing against that level the accuracy the model attains on the annotated validation portion after a round of further training, and, when the level is reached, treating the model as validated and passing it to the software that drives the vehicle on its detections, to teach wherein when the result indicates that the accuracy satisfies a recognition-accuracy requirement defined for the driving function, the trained primary model is validated for use by the driving function to generate the symbolic descriptions of the objects from the sensor data. This modification would have been motivated by the desire to reach an acceptable level of safety before an autonomous vehicle is deployed into the real world (Farabet: [0004]).
Regarding dependent claim 14, Frtunikj, in view of Zhdanov, teach the method of claim 1.
Frtunikj and Zhdanov do not expressly teach wherein, when the result indicates that the accuracy satisfies [[the]]a recognition-accuracy requirement (interpreted per Claim Objections set forth above) defined for the driving function, the trained primary model is validated for the driving function, and the driving function is enabled after the validation to use the trained primary model to generate the symbolic descriptions of the objects from the sensor data.
However, Farabet teaches wherein, when the result indicates that the accuracy satisfies a recognition-accuracy requirement defined for the driving function, the trained primary model is validated for the driving function (Farabet: [0046], "The conditions or a combination of the conditions which the current DNNs are not considered to perform sufficiently well on (e.g., have an accuracy below a desired or required level) may be used to direct mining and labeling of data (e.g., additional data) that may increase the accuracy of the DNNs with reference to the conditions or combination of conditions"; a desired or required level (a recognition-accuracy requirement) is set for the current DNNs and the accuracy each attains is weighed against it, an accuracy below the level calling for further data rather than release; [0032], "The validation/verification sub-system 112 may verify and/or validate performance, accuracy, and/or other criteria associated with the DNNs"; the sub-system that follows the training weighs the accuracy the network has reached, which is the determination made before anything is released; [0032], "Once verified and/or validated, the validated AV perception DNNs and/or the validated IX perception DNNs may be incorporated into software stack(s) 116 (e.g., the IX software stack and/or the autonomous driving software stack)"; only a network that has come through that determination is incorporated into the autonomous driving software stack (the driving function), so the network is validated for that software and for nothing sooner), and the driving function is enabled after the validation to use the trained primary model to generate the symbolic descriptions of the objects from the sensor data (Farabet: [0032], "Once incorporated into the software stack(s) 116, the vehicle(s) 102 may execute the software stack(s) using the vehicle hardware 104 to control the vehicle(s) 102 within real-world environments"; with the network so incorporated the vehicle then executes the autonomous driving software stack (the driving function) on its own hardware and is controlled through real-world environments on what that software returns, so the software is put to work on the network only after the validation; [0170], "3D location estimates of the object obtained from the neural network"; what the network returns to that software for each object it finds are 3D location estimates of the object (the symbolic descriptions of the objects), on which the vehicle's driving software then acts; [0029], "the vehicle hardware 104 may include the hardware of the vehicle 102 that is used to control the vehicle 102 through real-world environments based on the sensor data, one or more machine learning models (e.g., neural networks), and/or the like"; the hardware of the vehicle runs the neural networks on the sensor data in controlling the vehicle through real-world environments, so the readings the released network supplies are drawn from the sensor data).
Because Frtunikj, in view of Zhdanov, and Farabet are analogous art with all three addressing the release of a trained object-recognition network into the software that drives an autonomous vehicle, accordingly, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply Farabet's validate-then-incorporate sequence to the method of Frtunikj, in view of Zhdanov, with a reasonable expectation of success, by withholding the object detection model from the vehicle's driving software until the accuracy it attains on the annotated validation portion reaches the level the driving use demands, and then incorporating the validated model into that software so that the vehicle is thereafter driven on the object readings the model returns from the sensor data its sensors collect, to teach wherein, when the result indicates that the accuracy satisfies a recognition-accuracy requirement defined for the driving function, the trained primary model is validated for the driving function, and the driving function is enabled after the validation to use the trained primary model to generate the symbolic descriptions of the objects from the sensor data. This modification would have been motivated by the desire to widen the regions the vehicles running that software are able to navigate as each validated model is released to them (Farabet: [0041]).
Response to Arguments
Applicant's Remarks filed 6/24/2026, part II, directed to the rejection of claims 4 through 8 under 35 U.S.C. 112(b), are persuasive. Thus, the 35 U.S.C. 112(b) rejections set forth in the Office Action dated 3/25/2026 are hereby withdrawn.
Applicant's Remarks filed 6/24/2026, part III, directed to the rejection of claims 1-10 under 35 U.S.C. 101, are persuasive. Thus, the 35 U.S.C. 101 rejections set forth in the Office Action dated 3/25/2026 are hereby withdrawn.
Applicant's Remarks filed 6/24/2026, part IV, traversing the prior art rejections applied in the Office Action dated 3/25/2026 have been fully considered but are moot in view of the new grounds of rejection set forth above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights