Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made of the applicant's claim for Foreign priority to European Patent
Application No. 22216937 filed on December 28, 2022.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility Analysis
Step 1: Claims 1-14 are within the four statutory categories (a process, machine, manufacture or composition of matter).
Step 2A Prong One, Step 2A Prong Two, and Step 2B Analysis:
Step 2A Prong One asks if the claim recites a judicial exception (abstract idea, law of nature, or natural phenomenon). If the claim recites a judicial exception, analysis proceeds to Step 2A Prong Two, which asks if the claim recites additional elements that integrate the abstract idea into a practical application. If the claim does not integrate the judicial exception, analysis proceeds to Step 2B, which asks if the claim amounts to significantly more than the judicial exception. If the claim does not amount to significantly more than the judicial exception, the claim is not eligible subject matter under 35 U.S.C. 101.
None of the claims represent an improvement to technology.
Claims 1-8 are directed to a method consisting of a series of steps, meaning that it is directed to the statutory category of process. Claims 9-14 are directed to processing circuitry which are machines.
Regarding claim 1, the following claim elements are abstract ideas:
forming an input data set for the trained ensemble of ML algorithms by obtaining a synthetic training data set or by obtaining an unlabelled training data set, the input data set comprising one or more input data samples (This is an abstract idea of a mental process. It involves obtaining and organizing information for subsequent analysis. A person could gather unlabeled examples or generate hypothetical examples based on prior observations, historical information, or assumed conditions and compile those examples into a collection of input samples for evaluation. Such collection, generation and organization of information can be performed through human observation, judgement, and reasoning, either mentally or with the aid of pen and paper, and therefore falls within the mental process grouping of abstract ideas.);
generating, for each of the one or more input data samples, an ensemble prediction output (This is an abstract idea of a mental process. It involves evaluating information and generating a prediction based on prior observations, historical information, experience, or known outcomes. A person could review an input sample, consider multiple opinions, assessments, or prior experience relating to the sample, and determine a predicted outcome based on those considerations. Such evaluation, judgement, and decision-making can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.);
in response to a determined discrepancy in the ensemble prediction output for a particular input data sample: forming a second training data set based on that particular input data sample (This is an abstract idea of a mental process. It involves comparing multiple prediction results associated with a particular sample, identifying a difference or inconsistency among these results, and organizing additional information based on that determination for further evaluation. A person could review multiple opinions, assessments, or predicted outcomes relating to the same information, determine whether the outcomes agree or differ, and then compile additional examples related to the sample for subsequent review or analysis. Such comparison, evaluation, judgement, or organization of information can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
obtaining a trained ensemble of machine learning (ML) algorithms, comprising a plurality of ML algorithms that are trained at least partly based on a first set of training data (The step of “obtaining” a trained ensemble of machine learning algorithms is merely a generic data operation that amounts to storing and retrieving information in memory, which has been recognized by the courts as well-understood, routine, and conventional activity. Further, the recitation that the machine learning algorithms are trained at least in part based on a first training data amounts to insignificant extra-solution activity.);
providing the formed input data set to the trained ensemble of ML algorithms for (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
wherein the ensemble prediction output for each of the one or more input data samples comprises prediction outputs generated by each of the ML algorithms comprised in the ensemble of ML algorithms for that sample of the one or more input data samples (This limitation amounts adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the information generated and used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.);
updating the first training data set with the formed second training data set (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).).
Regarding claim 2, the rejection of claim 1 is incorporated herein. Further, claim 2 recites the following abstract ideas:
determining the discrepancy in the ensemble prediction output for a particular synthetic data sample by comparing, for each of the one or more synthetic data samples, the prediction output of each ML algorithm of the ensemble with the prediction output of each of a rest of the ML algorithms comprised in the ensemble (This is an abstract idea of a mental process. It involves comparing multiple prediction results associated with a particular sample and identifying differences or inconsistencies among those results. A person could review multiple opinions, assessments, or predicted outcomes relating to the same information and determine whether those outcomes agree or differ. Such comparison, evaluation, and judgement can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
in response to the determined discrepancy in the ensemble prediction output for that particular synthetic data sample: forming the second training data set comprising that particular synthetic data sample (This is an abstract idea of a mental process. It involves organizing information based on an identified discrepancy for further evaluation. A person could review differing prediction results associated with a particular sample, determine the results are inconsistent, and compile that sample into a collection of information for subsequent review or analysis. Such observation, evaluation, judgement, and organization of information can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
the one or more input data samples correspond to one or more synthetic data samples (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the type of information used in conjunction with the abstract idea and does not impose a meaningful limitation.);
wherein the synthetic training data set being artificially generated for constructing a scenario in a surrounding environment of a vehicle (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the source and content of the data used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.),
updating the first training data set with the formed second training data set (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).).
Regarding claim 3, the rejection of claim 1 is incorporated herein. Further, claim 3 recites the following abstract ideas:
determining the discrepancy in the ensemble prediction output for a particular unlabelled data sample by comparing, for each of the one or more unlabelled data samples, the prediction output of each ML algorithm of the ensemble with the prediction output of each of a rest of the other ML algorithms comprised in the ensemble (This is an abstract idea of a mental process. It involves comparing multiple prediction results associated with a particular unlabeled data sample and identifying differences or inconsistencies among those results. A person could review multiple opinions, assessments, or predicted outcomes relating to the same information and determine whether those outcomes agree or differ. Such comparison, evaluation, and judgement can be practically performed in the human or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
in response to the determined discrepancy in the ensemble prediction output for that particular unlabelled data sample: identifying one or more condition-specific parameters causing the determined discrepancy for that particular unlabelled data sample (This is an abstract idea of a mental process. It involves evaluating differing prediction results, identifying a discrepancy among those results, and determining one or more conditions that may have caused the discrepancy. A person could review multiple opinions, assessments, or predicted outcomes relating to the same information, recognize that the outcomes differ, and identify factors or conditions that contributed to the inconsistency. Such observation, evaluation, analysis, and judgement can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.);
generating a condition-specific synthetic training data set comprising one or more condition-specific synthetic data samples representative of the identified one or more condition specific parameters for that particular unlabelled data sample (This is an abstract idea of a mental process. It involves generating hypothetical examples based on identified conditions or factors and organizing those examples into a collection for further evaluation. A person could identify one or more conditions associated with a particular situation, create representative hypothetical examples into a set for subsequent review or analysis. Such observation, reasoning, judgement, and organization of information can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.);
forming the second training data set comprising the generated condition-specific synthetic training data set (This is an abstract idea of a mental process. It involves organizing information into a collection for further evaluation. A person could compile generated examples associated with identified conditions into a set of information for subsequent review and analysis. Such collection and organization can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
the one or more input data samples correspond to one or more unlabelled data samples (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the type of information used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.);
wherein the unlabelled training data set comprises information obtained at least partly from a sensor system of a vehicle and being a representative of a scenario in a surrounding environment of the vehicle (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the source and content of the information used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.);
updating the first training data set with the formed second training data set (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).).
Regarding claim 4, the rejection of claim 1 is incorporated herein. Further, claim 4 recites the following abstract ideas:
wherein the discrepancy in the ensemble prediction output for the particular input data sample is determined when the prediction output generated by at least one of the ML algorithms comprised in the ensemble is incompatible with the prediction outputs generated by one or more of the other ML algorithms of the ensemble for that particular input data sample (This is an abstract idea of a mental process. It involves comparing multiple prediction results associated with a particular input data sample and determining whether one or more of the prediction results are incompatible with one another. A person could review multiple predicted outcomes or assessments relating to the same information and determine whether the outcomes are consistent or inconsistent. Such comparison, evaluation, and judgement can practically be performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.).
Regarding claim 5, the rejection of claim 1 is incorporated herein. Further, claim 5 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
training the ensemble of ML algorithms by using the updated first set of training data (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).).
Regarding claim 6, the rejection of claim 1 is incorporated herein. Further, claim 6 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the input data set comprises information representative of a scenario in a surrounding environment of a vehicle; wherein the vehicle comprises an Automated Driving System (ADS) (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the environment and content of the information used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.).
Regarding claim 7, the rejection of claim 1 is incorporated herein. Further, claim 7 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
wherein the method is performed by a processing circuitry of a vehicle (The limitation merely amounts to instructions to apply the judicial exception using a generic computer component and does not impose a meaningful limitation on the judicial exception.).
Regarding claim 8, the rejection of claim 1 is incorporated herein. Further, claim 8 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors of an in-vehicle computer, causes the in-vehicle computer to carry out the method according to claim 1 (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).).
Regarding claim 1, the following claim elements are abstract ideas:
form an input data set for the trained ensemble of ML algorithms by obtaining a synthetic training data set or by obtaining an unlabelled training data set, the input data set comprising one or more input data samples (This is an abstract idea of a mental process. It involves obtaining and organizing information for subsequent analysis. A person could gather unlabeled examples or generate hypothetical examples based on prior observations, historical information, or assumed conditions and compile those examples into a collection of input samples for evaluation. Such collection, generation and organization of information can be performed through human observation, judgement, and reasoning, either mentally or with the aid of pen and paper, and therefore falls within the mental process grouping of abstract ideas.);
generating, for each of the one or more input data samples, an ensemble prediction output (This is an abstract idea of a mental process. It involves evaluating information and generating a prediction based on prior observations, historical information, experience, or known outcomes. A person could review an input sample, consider multiple opinions, assessments, or prior experience relating to the sample, and determine a predicted outcome based on those considerations. Such evaluation, judgement, and decision-making can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.);
in response to a determined discrepancy in the ensemble prediction output for a particular input data sample: form a second training data set based on that particular input data sample (This is an abstract idea of a mental process. It involves comparing multiple prediction results associated with a particular sample, identifying a difference or inconsistency among these results, and organizing additional information based on that determination for further evaluation. A person could review multiple opinions, assessments, or predicted outcomes relating to the same information, determine whether the outcomes agree or differ, and then compile additional examples related to the sample for subsequent review or analysis. Such comparison, evaluation, judgement, or organization of information can be practically performed in the human mind or with the aid of pen and paper and therefore falls within the mental process grouping of abstract ideas.); and
The following claim elements are additional elements which, taken alone or in combination with the other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
processing circuitry (This a high-level recitation of generic computer components for performing the abstract idea. See MPEP 2106.05(f).)
obtain a trained ensemble of machine learning (ML) algorithms, comprising a plurality of ML algorithms that are trained at least partly based on a first set of training data (The step of “obtaining” a trained ensemble of machine learning algorithms is merely a generic data operation that amounts to storing and retrieving information in memory, which has been recognized by the courts as well-understood, routine, and conventional activity. Further, the recitation that the machine learning algorithms are trained at least in part based on a first training data amounts to insignificant extra-solution activity.);
provide the formed input data set to the trained ensemble of ML algorithms for (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).)
wherein the ensemble prediction output for each of the one or more input data samples comprises prediction outputs generated by each of the ML algorithms comprised in the ensemble of ML algorithms for that sample of the one or more input data samples (This limitation amounts adding insignificant extra-solution activity to the judicial exception. The limitation merely describes the information generated and used in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.);
update the first training data set with the formed second training data set (This limitation constitutes mere instructions to apply the abstract idea and insignificant extra-solution activity. See MPEP 2106.05(f) and 2106.05(g).).
Regarding claim 10, the rejection of claim 9 is incorporated herein. The claim recites similar limitations corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 10.
Therefore, claim 10 is ineligible.
Regarding claim 11, the rejection of claim 9 is incorporated herein. The claim recites similar limitations corresponding to claim 3. Therefore, the same subject matter analysis that was utilized for claim 3, as described above, is equally applicable to claim 11.
Therefore, claim 11 is ineligible.
Regarding claim 12, the rejection of claim 9 is incorporated herein. The claim recites similar limitations corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 12.
Therefore, claim 12 is ineligible.
Regarding claim 13, the rejection of claim 9 is incorporated herein. The claim recites similar limitations corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 13.
Therefore, claim 13 is ineligible.
Regarding claim 14, the rejection of claim 9 is incorporated herein. Further, claim 14 recites the following additional elements, which taken alone or in combination with other elements, do not integrate the judicial exception into a practical application nor amount to significantly more than the judicial exception:
one or more vehicle-mounted sensors configured to monitor a surrounding environment of the vehicle (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely gathers information for use in conjunction with the abstract idea and does not impose a meaningful limitation.);
a localization system configured to monitor a geographical position of the vehicle (This limitation amounts to adding insignificant extra-solution activity to the judicial exception. The limitation merely gathers information for use in conjunction with the abstract idea and does not impose a meaningful limitation on the judicial exception.);
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14 are rejected under the 35 U.S.C. 103 as being unpatentable over Goldenberg et al., (Pat. No.: US 11526693 B1 (Filed: 2020)) in view of Ahuja et al., (Pub. No.: US 20200226430 A1 (Filed: 2020)).
Regarding claim 1, Goldenberg teaches the following limitations:
obtaining a trained ensemble of machine learning (ML) algorithms, comprising a plurality of ML algorithms that are trained at least partly based on a first set of training data (Goldenberg, [col. 2, lines 24-41] “ML models of the ensemble may be trained sequentially, with each new model trained to disregard input features learned by previously trained ML models of the ensemble and learn based on other features included in the training data samples…the first ML model of the ensemble may be trained on the training data samples without any feature restrictions… The second ML model may then be trained on the training data samples and the distilled data samples… ensemble E=(F,M) be defined by a collection of n models F={f.sub.1, f.sub.2, . . . , f.sub.n}” – teaches obtaining a trained ensemble of machine learning algorithms by defining an ensemble as a collection of multiple machine learning models. Goldenberg further teaches training machine learning models of the ensemble using training data samples, including a first machine learning model trained on training data samples and second machine learning model trained on data samples and distilled data samples. Thus, Goldenberg teaches a plurality of machine learning algorithms that are trained at least partly based on training data, corresponding to the claimed trained ensemble of machine learning algorithms.).
forming an input data set for the trained ensemble of ML algorithms by obtaining a synthetic training data set or by obtaining an unlabelled training data set, the input data set comprising one or more input data samples (Goldenberg, [col. 2, lines 30-34] “The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples” – teaches forming an input data set for the trained ensemble. Specifically, Goldenberg teaches determining learned features from a machine learning model and encoding those features into distilled data samples. Goldenberg further teaches training a subsequent machine learning model using the training data samples and the distilled data samples. Under the broadest reasonable interpretation, the distilled data samples constitute synthetic training data set because the distilled data samples are artificially generated from learned features rather than obtained directly from real-world observations. Accordingly, Goldenberg teaches forming an input data set comprising one or more input data samples by obtaining a synthetic training data set.);
providing the formed input data set to the trained ensemble of ML algorithms for generating, for each of the one or more input data samples, an ensemble prediction output (Goldenberg, [col. 2, lines 41-46] “collection of n models F={f.sub.1, f.sub.2, . . . , f.sub.n} and a combining function M For an input data sample x, each ML model of the ensemble produces a K-length probability vector f.sub.i(x)=(f.sub.i.sup.1(x), f.sub.i.sup.2(x), . . . , f.sub.i.sup.K(x)). A combining function M (f.sub.1(x), f.sub.2(x), . . . , f.sub.n(x)) merges individual model outputs to produce the final ensemble result.” [col. 2, lines 57-67] “ an ensemble 102 of machine learning models 102-1, 102-2, 102-3, through 102-N that determines a class for an in-distribution data sample 100… the ML models 102-1 through 102-N of an ensemble 102 are trained to each output a probability vector 103-1, 103-2, 103-3, through 103-N, respectively, indicating the probability of an input data sample 100 belonging to one of K known classes” – Goldenberg teaches providing an input data sample to a trained ensemble of machine learning models. Specifically, Goldenberg teaches that each ML model of the ensemble produces a probability vector for an input data sample and that a combining function merges the individual model outputs to produce the final ensemble result. Thus, Goldenberg teaches providing the formed input data set to the trained ensemble of ML algorithms for generating an ensemble prediction output for each input data sample.);
wherein the ensemble prediction output for each of the one or more input data samples comprises prediction outputs generated by each of the ML algorithms comprised in the ensemble of ML algorithms for that sample of the one or more input data samples (Goldenberg, [col. 3, lines 4-13] “when an in-distribution data sample 100 is provided to a trained ensemble 102 of ML models 102-1 through 102-N, each model generates a probability vector f (x) that assigns a probability to each of K classes… Each of the ML models 102-1 through 102-N generate respective probability vectors f.sub.1(x) 103-1, f.sub.2(x) 103-2, f.sub.3(x) 103-3 through f.sub.N(x) 103-N that assign probability scores to each class for the input data sample.” [col. 3, lines 46-51] "the combining function 106 of the ensemble 102 uses the results 104 from each ML model of the ensemble to determine an ensemble results 108, in this example an ensemble result indicating that the ensemble determined that the input data sample 100 corresponds to class three CL3 105-3.” – Goldenberg teaches that an input data sample is provided to a trained ensemble of machine learning models, wherein each machine learning model generates a respective probability vector for the input data sample. Goldenberg further teaches that the ensemble uses the results from each machine learning model to determine an ensemble result. Thus, the ensemble prediction output comprises prediction outputs generated by each of the machine learning models comprised in the ensemble for the input data sample, corresponding to the claimed ensemble prediction output.);
However, Goldenberg does not teach but Goldenberg in view of Ahuja teaches the following limitations:
in response to a determined discrepancy in the ensemble prediction output for a particular input data sample: forming a second training data set based on that particular input data sample(Goldenberg, [col. 4, lines 33-51] “the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200…the combining function 106 of the ensemble 102 uses the results 204 from each ML model of the ensemble to determine an ensemble results 208, in this example an ensemble result indicating that the ensemble determined that the input data sample 200 is out-of distribution… the ensemble may determine that the input is out-of distribution because there is no majority agreement as to class for the input data sample.” Ahuja, paragraph [0074] “ Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0077] “Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” – Goldenberg teaches determining discrepancy in ensemble prediction outputs by identifying input data samples for which the machine learning models do not agree on a class assignment and determining that such samples are out-of-distribution. Ahuja teaches identifying uncertain or out-of-distribution data samples, generating labelled data from the identified samples, and utilizing the identified samples within a continuous learning framework for subsequent training. Under the broadest reasonable interpretation, the uncertain or out-of-distribution data samples identified by Ahuja correspond to a particular synthetic data sample associated with determining discrepancy identified by Goldenberg, and the labeled data generated from the identified sample corresponds to forming a second training data set comprising that particular synthetic data sample.);
updating the first training data set with the formed second training data set (Ahuja, paragraph [0074] “Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0077] “Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “ Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0095] “Newly annotated data 410 may be used to retrain Probabilistic DNN model 420.” – Ahuja teaches identifying uncertain or out-of-distribution data samples, annotating or labeling those samples, and utilizing the resulting annotated data for retraining a probabilistic model within a continuous framework. Under BRI, the newly annotated data corresponds to the claimed second training data set. Further, utilizing the newly annotated data for retraining corresponds to incorporating the newly annotated data into the existing training corpus. Accordingly, Ahuja teaches updating the first training data set with the formed second training data set.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Goldenberg and Ahuja before them, to incorporate the continuous learning framework of Ahuja into the ensemble learning framework of Goldenberg such that uncertain or out-of-distribution data samples identified through ensemble disagreement are annotated and incorporated into subsequent training data. One would have been motivated to make such a combination in order to improve machine learning model performance by expanding the available training data with newly identified uncertain samples. This would allow more robust and accurate model predictions by enabling the machine learning models to learn from previously unseen or difficult to classify data samples.
Regarding claim 2, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
wherein for the input data set formed based on the obtained synthetic training data set: the one or more input data samples correspond to one or more synthetic data samples (Goldenberg, [col. 2, lines 30-34] “The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples…” [col. 10, lines 48-59] “the process may start with a blank or empty distilled image 802, in this example a blank image (blank distilled data sample). The blank distilled image 802 is then iteratively modified…the final distilled image 804 (final distilled embedding vector) includes the features of the selected input image 800 (input data sample) that were learned by the ML model to identify the selected input image 800 (input data sample).” – Goldenberg teaches encoding learned features into distilled data samples and training machine learning models using the distilled data samples. Goldenberg further teaches generating a blank distilled data sample and iteratively modifying the blank distilled data sample to produce a final distilled image including features of a selected input image. Under BRI, the distilled data samples constitute synthetic data samples because they are artificially generated through an iterative modification process rather than directly obtained from real-world observations. Accordingly, Goldenberg teaches that the one or more input data samples correspond to one or more synthetic data samples.),
wherein the synthetic training data set being artificially generated for constructing a scenario in a surrounding environment of a vehicle, wherein the method further comprises: determining the discrepancy in the ensemble prediction output for a particular synthetic data sample by comparing, for each of the one or more synthetic data samples, the prediction output of each ML algorithm of the ensemble with the prediction output of each of a rest of the ML algorithms comprised in the ensemble (Ahuja, paragraph [0071] “Identifying a data shift scene (e.g. adverse weather conditions including fog/snow/frost) or out-of-distribution data (i.e. unseen object class during model training, new geological location) demonstrates the need for updating the visual perception system with a continuous learning framework model.” [0114] “The methods and devices may also implement training schemes for training probabilistic DNN model. FIG. 5A shows an exemplary representation 500A of how a probabilistic model may be trained. Detection system 501 may detect or perceive an vehicle environment 502A. Detection system 501 may employ data acquisitions devices such 112 to perceive vehicle environment 502A. Detection system 501 may use a trained deep neural network to predict objects in the perceived vehicle environment 502A. Data associated with perceived vehicle environment 502A may be annotated to correctly identify the objects in the environment.” Goldenberg, [col. 10, lines 48-55] “ the process may start with a blank or empty distilled image 802, in this example a blank image (blank distilled data sample). The blank distilled image 802 is then iteratively modified, with each iteration moving the distilled image embedding vector (distilled data sample embedding vector) closer to the selected input image embedding vector (input data sample embedding vector) in the ML models embedding space.” [col. 3, lines 9-10] “Each of the ML models 102-1 through 102-N generate respective probability vectors… with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models agree on the correct class but determine different probability values for the other classes of the set of classes (i.e., present disagreement for incorrect classes).” [col. 4, lines 31-51] “ with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200…. In other examples, the ensemble may determine that the input is out-of distribution because there is no majority agreement as to class for the input data sample.” – Ahuja teaches machine-learning systems that process data representing scenarios occurring in a surrounding environment of a vehicle, including data-shift scenes and out-of-distribution conditions within a vehicle environment. Goldenberg teaches generating distilled data samples by starting with a blank distilled sample and iteratively modifying the distilled data sample to produce a final distilled data sample. Under BRI, the distilled data samples correspond to the claimed synthetic data samples. Goldenberg further teaches that each machine learning model of an ensemble generates a respective prediction output and that discrepancies are determined by comparing prediction outputs generated by the machine learning models, including identifying disagreement among the machine learning models and determining that no majority agreement exists for a particular input data sample. Accordingly, the combination of Ahuja and Goldenberg teaches synthetic data samples representing scenarios in a surrounding environment of a vehicle and determining discrepancies by comparing the prediction outputs generated by machine learning models of an ensemble, as claimed.); and
in response to the determined discrepancy in the ensemble prediction output for that particular synthetic data sample: forming the second training data set comprising that particular synthetic data sample; and updating the first training data set with the formed second training data set (Goldenberg, [col. 4, lines 33-51] “ the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200… the combining function 106 of the ensemble 102 uses the results 204 from each ML model of the ensemble to determine an ensemble results 208, in this example an ensemble result indicating that the ensemble determined that the input data sample 200 is out-of distribution…the ensemble may determine that the input is out-of distribution because there is no majority agreement as to class for the input data sample.” [col. 10, lines 48-51] “ the process may start with a blank or empty distilled image 802, in this example a blank image (blank distilled data sample). The blank distilled image 802 is then iteratively modified,” Ahuja, paragraph [0074] “Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0077] “ Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0095] “Newly annotated data 410 may be used to retrain Probabilistic DNN model 420.” – Goldenberg teaches determining a discrepancy in ensemble prediction outputs by identifying input data samples from which the machine learning models do not agree on a class assignment and determining that such samples are out-of-distribution. Goldenberg further teaches generating distilled data samples by starting with a blank distilled data sample and iteratively modifying the distilled data sample. Under BRI, the distilled data samples correspond to the claimed synthetic data samples. Ahuja teaches identifying uncertain or out-of-distribution data samples, generating annotated data from the identified samples, and utilizing the resulting data within a continuous framework for retraining. Under BRI, the annotated data corresponds to the claimed second training data set, and utilizing the annotated data for retraining corresponds to updating the first training data set with the formed second training data set. Accordingly, the combination of Goldenberg and Ahuja teaches forming a second training data set comprising the particular synthetic data sample in response to a determined discrepancy and updating the first training data set with the formed second training data set.).
Regarding claim 3, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
wherein for the input data set formed based on the obtained unlabelled training data set: the one or more input data samples correspond to one or more unlabelled data samples (Ahuja, paragraph [0077] “ Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0074] “Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” – Ahuja teaches obtaining newly observed data associated with uncertainty and utilizing the newly observed data within an active learning framework. Ahuja further teaches subsequently annotating the uncertain data and retraining the probabilistic model using the resulting labeled data. Under BRI, the newly observed uncertain data corresponds to unlabeled data samples because the data is identified and processed prior to annotation or labeling. Accordingly, Ahuja teaches an input data set comprising one or more unlabeled data samples.);
wherein the unlabelled training data set comprises information obtained at least partly from a sensor system of a vehicle and being a representative of a scenario in a surrounding environment of the vehicle (Ahuja, paragraph [0041] “The control system 200 may include… one or more data acquisition devices 112, one or more position sensors 114 such as a Global Navigation Satellite System (GNSS) and/or a Global Positioning System (GPS), and one or more measurement sensors 116, e.g. speedometer, altimeter, gyroscope, velocity sensors, etc.” [0042] “The control system 200 may be configured to control the vehicle's 100 mobility via mobility system 120 and/or interactions with its environment” [0114] “Detection system 501 may detect or perceive an vehicle environment 502A. Detection system 501 may employ data acquisitions devices such 112 to perceive vehicle environment 502A…Data associated with perceived vehicle environment 502A may be annotated to correctly identify the objects in the environment.” [0077] “Newly observed data may yield predictions associated with high uncertainty.” – Ahuja teaches a vehicle control system including data acquisition devices, position sensors, and measurement sensors that obtain information associated with vehicle operation and the vehicle environment. Ahuja further teaches detecting and perceiving a vehicle environment and utilizing newly observed data associated with the perceived environment. Under BRI, the newly observed data corresponds to an unlabeled training data set because the data is observed prior to annotation, and the information obtained from the data acquisition devices, position sensors, and measurement sensors corresponds to information obtained at least partly from a sensor system of a vehicle that is representative of a scenario in a surrounding environment of the vehicle.);
wherein the method further comprises: determining the discrepancy in the ensemble prediction output for a particular unlabelled data sample by comparing, for each of the one or more unlabelled data samples, the prediction output of each ML algorithm of the ensemble with the prediction output of each of a rest of the other ML algorithms comprised in the ensemble (Ahuja, paragraph [0136] “ For example, there is an object in the image that was never observed during training and therefore is incorrectly predicted by the perception system. Image 1006 represents an image of a vehicle environment. Image 1008 represents a prediction of detected objects in image 1006 based on labels 1002. Because a suitcase was not included in the images used to train the perception system, there is no label for suitcase, and its prediction is incorrect 1012. Uncertainty indicator 1004 shows that dark pixels are associated with high certainty and light pixels are associated with high uncertainty. As shown in image 1010, the pixels associated with the suitcase in image 1006 are very light and therefore the prediction of the suitcase as a person is highly uncertain 1014.” [0138] “ Comparing the uncertainty metric with the uncertainty metric threshold 1106. Determining an annotation method based on the comparison of the uncertainty metric and the uncertainty metric threshold 1108. Collecting data associated with the object 1110. Annotating the data to more accurately identify the object,” Goldenberg, [col. 3, lines 9-10] “Each of the ML models 102-1 through 102-N generate respective probability vectors…” [col. 3, lines 41-45] “ with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models agree on the correct class but determine different probability values for the other classes of the set of classes (i.e., present disagreement for incorrect classes).” [col. 4, lines 33-51] “with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200…. In other examples, the ensemble may determine that the input is out-of distribution because there is no majority agreement as to class for the input data sample.” – Ahuja teaches data associated with an object for which no label exists in the training data and further teaches collecting the data and subsequently annotating the data. Under BRI, the collected data corresponds to an unlabeled data sample because the data lacks a corresponding label prior to annotation. Goldenberg teaches that each machine learning model of an ensemble generates a respective prediction output and that discrepancies are determined by comparing prediction outputs generated by the machine learning models, including identifying disagreement among the machine learning models and determining no majority agreement exists for a particular input data sample. Accordingly, the combination of Ahuja and Goldenberg teaches determining a discrepancy in ensemble prediction outputs for a particular unlabeled data sample by comparing prediction outputs generated by each machine learning model of the ensemble with prediction outputs generated by the other machine learning models of the ensemble.); and
in response to the determined discrepancy in the ensemble prediction output for that particular unlabelled data sample: identifying one or more condition-specific parameters causing the determined discrepancy for that particular unlabelled data sample (Ahuja, paragraph [0071] “Identifying a data shift scene (e.g. adverse weather conditions including fog/snow/frost) or out-of-distribution data (i.e. unseen object class during model training, new geological location)” [0135] “When comparing the uncertainty map of row 901 with that of rows 902 and 903, the uncertainty map of row 901 is much darker than that of row 902 and 903 and therefore more certain.” [0136] “Because a suitcase was not included in the images used to train the perception system, there is no label for suitcase, and its prediction is incorrect 1012. Uncertainty indicator 1004 shows that dark pixels are associated with high certainty and light pixels are associated with high uncertainty. As shown in image 1010, the pixels associated with the suitcase in image 1006 are very light and therefore the prediction of the suitcase as a person is highly uncertain 1014.” – Ahuja teaches identifying conditions associated with uncertain or incorrect predictions, including adverse weather conditions such as fog, snow, and frost, unseen object classes, new geographical locations, and objects lacking corresponding training labels. Ahuja further teaches determining uncertainty associated with such conditions and identifying data-shift and out-of-distribution scenarios that cause prediction errors. Under BRI, the identified weather conditions, object classes, geographical locations, and missing-label conditions correspond to condition-specific parameters causing the determined discrepancy for the particular unlabeled data sample. Accordingly, Ahuja teaches identifying one or more condition-specific parameters causing the determined discrepancy for the particular unlabeled data sample.);
generating a condition-specific synthetic training data set comprising one or more condition-specific synthetic data samples representative of the identified one or more condition-specific parameters for that particular unlabelled data sample (Ahuja, paragraph [0071] “Identifying a data shift scene (e.g. adverse weather conditions including fog/snow/frost) or out-of-distribution data (i.e. unseen object class during model training, new geological location)” [0136] “Because a suitcase was not included in the images used to train the perception system, there is no label for suitcase, and its prediction is incorrect 1012.” Goldenberg, [col. 2, lines 28-34] “For example, the first ML model of the ensemble may be trained on the training data samples without any feature restrictions. The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples.” [col. 10, lines 57-67] “ As discussed, the process may start with a blank or empty distilled image 802, in this example a blank image (blank distilled data sample). The blank distilled image 802 is then iteratively modified…Upon completion of the iteration, the final distilled image 804 (final distilled embedding vector) includes the features of the selected input image 800” – Ahuja teaches identifying conditions associated with uncertain or incorrect predictions, including adverse weather conditions, unseen object classes, and other out-of-distribution conditions. Goldenberg teaches generating distilled data samples by encoding learned features into distilled data samples and by iteratively modifying a blank distilled data sample to produce a final distilled data sample representative of selected input data. Under BRI, the distilled data samples correspond to synthetic data samples because they are artificially generated rather than directly obtained from real-world observations. Further, the generated distilled data samples represent characteristics of the identified input data. Accordingly, the combination of Ahuja and Goldenberg teaches generating a condition-specific synthetic training data set comprising one or more condition-specific synthetic data samples representative of identified condition-specific parameters for a particular unlabeled data sample.);
forming the second training data set comprising the generated condition-specific synthetic training data set; and updating the first training data set with the formed second training data set (Goldenberg, [col. 2, lines 28-34] “For example, the first ML model of the ensemble may be trained on the training data samples without any feature restrictions. The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples” [col. 10, lines 48-51] “ the process may start with a blank or empty distilled image 802, in this example a blank image (blank distilled data sample). The blank distilled image 802 is then iteratively modified, “ Ahuja, paragraph [0074] “ Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0095] “ Newly annotated data 410 may be used to retrain Probabilistic DNN model 420.” – Goldenberg teaches generating distilled data samples and utilizing the distilled data samples together with training data samples for subsequent machine learning training. Under BRI, the distilled data samples correspond to the claimed condition-specific synthetic training data. Ahuja teaches utilizing newly generated annotated data within a continuous learning framework and retraining a probabilistic model using the newly generated data. Under BRI, utilizing newly generated data for retraining corresponds to forming a second training data set comprising the generated condition-specific synthetic training data set and updating the first training data set with the formed second training data set. Accordingly, the combination of Goldenberg and Ahuja teaches the claimed limitation.).
Regarding claim 4, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
wherein the discrepancy in the ensemble prediction output for the particular input data sample is determined when the prediction output generated by at least one of the ML algorithms comprised in the ensemble is incompatible with the prediction outputs generated by one or more of the other ML algorithms of the ensemble for that particular input data sample (col. 3, lines 9-45] “ Each of the ML models 102-1 through 102-N generate respective probability vectors… with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models agree on the correct class but determine different probability values for the other classes of the set of classes (i.e., present disagreement for incorrect classes).” [col. 4, lines 33-51] “ the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200… Each of the ML models 102-1 through 102-N generate respective probability vectors f.sub.1(x) 103-1, f.sub.2(x) 103-2, f.sub.3(x) 103-3 through f.sub.N(x) 103-N that assign probability scores to each class for the input data sample. As illustrated, in this example, because the input data sample is an in-distribution data sample that corresponds to class three, each ML model of the ensemble 102 assigns a high probability score to CL3 105-3 in the respective probability vectors 103-1 through 103-N. However, because the models are trained with the disclosed implementations to be de-correlated for incorrect classes… the second ML model 103-2 of the ensemble assigned the following probabilities of 0.1 to class CL1 105-1, 0.0 to class CL2 105-2, 0.6 to class CL3 105-3, 0.2 to class CL4 105-4, 0.0 to class CL5 105-5, and 0.1 to class CLK 105-K. The third ML model 103-3 of the ensemble 102 assigned the following probabilities of 0.0 to class CL1 105-1, 0.2 to class CL2 105-2, 0.7 to class CL3 105-3, 0.0 to class CL4 105-4, 0.1 to class CL5 105-5, and 0.0 to class CLK 105-K. The Nth ML model 103-N of the ensemble 102 assigned the following probabilities… with the ML models of the ensemble trained in accordance with the disclosed implementations, the ML models agree on the correct class but determine different probability values for the other classes of the set of classes (i.e., present disagreement for incorrect classes” – Goldenberg teaches that each machine learning model of an ensemble generates a respective prediction output for an input data sample. Goldenberg further teaches that different machine learning models generate different probability values for the same input data sample and expressly teaches that the machine learning models “present disagreement for incorrect classes.” Under BRI, the disagreement among the prediction outputs generated by the machine learning models correspond to a prediction output generated by at least one machine learning model being incompatible with prediction outputs generated by one or more machine learning models of the ensemble.).
Regarding claim 5, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
training the ensemble of ML algorithms by using the updated first set of training data (Goldenberg, [col. 2, lines 24-34] “ ML models of the ensemble may be trained sequentially, with each new model trained to disregard input features learned by previously trained ML models of the ensemble and learn based on other features included in the training data samples. For example, the first ML model of the ensemble may be trained on the training data samples without any feature restrictions. The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples” [col. 2, lines 39-42] In the disclosed implementations we let ensemble E=(F,M) be defined by a collection of n models F={f.sub.1, f.sub.2, . . . , f.sub.n} and a combining function M For an input data sample x” Ahuja, paragraph [0074] “Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0095] “Newly annotated data 410 may be used to retrain Probabilistic DNN model 420.” – Goldenberg teaches an ensemble comprising a collection of machine learning models and further teaches training the machine learning models of the ensemble using training data samples and distilled data samples. Ahuja teaches retraining machine learning models using newly identified and annotated data within a continuous framework. Under BRI, retraining machine learning models using newly identified and annotated data corresponds to training the ensemble of machine learning models of Goldenberg using the updated first training data set. Accordingly, the combination of Goldenberg and Ahuja teaches training the ensemble of machine learning algorithms by using the updated first set of training data.).
Regarding claim 6, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
wherein the input data set comprises information representative of a scenario in a surrounding environment of a vehicle; wherein the vehicle comprises an Automated Driving System (ADS) (Ahuja, paragraph [0105] “a probabilistic deep neural network may be trained for individual users of an autonomous vehicle. In this way, driving habits of individuals may be taken into account when training the probabilistic model. For example, driver A may drive in scenarios where pedestrians occur more frequently than driver B.” [0107] “An anomalous situation which may be used trigger the autonomous vehicle to shift into a more cautious/conservative mode of operation.” [0108] “A trigger to update the probabilistic model with the new contextual information encountered.” [0114] “Detection system 501 may employ data acquisitions devices such 112 to perceive vehicle environment 502A. Detection system 501 may use a trained deep neural network to predict objects in the perceived vehicle environment 502A. Data associated with perceived vehicle environment 502A may be annotated to correctly identify the objects in the environment.”).
Regarding claim 7, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
wherein the method is performed by a processing circuitry of a vehicle (Ahuja, paragraph [0181] “a device for a vehicle to incorporate annotated data, the device comprising one or more processors configured to: obtain an uncertainty metric of an object, may include that the uncertainty metric describes a probability that the object occurs; determine an uncertainty metric threshold based on a previously trained model of the continuous learning device; compare the uncertainty metric with the uncertainty metric threshold; determine an annotation method based on the comparison of the uncertainty metric and the uncertainty metric threshold; collect data associated with the object; annotate the data to more accurately identify of the object, according to the determined annotation method; and retrain the model of the continuous learning device with the annotated data.”).
Regarding claim 8, Goldenberg in view of Ahuja teaches all the elements of claim 1, therefore is rejected for the same reasons as those presented for claim 1. Goldenberg in view of Ahuja further teaches:
A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors of an in-vehicle computer, causes the in-vehicle computer to carry out the method according to claim 1 (Ahuja, pargraph [0178] “In Example 35, one or more non-transitory computer readable media comprising programmable instructions thereon, that when executed by one or more processors of a device, cause the device to perform any one of the method of Examples 20-35.” [0181] “ a device for a vehicle to incorporate annotated data, the device comprising one or more processors configured to”).
Regarding claim 9, Goldenberg teaches the following limitations:
A system comprising processing circuitry configured to (Goldenberg, [col. 12, lines 18-21] “Each of these server(s) 1120 may include one or more controllers/processors 1104, that may each include a central processing unit (CPU) for processing data and computer-readable instructions”):
obtain a trained ensemble of machine learning (ML) algorithms, comprising a plurality of ML algorithms that are trained at least partly based on a first set of training data (Goldenberg, [col. 2, lines 24-41] “ML models of the ensemble may be trained sequentially, with each new model trained to disregard input features learned by previously trained ML models of the ensemble and learn based on other features included in the training data samples…the first ML model of the ensemble may be trained on the training data samples without any feature restrictions… The second ML model may then be trained on the training data samples and the distilled data samples… ensemble E=(F,M) be defined by a collection of n models F={f.sub.1, f.sub.2, . . . , f.sub.n}” – teaches obtaining a trained ensemble of machine learning algorithms by defining an ensemble as a collection of multiple machine learning models. Goldenberg further teaches training machine learning models of the ensemble using training data samples, including a first machine learning model trained on training data samples and second machine learning model trained on data samples and distilled data samples. Thus, Goldenberg teaches a plurality of machine learning algorithms that are trained at least partly based on training data, corresponding to the claimed trained ensemble of machine learning algorithms.).
form an input data set for the trained ensemble of ML algorithms by obtaining a synthetic training data set or by obtaining an unlabelled training data set, the input data set comprising one or more input data samples (Goldenberg, [col. 2, lines 30-34] “The features learned by the first ML model may then be determined and those features, referred to herein as distilled features, encoded into distilled data samples. The second ML model may then be trained on the training data samples and the distilled data samples” – teaches forming an input data set for the trained ensemble. Specifically, Goldenberg teaches determining learned features from a machine learning model and encoding those features into distilled data samples. Goldenberg further teaches training a subsequent machine learning model using the training data samples and the distilled data samples. Under the broadest reasonable interpretation, the distilled data samples constitute synthetic training data set because the distilled data samples are artificially generated from learned features rather than obtained directly from real-world observations. Accordingly, Goldenberg teaches forming an input data set comprising one or more input data samples by obtaining a synthetic training data set.);
provide the formed input data set to the trained ensemble of ML algorithms for generating, for each of the one or more input data samples, an ensemble prediction output (Goldenberg, [col. 2, lines 41-46] “collection of n models F={f.sub.1, f.sub.2, . . . , f.sub.n} and a combining function M For an input data sample x, each ML model of the ensemble produces a K-length probability vector f.sub.i(x)=(f.sub.i.sup.1(x), f.sub.i.sup.2(x), . . . , f.sub.i.sup.K(x)). A combining function M (f.sub.1(x), f.sub.2(x), . . . , f.sub.n(x)) merges individual model outputs to produce the final ensemble result.” [col. 2, lines 57-67] “ an ensemble 102 of machine learning models 102-1, 102-2, 102-3, through 102-N that determines a class for an in-distribution data sample 100… the ML models 102-1 through 102-N of an ensemble 102 are trained to each output a probability vector 103-1, 103-2, 103-3, through 103-N, respectively, indicating the probability of an input data sample 100 belonging to one of K known classes” – Goldenberg teaches providing an input data sample to a trained ensemble of machine learning models. Specifically, Goldenberg teaches that each ML model of the ensemble produces a probability vector for an input data sample and that a combining function merges the individual model outputs to produce the final ensemble result. Thus, Goldenberg teaches providing the formed input data set to the trained ensemble of ML algorithms for generating an ensemble prediction output for each input data sample.);
wherein the ensemble prediction output for each of the one or more input data samples comprises prediction outputs generated by each of the ML algorithms comprised in the ensemble of ML algorithms for that sample of the one or more input data samples (Goldenberg, [col. 3, lines 4-13] “when an in-distribution data sample 100 is provided to a trained ensemble 102 of ML models 102-1 through 102-N, each model generates a probability vector f (x) that assigns a probability to each of K classes… Each of the ML models 102-1 through 102-N generate respective probability vectors f.sub.1(x) 103-1, f.sub.2(x) 103-2, f.sub.3(x) 103-3 through f.sub.N(x) 103-N that assign probability scores to each class for the input data sample.” [col. 3, lines 46-51] "the combining function 106 of the ensemble 102 uses the results 104 from each ML model of the ensemble to determine an ensemble results 108, in this example an ensemble result indicating that the ensemble determined that the input data sample 100 corresponds to class three CL3 105-3.” – Goldenberg teaches that an input data sample is provided to a trained ensemble of machine learning models, wherein each machine learning model generates a respective probability vector for the input data sample. Goldenberg further teaches that the ensemble uses the results from each machine learning model to determine an ensemble result. Thus, the ensemble prediction output comprises prediction outputs generated by each of the machine learning models comprised in the ensemble for the input data sample, corresponding to the claimed ensemble prediction output.);
However, Goldenberg does not teach but Goldenberg in view of Ahuja teaches the following limitations:
in response to a determined discrepancy in the ensemble prediction output for a particular input data sample: form a second training data set based on that particular input data sample(Goldenberg, [col. 4, lines 33-51] “the ML models do not agree on any class for out-of distribution input data samples, such as input data sample 200…the combining function 106 of the ensemble 102 uses the results 204 from each ML model of the ensemble to determine an ensemble results 208, in this example an ensemble result indicating that the ensemble determined that the input data sample 200 is out-of distribution… the ensemble may determine that the input is out-of distribution because there is no majority agreement as to class for the input data sample.” Ahuja, paragraph [0074] “ Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0077] “Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” – Goldenberg teaches determining discrepancy in ensemble prediction outputs by identifying input data samples for which the machine learning models do not agree on a class assignment and determining that such samples are out-of-distribution. Ahuja teaches identifying uncertain or out-of-distribution data samples, generating labelled data from the identified samples, and utilizing the identified samples within a continuous learning framework for subsequent training. Under the broadest reasonable interpretation, the uncertain or out-of-distribution data samples identified by Ahuja correspond to a particular synthetic data sample associated with determining discrepancy identified by Goldenberg, and the labeled data generated from the identified sample corresponds to forming a second training data set comprising that particular synthetic data sample.);
update the first training data set with the formed second training data set (Ahuja, paragraph [0074] “Once identifying data for which the probabilistic DNN is uncertain, the ADAS can employ a continuous learning framework to identify dataset shift and out-of-distribution data and retrain the probabilistic DNN with correctly identified or labeled data.” [0077] “Newly observed data may yield predictions associated with high uncertainty. The newly observed data may be used within an active learning framework to update the probabilistic model.” [0088] “ Data associated with objects that are uncertain may be annotated and used to retrain the probabilistic model so that it may “learn” about objects that had not been observed during previous trainings of the DNN.” [0095] “Newly annotated data 410 may be used to retrain Probabilistic DNN model 420.” – Ahuja teaches identifying uncertain or out-of-distribution data samples, annotating or labeling those samples, and utilizing the resulting annotated data for retraining a probabilistic model within a continuous framework. Under BRI, the newly annotated data corresponds to the claimed second training data set. Further, utilizing the newly annotated data for retraining corresponds to incorporating the newly annotated data into the existing training corpus. Accordingly, Ahuja teaches updating the first training data set with the formed second training data set.)
Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, having a combination of Goldenberg and Ahuja before them, to incorporate the continuous learning framework of Ahuja into the ensemble learning framework of Goldenberg such that uncertain or out-of-distribution data samples identified through ensemble disagreement are annotated and incorporated into subsequent training data. One would have been motivated to make such a combination in order to improve machine learning model performance by expanding the available training data with newly identified uncertain samples. This would allow more robust and accurate model predictions by enabling the machine learning models to learn from previously unseen or difficult to classify data samples.
Regarding claim 10, Goldenberg in view of Ahuja teaches all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9. The claim recites similar limitations corresponding to claim 2 and is rejected for similar reasons as claim 2 using similar teachings and rationale.
Regarding claim 11, Goldenberg in view of Ahuja teaches all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9. The claim recites similar limitations corresponding to claim 3 and is rejected for similar reasons as claim 3 using similar teachings and rationale.
Regarding claim 12, Goldenberg in view of Ahuja teaches all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9. The claim recites similar limitations corresponding to claim 4 and is rejected for similar reasons as claim 4 using similar teachings and rationale.
Regarding claim 13, Goldenberg in view of Ahuja teaches all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9. The claim recites similar limitations corresponding to claim 5 and is rejected for similar reasons as claim 5 using similar teachings and rationale.
Regarding claim 14, Goldenberg in view of Ahuja teaches all the elements of claim 9, therefore is rejected for the same reasons as those presented for claim 9. Goldenberg in view of Ahuja further teaches:
A vehicle comprising: one or more vehicle-mounted sensors configured to monitor a surrounding environment of the vehicle; a localization system configured to monitor a geographical position of the vehicle (Ahuja, paragraph [0041] “the control system 200 may include one or more processors 102, one or more memories 104, an antenna system 106 which may include one or more antenna arrays at different locations on the vehicle for radio frequency (RF) coverage, one or more radio frequency (RF) transceivers 108, one or more data acquisition devices 112, one or more position sensors 114 such as a Global Navigation Satellite System (GNSS) and/or a Global Positioning System (GPS), and one or more measurement sensors 116, e.g. speedometer, altimeter, gyroscope, velocity sensors, etc.” [0114] “Detection system 501 may detect or perceive an vehicle environment 502A. Detection system 501 may employ data acquisitions devices such 112 to perceive vehicle environment 502A. Detection system 501 may use a trained deep neural network to predict objects in the perceived vehicle environment 502A. Data associated with perceived vehicle environment 502A may be annotated to correctly identify the objects in the environment.”);
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Daravanh Phakousonh whose telephone number is (571)272-6324. The examiner can normally be reached Mon - Thurs 7 AM - 5 PM, Every other Friday 7 AM - 4PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 571-272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daravanh Phakousonh/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights