DETAILED ACTION
Claims 1-20 are currently pending and have been examined in this application. This communication is the first action on the merits.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the application filed 11/05/2024.
Claim Objections
Claims 2-10, 13-20 objected to because of the following informalities:
In all dependent claims, it is repeatedly the case that restated limitations subject to further limitations are not updated to reflect proper antecedent basis. An example of the proper way to correct this mistake is shown below, but all similar instances in all dependent claims must be corrected as well.
(Claim 2) “…the performing perception failure simulation processing on the first perception result information based on [[a]]the perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios comprises:…”
Appropriate correction is required.
Claim Rejections - 35 USC § 101
Claims 1-8, 11-19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
The claims are either directed to a method or apparatus, which is one of the statutory categories of invention.
(Step 1: YES)
The examiner has identified Claim 1 as the claim which represents the claimed invention for analysis and is similar to independent Claims 11 & 12. Claim 1 recites the following limitations (additional elements emphasized in bold are considered to be parsed from the remaining abstract idea):
A method for verifying a vehicle function, comprising: acquiring operation scenario data respectively corresponding to each of one or more operation scenarios, the operation scenario data comprising state information of an ego vehicle and surrounding environmental information of the ego vehicle; determining, based on the operation scenario data respectively corresponding to each of the operation scenarios, first perception result information corresponding to a preset vehicle function of the ego vehicle in each of the operation scenarios; performing perception failure simulation processing on the first perception result information based on a perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios; and verifying the preset vehicle function based on the second perception result information to obtain a verification result.
Under broadest reasonable interpretation, this claim covers performance of the limitations as a mental process (concepts performed in the human mind ) of acquiring data and verifying a result based on processing of the data.
If a claim limitation, under its broadest reasonable interpretation (BRI), covers performance of the limitation as concepts performed in the human mind, then it falls within the "mental process" grouping of abstract ideas.
Accordingly, the claim recites an abstract idea.
(Step 2A-Prong 1: YES. The claims are abstract)
This judicial exception is not integrated into a practical application. Limitations that are not indicative of integration into a practical application include: (1) Adding the words "apply it" ( or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses computer as a tool to perform an abstract idea (MPEP 2106.05.f), (2) Adding insignificant extra-solution activity to the judicial exception (MPEP 2106.05.g), (3) Generally Linking the use of the judicial exception to a particular technological environment or field of use (MPEP 2106.05.h). The ego vehicle in Claim 1 as well as storage medium and processor in Claim 11 and processor and memory in Claim 12 are just generic computer components. The computer hardware is recited at a high-level of generality (i.e., as a generic processor performing a generic computer function) such that it amounts to no more than mere instructions to implement an abstract idea by adding the words "apply it" (or an equivalent) with the judicial exception. Accordingly, these additional elements, when considered separately and as an ordered combination, do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore claim 1 is directed to an abstract idea without a practical application.
(Step 2A-Prong 2: NO. The additional claimed elements are not integrated into a practical application)
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered separately and as an ordered combination, they do not add significantly more (also known as an "inventive concept") to the exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using computer hardware amounts to no more than mere instructions to implement an abstract idea by adding the words "apply it" (or an equivalent) with the judicial exception. Mere instructions to implement an abstract idea on or with the use of generic computer components, cannot provide an inventive concept - rendering the claim patent ineligible. Thus claim 1 is not patent eligible.
(Step 2B: NO. The claims do not provide significantly more)
The dependent Claims 2-8, & 13-19 further define the abstract idea that is present in their respective independent claims and hence are abstract for at least the reasons presented above. The dependent claims do not include any additional elements that integrate the abstract idea into a practical application or are sufficient to amount to significantly more than the judicial exception when considered both individually and as an ordered combination. Therefore, the dependent claims are directed to an abstract idea. Thus, the aforementioned claims are not patent-eligible.
Claim 11 rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the recitation of “A computer-readable storage medium” as written does not exclude the possibility of applying to Transitory forms of signal transmission (often referred to as "signals per se"), such as a propagating electrical or electromagnetic signal or carrier wave. The BRI of machine readable media can encompass non-statutory transitory forms of signal transmission, such as a propagating electrical or electromagnetic signal per se. See In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). Thus, the claim is rejected as being directed to non-statutory subject matter.
Examiner Note: Amending the claim to include the word “non-transitory” would be sufficient to overcome this rejection.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Modalavalasa (US20210096571).
Claim 1:
Modalavalasa explicitly teaches:
A method for verifying a vehicle function, comprising: acquiring operation scenario data respectively corresponding to each of one or more operation scenarios, the operation scenario data comprising state information of an ego vehicle and surrounding environmental information of the ego vehicle;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
Examiner Note: vehicle function is recited broadly. Any function of a vehicle may correspond with this limitation.
determining, based on the operation scenario data respectively corresponding to each of the operation scenarios, first perception result information corresponding to a preset vehicle function of the ego vehicle in each of the operation scenarios;
(Modalavalasa) – “When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various and specific examples. Each instantiation of such an environment can be unique and defined. Enumerating all possible scenarios manually can require inordinate amounts of time and, if not every possible scenario is constructed, various scenarios can go untested. A scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.” (Para 0017)
“For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.” (Para 0018)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Examiner Note: Perception result data is recited broadly. Per BRI, it includes, but is not limited to, sensor data.
performing perception failure simulation processing on the first perception result information based on a perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios; and
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
verifying the preset vehicle function based on the second perception result information to obtain a verification result.
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 2:
Modalavalasa teaches the respective limitations of Claim 1: Modalavalasa further teaches:
wherein the first perception result information comprises a first perception result of at least one object around the ego vehicle; and
(Modalavalasa) – “When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various and specific examples. Each instantiation of such an environment can be unique and defined. Enumerating all possible scenarios manually can require inordinate amounts of time and, if not every possible scenario is constructed, various scenarios can go untested. A scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.” (Para 0017)
“For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.” (Para 0018)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
the performing perception failure simulation processing on the first perception result information based on a perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios comprises: for any of the operation scenarios, determining, based on the first perception result information in the operation scenario, a target object meeting a first condition;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
determining, based on the perception failure simulation rule, a failure mode corresponding to the target object;
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
performing perception failure simulation processing on the first perception result of the target object based on the failure mode to obtain a second perception result of perception failure of the target object; and
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
determining, based on the second perception result of the target object, the second perception result information corresponding to the operation scenario.
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 3:
Modalavalasa teaches the respective limitations of Claim 2. Modalavalasa further teaches:
wherein the failure mode comprises at least one type of sub-mode; and
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
for any sub-mode, the performing perception failure simulation processing on the first perception result of the target object based on the failure mode to obtain a second perception result of perception failure of the target object comprises: performing perception failure simulation processing on the first perception result of the target object based on a simulation sub-rule corresponding to the sub-mode to obtain the second perception result of the target object in the sub-mode; and
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
the determining, based on the second perception result of the target object, the second perception result information corresponding to the operation scenario comprise: determining, based on the second perception result corresponding to the target object in the sub-mode, the second perception result information corresponding to the sub-mode in the operation scenario.
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 4:
Modalavalasa teaches the respective limitations of Claim 3. Modalavalasa further teaches:
wherein the verifying the preset vehicle function based on the second perception result information to obtain a verification result comprises: determining, based on the second perception result information corresponding to the sub-mode in the operation scenario, a function response state of the preset vehicle function in the sub-mode of the operation scenario;
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
determining, based on the function response state of the preset vehicle function in the sub-mode of the operation scenario, a perception performance threshold of the preset vehicle function in the operation scenario; and
(Modalavalasa) – “An analysis component 128 can be configured to determine degrees of a success or a failure. By way of example and without limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.” (Para 0059)
“In some instances, the analysis component 128 can adjust a degree by which a scenario parameter is adjusted. By way of example and without limitation, a scenario parameter can indicate wet environment conditions (e.g., rainy conditions). The scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.). The analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation. In some instances, a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.” (Para 0063)
“In some instances, the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.” (Para 0065)
“Additionally, the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result. By way of example and without limitation, the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second. The simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.” (Para 0085)
determining, based on the perception performance threshold of the preset vehicle function in the operation scenario, a target perception performance threshold corresponding to the preset vehicle function as the verification result.
(Modalavalasa) – “Based on the probability of the perception error, a planning system of the autonomous vehicle can more safely traverse the environment by accounting for the probability of the perception error. In some instances, the planning system can use a probability threshold and determine a trajectory that avoids a location of an object that meets or exceeds a probability threshold. By way of example and without limitation, the planning system can use a probability threshold of 1% and traverse around locations of objects that have probabilities of 1% or greater. In some instances, the planning system can use probability thresholds associated with a classification of the object. By way of example and without limitation, the planning system can have a lower probability threshold associated with a pedestrian classification compared to a probability threshold associated with debris on a driving surface.” (Para 0152)
Claim 5:
Modalavalasa teaches the respective limitations of Claim 3. Modalavalasa further teaches:
wherein the performing perception failure simulation processing on the first perception result of the target object based on a simulation sub-rule corresponding to the sub-mode to obtain the second perception result of the target object in the sub-mode comprises: determining at least one perception performance indicator value based on the simulation sub-rule corresponding to the sub-mode; and
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
for any perception performance indicator value, updating the first perception result of the target object base on the perception performance indicator value to obtain the second perception result of the target object corresponding to the perception performance indicator value.
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 6:
Modalavalasa teaches the respective limitations of Claim 5. Modalavalasa further teaches:
wherein for any sub-mode, the determining, based on the second perception result corresponding to the target object in the sub-mode, the second perception result information corresponding to the sub-mode in the operation scenario comprises: determining, based on the second perception result of the target object corresponding to the perception performance indicator value, the second perception result information corresponding to the perception performance indicator value; and
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
the verifying the preset vehicle function based on the second perception result information to obtain a verification result comprises: determining, based on the second perception result information corresponding to the perception performance indicator value, a function response state corresponding to the preset vehicle function at the perception performance indicator value;
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
determining a perception performance threshold of the preset vehicle function in the sub-mode based on the function response state corresponding to the preset vehicle function at the perception performance indicator value; and
(Modalavalasa) – “An analysis component 128 can be configured to determine degrees of a success or a failure. By way of example and without limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.” (Para 0059)
“In some instances, the analysis component 128 can adjust a degree by which a scenario parameter is adjusted. By way of example and without limitation, a scenario parameter can indicate wet environment conditions (e.g., rainy conditions). The scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.). The analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation. In some instances, a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.” (Para 0063)
“In some instances, the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.” (Para 0065)
“Additionally, the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result. By way of example and without limitation, the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second. The simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.” (Para 0085)
determining, based on the perception performance threshold corresponding to the preset vehicle function in the sub-mode, a perception performance threshold of the preset vehicle function in the operation scenario as the verification result.
(Modalavalasa) – “Based on the probability of the perception error, a planning system of the autonomous vehicle can more safely traverse the environment by accounting for the probability of the perception error. In some instances, the planning system can use a probability threshold and determine a trajectory that avoids a location of an object that meets or exceeds a probability threshold. By way of example and without limitation, the planning system can use a probability threshold of 1% and traverse around locations of objects that have probabilities of 1% or greater. In some instances, the planning system can use probability thresholds associated with a classification of the object. By way of example and without limitation, the planning system can have a lower probability threshold associated with a pedestrian classification compared to a probability threshold associated with debris on a driving surface.” (Para 0152)
Claim 7:
Modalavalasa teaches the respective limitations of Claim 2. Modalavalasa further teaches:
wherein the determining, based on the operation scenario data respectively corresponding to each of the operation scenario, first perception result information corresponding to a preset vehicle function of the ego vehicle in each of the operation scenarios comprises: for any operation scenario, determining, based on the operation scenario data corresponding to the operation scenario, first perception result information of the ego vehicle at at least one trajectory point in the operation scenario; and
(Modalavalasa) – “To provide safe operation of such a system, an error model can be determined that can provide a probability associated with perception data and a vehicle can determine a trajectory based on the probability of an error associated with the perception data.” (Abstract)
“When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various and specific examples. Each instantiation of such an environment can be unique and defined. Enumerating all possible scenarios manually can require inordinate amounts of time and, if not every possible scenario is constructed, various scenarios can go untested. A scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.” (Para 0017)
“For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.” (Para 0018)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“That is, simulated environments resulting from generation techniques described herein can be used for testing, training, and validating a planner system and/or a prediction system of an autonomous vehicle controller, which can be used by an autonomous vehicle to navigate the autonomous vehicle along a trajectory in a real environment.” (Para 0027)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“The vehicle computing device can use the sensor data to generate a trajectory for the vehicle.” (Para 0031)
“Such log data may further include intermediate output by any one or more systems or subsystems of the vehicle including, but not limited to, messages indicating object detections, object tracks, predictions of future object locations, pluralities of trajectories generated in response to such detections, control signals passed to one or more systems or subsystems used to effectuate commands, and the like.” (Para 0032)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
“A reference cost can comprise a cost associated with a difference between a point (also referred to as a reference point) on the reference trajectory and a corresponding point (also referred to as a point or a target point) on the target trajectory, whereby the difference represents one or more difference in a yaw, lateral offset, velocity, acceleration, curvature, curvature rate, and/or the like.” (Para 0087)
the determining, based on the first perception result information in the operation scenario, a target object meeting a first condition comprises: determining, based on the first perception result information corresponding to the trajectory point, the target object in different areas around the ego vehicle.
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“The vehicle computing device can use the sensor data to generate a trajectory for the vehicle.” (Para 0031)
“Such log data may further include intermediate output by any one or more systems or subsystems of the vehicle including, but not limited to, messages indicating object detections, object tracks, predictions of future object locations, pluralities of trajectories generated in response to such detections, control signals passed to one or more systems or subsystems used to effectuate commands, and the like.” (Para 0032)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“For example, the simulation component 124 can perturb the simulation by continuously injecting the error into the simulation. By way of example and without limitation, example 402 depicts a bounding box 404 associated with an object at a time to. The bounding box 404 can represent a detection of the object by a vehicle that includes an error such as an error in a size of the bounding box and/or a position of the bounding box.” (Para 0076)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
“A reference cost can comprise a cost associated with a difference between a point (also referred to as a reference point) on the reference trajectory and a corresponding point (also referred to as a point or a target point) on the target trajectory, whereby the difference represents one or more difference in a yaw, lateral offset, velocity, acceleration, curvature, curvature rate, and/or the like.” (Para 0087)
“In some examples, an obstacle cost can comprise a cost associated with a distance between a point on the reference trajectory or the target trajectory and a point associated with an obstacle in the environment.” (Para 0088)
Claim 8:
Modalavalasa teaches the respective limitations of Claim 7. Modalavalasa further teaches:
wherein the verifying the preset vehicle function based on the second perception result information to obtain a verification result comprises: determining, based on the second perception result information, a perception performance threshold corresponding to the preset vehicle function in each of the different areas around the ego vehicle; and
(Modalavalasa) – “An analysis component 128 can be configured to determine degrees of a success or a failure. By way of example and without limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.” (Para 0059)
“In some instances, the analysis component 128 can adjust a degree by which a scenario parameter is adjusted. By way of example and without limitation, a scenario parameter can indicate wet environment conditions (e.g., rainy conditions). The scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.). The analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation. In some instances, a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.” (Para 0063)
“In some instances, the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.” (Para 0065)
“Additionally, the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result. By way of example and without limitation, the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second. The simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.” (Para 0085)
taking the perception performance threshold corresponding to the preset vehicle function in each of the different areas around the ego vehicle as the verification result.
(Modalavalasa) – “Based on the probability of the perception error, a planning system of the autonomous vehicle can more safely traverse the environment by accounting for the probability of the perception error. In some instances, the planning system can use a probability threshold and determine a trajectory that avoids a location of an object that meets or exceeds a probability threshold. By way of example and without limitation, the planning system can use a probability threshold of 1% and traverse around locations of objects that have probabilities of 1% or greater. In some instances, the planning system can use probability thresholds associated with a classification of the object. By way of example and without limitation, the planning system can have a lower probability threshold associated with a pedestrian classification compared to a probability threshold associated with debris on a driving surface.” (Para 0152)
Claim 9:
Modalavalasa teaches the respective limitations of Claim 1. Modalavalasa further teaches:
wherein the verifying the preset vehicle function based on the second perception result information to obtain a verification result comprises: for any operation scenario, determining map data of the operation scenario based on the operation scenario data of the operation scenario;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“For example, scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), an object characteristic (e.g., whether the object is changing lanes, whether the object 212 is a double-parked vehicle, etc.), proximity with one or more objects (in any coordinate frame), lane types (e.g., direction of a lane, parking lane), road markings (e.g., indicative of whether passing or lane changes are permitted, etc.), an object density, and the like.” (Para 0070)
determining, based on the second perception result information and the map data of the operation scenario, control information of the ego vehicle by using a functional algorithm corresponding to the preset vehicle function;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“For example, scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), an object characteristic (e.g., whether the object is changing lanes, whether the object 212 is a double-parked vehicle, etc.), proximity with one or more objects (in any coordinate frame), lane types (e.g., direction of a lane, parking lane), road markings (e.g., indicative of whether passing or lane changes are permitted, etc.), an object density, and the like.” (Para 0070)
“In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory 718 and 732 can be implemented as a neural network.” (Para 0112)
determining, based on the control information and a dynamic model of the ego vehicle, driving state information of the ego vehicle controlled according to the control information;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“For example, scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), an object characteristic (e.g., whether the object is changing lanes, whether the object 212 is a double-parked vehicle, etc.), proximity with one or more objects (in any coordinate frame), lane types (e.g., direction of a lane, parking lane), road markings (e.g., indicative of whether passing or lane changes are permitted, etc.), an object density, and the like.” (Para 0070)
“In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory 718 and 732 can be implemented as a neural network.” (Para 0112)
determining, based on the driving state information and the operation scenario data corresponding to the second perception result information, a function response state of the preset vehicle function under the second perception result information; and
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“For example, scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), an object characteristic (e.g., whether the object is changing lanes, whether the object 212 is a double-parked vehicle, etc.), proximity with one or more objects (in any coordinate frame), lane types (e.g., direction of a lane, parking lane), road markings (e.g., indicative of whether passing or lane changes are permitted, etc.), an object density, and the like.” (Para 0070)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
“In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory 718 and 732 can be implemented as a neural network.” (Para 0112)
determining the verification result based on the function response state of the preset vehicle function under the second perception result information.
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“For example, scenario parameters can include, but are not limited to, a velocity of the object 212, an acceleration of the object 212, an x-position of the object 212 (e.g., a global position, local position, and/or a position with respect to any other frame of reference), a y-position of the object 212 (e.g., a size, a pose, a local position, a global position and/or a position with respect to any other frame of reference), a bounding box associated with the object 212 (e.g., extents (length, width, and/or height), yaw, pitch, roll, etc.), lighting states (e.g., brake light(s), blinker light(s), hazard light(s), headlight(s), reverse light(s), etc.), a wheel orientation of the object 212, map elements (e.g., a distance between the object 212 and a stop light, stop sign, speed bump, intersection, yield sign, and the like), a classification of the object 212 (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.), an object characteristic (e.g., whether the object is changing lanes, whether the object 212 is a double-parked vehicle, etc.), proximity with one or more objects (in any coordinate frame), lane types (e.g., direction of a lane, parking lane), road markings (e.g., indicative of whether passing or lane changes are permitted, etc.), an object density, and the like.” (Para 0070)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
“In some instances, aspects of some or all of the components discussed herein can include any models, algorithms, and/or machine learning algorithms. For example, in some instances, the components in the memory 718 and 732 can be implemented as a neural network.” (Para 0112)
Claim 10:
Modalavalasa teaches the respective limitations of Claim 9. Modalavalasa further teaches:
wherein the determining, based on the driving state information and the operation scenario data corresponding to the second perception result information, a function response state of the preset vehicle function under the second perception result information comprises: determining, based on the driving state information and the operation scenario data corresponding to the second perception result information, a state of an expected result of the ego vehicle under the preset vehicle function;
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“An analysis component 128 can be configured to determine degrees of a success or a failure. By way of example and without limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.” (Para 0059)
“In some instances, the analysis component 128 can adjust a degree by which a scenario parameter is adjusted. By way of example and without limitation, a scenario parameter can indicate wet environment conditions (e.g., rainy conditions). The scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.). The analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation. In some instances, a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.” (Para 0063)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“In some instances, the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.” (Para 0065)
“Additionally, the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result. By way of example and without limitation, the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second. The simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.” (Para 0085)
in response to the state of the expected result being achieved, determining the function response state of the preset vehicle function under the second perception result information as success; or in response to the state of the expected result being unachieved, determining the function response state of the preset vehicle function under the second perception result information as failure.
(Modalavalasa) – “In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“An analysis component 128 can be configured to determine degrees of a success or a failure. By way of example and without limitation, a rule can indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. The simulation data 126 can indicate that in a first variation of the parameterized scenario 122, the simulated vehicle stopped in excess of 5 meters from the threshold distance. In a second variation of the parameterized scenario 122, the simulation data 126 can indicate that the simulated vehicle stopped in excess of 10 meters from the threshold distance. The analysis component 128 can indicate that the simulated vehicle performed more successfully in the second variation compared to the simulated vehicle in the first variation. For example, the analysis component 128 can determine an ordered list (e.g., ordered according to a relative success scale) that includes simulated vehicles and the associated variations of the parameterized scenario 122. Such variations may also be used to determine limitations of the various components of the system being simulated.” (Para 0059)
“In some instances, the analysis component 128 can adjust a degree by which a scenario parameter is adjusted. By way of example and without limitation, a scenario parameter can indicate wet environment conditions (e.g., rainy conditions). The scenario parameter can be adjusted on a range (e.g., a quarter inch of rainfall, an inch of rainfall, etc.). The analysis component 128 can adjust a magnitude of the scenario parameter and perform the sensitivity analysis based on the magnitude of the scenario parameter to determine a threshold associated with the scenario parameter that can result in a successful or an unsuccessful outcome of the simulation. In some instances, a binary search algorithm, a particle filter algorithm, and/or a Monte Carlo method can be used to determine the threshold, although other suitable algorithms are contemplated.” (Para 0063)
“The vehicle performance component 130 can, based on the simulation data 126 (and/or the additional simulation data based on the additional variations from the analysis component 128) and the types of failures, determine the vehicle performance data 132. In some instances, the vehicle performance data 132 can indicate how a vehicle performs in an environment. By way of example and without limitation, the vehicle performance data 132 can indicate that a vehicle traveling at a speed of 15 meters per second has a stopping distance of 15 meters. In some instances, the vehicle performance data can indicate safety metrics. By way of example and without limitation, the vehicle performance data 132 can indicate an event (e.g., a failure) and a cause of the event. In at least some examples, such indication may be binary (failure or not), coarse (levels of failure, e.g., “critical”, “non-critical”, and “pass”), or continuous (e.g., representing a probability of failure), though any other indication is contemplated. For example, for an event type 1 and a cause type 1, the data 134(1) can indicate a safety rating and similarly for data 134(2)-134(4). In some instances, the cause type 1 and the cause type 2 can indicate a fault such as a fault of the vehicle or a fault of an object (e.g., a bicyclist). The vehicle performance data 132 can then indicate a safety metric associated with the parameterized scenario 122. In some instances, the vehicle performance component 130 can use a target metric and compare the vehicle performance data 132 with the target metric to determine if the safety metric meets or exceeds the target metric. In some instances, the target metric can be based on standards and/or regulations associated with autonomous vehicles.” (Para 0064)
“In some instances, the vehicle performance data 132 can be input into a filter component 136 to determine filtered data 138 based on the vehicle performance data 132. For example, the filter component 136 can be used to determine filtered data 138 that identifies regions that do not meet a coverage threshold.” (Para 0065)
“Additionally, the simulation component 124 can identify variations of the parameterized scenario for additional simulation based on the result. By way of example and without limitation, the result of a simulation can be a failure where a scenario parameter was associated with a vehicle velocity of 15 meters per second. The simulation component 124 can identify velocities that are near the 15 meters per second velocity to determine a threshold at which the simulation will pass which can further assist the development of safer vehicle controllers.” (Para 0085)
Claim 11:
Modalavalasa explicitly teaches:
A computer-readable storage medium, wherein the storage medium stores a computer program that, when executed by a processor, implements the following steps: acquiring operation scenario data respectively corresponding to each of one or more operation scenarios, the operation scenario data comprising state information of an ego vehicle and surrounding environmental information of the ego vehicle;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“The computing device(s) 702 can include one or more processor(s) 516 and memory 518 communicatively coupled with the one or more processor(s) 716. In the illustrated example, the memory 718 of the computing device(s) 702 stores a localization component 720, a perception component 722, a prediction component 724, a planning component 726, and one or more system controller(s) 728. Though depicted as residing in the memory 718 for illustrative purposes, it is contemplated that the localization component 720, the perception component 722, the prediction component 724, the planning component 726, and the one or more system controller(s) 728 can additionally, or alternatively, be accessible to the computing device(s) 702 (e.g., stored in a different component of vehicle(s) 202 and/or be accessible to the vehicle(s) 202 (e.g., stored remotely).” (Para 0100)
Examiner Note: vehicle function is recited broadly. Any function of a vehicle may correspond with this limitation.
determining, based on the operation scenario data respectively corresponding to each of the operation scenarios, first perception result information corresponding to a preset vehicle function of the ego vehicle in each of the operation scenarios;
(Modalavalasa) – “When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various and specific examples. Each instantiation of such an environment can be unique and defined. Enumerating all possible scenarios manually can require inordinate amounts of time and, if not every possible scenario is constructed, various scenarios can go untested. A scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.” (Para 0017)
“For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.” (Para 0018)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Examiner Note: Perception result data is recited broadly. Per BRI, it includes, but is not limited to, sensor data.
performing perception failure simulation processing on the first perception result information based on a perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios; and
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
verifying the preset vehicle function based on the second perception result information to obtain a verification result.
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 12:
Modalavalasa explicitly teaches:
An electronic device, comprising: a processor; and a memory configured to store an executable instruction; wherein the processor is configured to read the executable instruction from the memory and execute the instruction to implement the following steps: acquiring operation scenario data respectively corresponding to each of one or more operation scenarios, the operation scenario data comprising state information of an ego vehicle and surrounding environmental information of the ego vehicle;
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“The computing device(s) 702 can include one or more processor(s) 516 and memory 518 communicatively coupled with the one or more processor(s) 716. In the illustrated example, the memory 718 of the computing device(s) 702 stores a localization component 720, a perception component 722, a prediction component 724, a planning component 726, and one or more system controller(s) 728. Though depicted as residing in the memory 718 for illustrative purposes, it is contemplated that the localization component 720, the perception component 722, the prediction component 724, the planning component 726, and the one or more system controller(s) 728 can additionally, or alternatively, be accessible to the computing device(s) 702 (e.g., stored in a different component of vehicle(s) 202 and/or be accessible to the vehicle(s) 202 (e.g., stored remotely).” (Para 0100)
Examiner Note: vehicle function is recited broadly. Any function of a vehicle may correspond with this limitation.
determining, based on the operation scenario data respectively corresponding to each of the operation scenarios, first perception result information corresponding to a preset vehicle function of the ego vehicle in each of the operation scenarios;
(Modalavalasa) – “When creating a simulation environment to perform testing and validation, it is possible to specifically enumerate the environment with various and specific examples. Each instantiation of such an environment can be unique and defined. Enumerating all possible scenarios manually can require inordinate amounts of time and, if not every possible scenario is constructed, various scenarios can go untested. A scenario parameter can be used to parameterize characteristics and/or attributes of an object within the scenario and provide variations of the scenario.” (Para 0017)
“For example, a vehicle or multiple vehicles can traverse an environment and generate log data associated with the environment. The log data can include sensor data captured by one or more sensors of the vehicle, perception data indicating objects identified by one or more systems onboard the vehicle (or produced during a post-processing phase), prediction data indicating an intent of objects (whether produced during the recording or subsequent thereto), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit the log data, via a network, to a database that stores log data and/or to a computing device that analyzes the log data.” (Para 0018)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 1 illustrates an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. To generate a scenario, input data 102 can be used. The input data 102 can include vehicle data 104 and/or additional situational data 106. The vehicle data 104 can include log data captured by a vehicle traveling through an environment.” (Para 0028)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Examiner Note: Perception result data is recited broadly. Per BRI, it includes, but is not limited to, sensor data.
performing perception failure simulation processing on the first perception result information based on a perception failure simulation rule to obtain second perception result information respectively corresponding to each of the operation scenarios; and
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
verifying the preset vehicle function based on the second perception result information to obtain a verification result.
(Modalavalasa) – “Scenarios that include the same or similar stereotypes can be used to verify functionality of the autonomous vehicle controller.” (Para 0051)
“Techniques described herein are directed to various aspects of determining performance metrics of a system. In at least some examples described herein, such performance metrics may be determined, for example, using simulations in conjunction with other performance metric determinations. Simulations can be used to validate software (e.g., a vehicle controller) executed on vehicles (e.g., autonomous vehicles) and gather safety metrics to ensure that the software is able to safely control such vehicles in various scenarios. In additional or alternative examples, simulations can be used to learn about the constraints of autonomous vehicles that use the autonomous controller. For instance, simulations can be used to understand the operational space of an autonomous vehicle (e.g., an envelope in which the autonomous controller effectively controls the autonomous vehicle) in view of surface conditions, ambient noise, faulty components, etc.” (Para 0016)
“In some instances, the simulation can be used to test and validate a response of the autonomous vehicle controller to a defective (and/or a faulty) sensor of the vehicle and/or a defective (and/or faulty) processing of sensor data. In such an example, the computing device can be configured to introduce inconsistencies in a scenario parameter of an object. For example, an error model can indicate an error and/or an error percentage associated with a scenario parameter. The scenario can incorporate the error and/or the error percentage into a simulated scenario and simulate the response of the autonomous vehicle controller. Such errors may be represented by, without limitation, look-up tables determined based on statistical aggregation using ground-truth data, functions (e.g., errors based on input parameters), or any other model which maps a parameter to a particular error. In at least some examples, such error models may map particular errors with probabilities/frequencies of occurrence.” (Para 0021)
“FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of an error or an uncertainty associated with a simulated environment. As discussed above, the error model data can include an error model that associates an error 306 with a probability 308 and the error model can be associated with environmental parameters. A simulation component 124 can use the error model data 304 to inject an error which can result in a perturbed parameterized scenario to perturb the simulation. Based on the injected error, simulation data can indicate how an autonomous controller responds to the injected error.” (Para 0075)
“FIG. 5 illustrates an example 500 of vehicle(s) 202 generating vehicle data 104 and transmitting the vehicle data 104 to the computing device(s) 204. As discussed above, the error model component 116 can determine a perception error model that can indicate an error associated with a scenario parameter. As discussed above, the vehicle data 104 can include sensor data generated by a sensor of the vehicle(s) 202 and/or perception data generated by a perception system of the vehicle(s) 202. The perception error model can be determined by comparing the vehicle data 104 against ground truth data 302. The ground truth data 302 can be manually labeled and can be associated with the environment and can represent a known result. Therefore, a deviation from the ground truth data 302 in the vehicle data 104 can be identified as an error in a sensor system and/or the perception system of the vehicle(s) 202.” (Para 0078)
Claim 13:
Rejected for the same reasons as Claim 2
Claim 14:
Rejected for the same reasons as Claim 3
Claim 15:
Rejected for the same reasons as Claim 4
Claim 16:
Rejected for the same reasons as Claim 5
Claim 17:
Rejected for the same reasons as Claim 6
Claim 18:
Rejected for the same reasons as Claim 7
Claim 19:
Rejected for the same reasons as Claim 8
Claim 20:
Rejected for the same reasons as Claim 9
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Oboril (US20240367680) teaches the use of MTBF models that identify MTBFs correlated to perception errors of autonomous vehicles.
Son (US20240343270) teaches a similar simulation system.
Rider (US20230080379) teaches a Digital Twin for an autonomous vehicle.
Reschka (US11577741) teaches a similar method for testing vehicle systems.
Jiang (US202200227397) teaches evaluating the accuracy of a dynamic model in predicting the trajectory of ADV when simulating the behavior of the ADV.
Odblom (US20220048533) teaches validating an automated control software for a vehicle.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID RUBEN PEDERSEN whose telephone number is (571)272-9696. The examiner can normally be reached M-Th: 07:00 -16:00 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID RUBEN PEDERSEN/Examiner, Art Unit 3658
