Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 4/03/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5 and 7-20 are rejected under 35 U.S.C. 103 as being unpatentable over Nassar, et al. (U.S. Patent Application Pub. No. 2021/0294944) in view of Sholingar, et al. (U.S. Patent Application Pub. No. 2020/0082034).
Regarding Claim 1, Nassar teaches: A device for vehicle simulation (Nassar, Para. 0006, 0021 – “systems and methods for training, testing, and verifying autonomous machines using simulated environments”, where “autonomous machines” include “autonomous vehicle[s]”), comprising:
a memory (Nassar, Para. 0024 – “memory”); and
a processor operatively coupled to the memory, the processor being configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to:
identify a first parameter set defined using a first scenario format (Nassar, Para. 0028 – a “scenario determiner” may “determine concrete values and/or parameters corresponding to undefined criteria in order to generate the one or more scenarios and/or observers” from an “initial declarative description of the scenario”), wherein the first parameter set comprises:
one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints (Nassar, Para. 0028, 0036, 0042 – “a scenario designer 122 which may use a declarative description in addition to a domain ontology 130 to define information for the scenario generator 126”, the “domain ontology representation” comprised of “various ontologies related to different aspects of the simulated environment”, for example “each ontology can include primitive concepts (e.g., object, activity, activity occurrence, time point, etc.), constraints, functions (e.g., beginning, end), and relations (occurrence of, participates in, between, before, exists at, occurring at)”);
;
generate a test configuration object using the the “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle 102, a semi-autonomous vehicle, and/or a component or feature thereof against the defined scenarios generated using the scenario generator 126”; where the “simulation system” may “generate a simulated environment 610 that may include AI objects 612 (e.g., AI objects 612A and 612B), HIL objects 614, SIL objects 616, PIL, objects 618, and/or other object types”); and
send the test configuration object to a testing modality for testing (Nassar, Para. 0063-0070, 0094-0096 – a “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle and/or actual hardware or software of the autonomous vehicle (e.g., HIL or SIL)”, different testing modalities, “against the defined scenarios generated using the scenario generator”; where the “HIL configuration may include a two-box solution” and “SIL vehicles or objects may use software to simulate or emulate the hardware from the HIL vehicles or objects”).
Nassar does not specifically teach map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format comprises: one or more internal axes, one or more internal relationships, and one or more internal constraints; generate a test configuration object using the internal parameter set.
However, Sholingar teaches map the first parameter set to an internal parameter set using an internal parameter format (Sholingar, Para. 0032-0033 – “convert environments defined in database 201 into test case descriptions” to create a “scenario description file”; where the database includes “a scenario for every possible different combination of feature values”, or internal parameters, the feature values include “weather, ground surface, lighting conditions, sun angle, etc.”), wherein the internal parameter format comprises:
one or more internal axes, one or more internal relationships, and one or more internal constraints (Sholingar, Para. 0031-0033, 0035-0036 – the database includes “a scenario for every possible different combination of feature values”, or relationships, where each “feature represents a variable…. that can have any variety of different values”, or axes, and each feature is “associated with two or more values and up to a dozen or more values”; a “machine learning classifier” which can “determine an importance, for example, a weight, for each feature” and statistics indicating an “algorithm performance” indicating failure of autonomous vehicle algorithm associated with each feature);
generate a test configuration object using the internal parameter set (Sholingar, Para. 0033 – “Test case description module 202 is configured to convert environments defined in database 201 into test case descriptions”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device of Nassar to include map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format comprises: one or more internal axes, one or more internal relationships, and one or more internal constraints; generate a test configuration object using the internal parameter set, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” (Sholingar, Para. 0018).
In regards to Claim 2, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar in view of Sholingar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: identify a compilation source in the internal parameter format to map the internal parameter set to the first parameter set (Sholingar, Para. 0013, 0018, 0037 – “automatic identification of features and feature values” using an “iterative approach”, wherein after “higher-dimensional feature space report and feature importance report” is provided and can be used to query the database “loaded with (a potentially exhaustive) list of different feature combinations for a scenario”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to further include identify a compilation source in the internal parameter format to map the internal parameter set to the first parameter set, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” (Sholingar, Para. 0018).
In regards to Claim 3, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar in view of Sholingar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: identify a metric in the internal parameter format after testing (Sholingar, Para. 0013, 0035 – “the algorithm determines metrics for the scenario, such as, a binary metric (e.g.,) if a scenario passed (succeeded) or failed, a non-binary metric, or other custom defined metric”, such as “an importance, for example, a weight, for each feature”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to further include identify a metric in the internal parameter format after testing, as taught by Sholingar, in order to improve testing accuracy and learning.
In regards to Claim 4, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar further teaches wherein the testing modality includes one or more of: a software in the loop (SIL) simulator plugin interface (Nassar, Para. 0095, 0100 – “a software-in-the-loop configuration”; “vehicle simulator component(s) 620 (e.g., for a SIL object)”), a hardware in the loop (HIL) simulator plugin interface (Nassar, Para. 0094, 0100 – “a hardware-in-the-loop configuration”; “vehicle simulator component(s) 606 (e.g., for a HIL object)”), or a track testing plugin interface.
In regards to Claim 5, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: format the test configuration object into a testing modality format for the testing modality (Nassar, Para. 0030, 0072, 0106 – the “simulation system” generates “a simulated environment 610 that may include AI objects 612 (e.g., AI objects 612A and 612B), HIL objects 614, SIL objects 616, PIL, objects 618, and/or other object types”); and send the test configuration object to the testing modality (Nassar, Para. 0063-0070, 0094-0096 – a “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle and/or actual hardware or software of the autonomous vehicle (e.g., HIL or SIL)”, different testing modalities, “against the defined scenarios generated using the scenario generator”).
In regards to Claim 7, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: convert, at an internal parameter format converter plugin, testing modality format data into internal parameter format data (Sholingar, Para. 0033, 0050 – a “test case description module” configured to “convert environments defined in database 201 into test case descriptions”, for example, creating “a scenario description file (e.g., a YAML file, a JSON file, a txt file, a csv file, etc.) for each scenario” and taking “a scenario description file, such as, a YAML file, to run a simulation in a virtual environment”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to include convert, at an internal parameter format converter plugin, testing modality format data into internal parameter format data, as taught by Sholingar, in order to reduce the computation effort and time to run batch testing by utilizing a standardized format.
In regards to Claim 8, Nassar in view of Sholingar teaches the device of Claim 1, and Nassar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: convert axes data in a time-based metrics signal format into parameter format data (Nassar, Para. 0044 – “the system can scan through time series data for time variables and then evaluate for defined information from the user specified declarative description”).
Nassar does not teach convert, at an internal parameter format converter plugin, axes data into internal parameter format data; and write the axes data to a database.
However, Sholingar teaches convert, at an internal parameter format converter plugin, axes data into internal parameter format data (Sholingar, Para. 0033 – “Test case description module 202 is configured to convert environments defined in database 201 into test case descriptions”); and write the axes data to a database (Sholingar, Para. 0031-0033, 0035-0036 – the database includes “a scenario for every possible different combination of feature values” where each “feature represents a variable…. that can have any variety of different values”, or axes, and each feature is “associated with two or more values and up to a dozen or more values”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to further include convert, at an internal parameter format converter plugin, axes data into internal parameter format data; and write the axes data to a database, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” by automatic identification of features using known data (Sholingar, Para. 0018).
Regarding Claim 9, Nassar teaches: A device for vehicle testing (Nassar, Para. 0006, 0021 – “systems and methods for training, testing, and verifying autonomous machines using simulated environments”, where “autonomous machines” include “autonomous vehicle[s]”), comprising:
a memory (Nassar, Para. 0024 – “memory”); and
a processor operatively coupled to the memory, the processor being configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to:
identify a first parameter set defined using a first scenario format (Nassar, Para. 0028 – a “scenario determiner” may “determine concrete values and/or parameters corresponding to undefined criteria in order to generate the one or more scenarios and/or observers” from an “initial declarative description of the scenario”), wherein the first parameter set comprises:
one or more first scenario format axes, one or more first scenario format relationships, and one or more first scenario format constraints (Nassar, Para. 0028, 0036, 0042 – “a scenario designer 122 which may use a declarative description in addition to a domain ontology 130 to define information for the scenario generator 126”, the “domain ontology representation” comprised of “various ontologies related to different aspects of the simulated environment”, for example “each ontology can include primitive concepts (e.g., object, activity, activity occurrence, time point, etc.), constraints, functions (e.g., beginning, end), and relations (occurrence of, participates in, between, before, exists at, occurring at)”);
compute a parametric metric, wherein the parametric metric is based on one or more of a parametric performance metric (Nassar, Para. 0031-0033 – an “analyzer” used “to evaluate the variations of the defined scenario” including determining a “scenario-quality metric” and “key performance indicator (KPI) metric”) or a parametric coverage metric (Nassar, Para. 0032 – the “analyzer” may determine a “coverage”, where coverage “may be represented using a percentage of how many variations of the defined scenario meet the user specified declarative description (e.g., 900%) and/or in any other key performance indicator (KPI) metric”).
Nassar does not specifically teach map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format comprises: one or more internal axes, one or more internal relationships, and one or more internal constraints.
However, Sholingar teaches map the first parameter set to an internal parameter set using an internal parameter format (Sholingar, Para. 0032-0033 – “convert environments defined in database 201 into test case descriptions” to create a “scenario description file”; where the database includes “a scenario for every possible different combination of feature values”, or internal parameters, the feature values include “weather, ground surface, lighting conditions, sun angle, etc.”), wherein the internal parameter format comprises:
one or more internal axes, one or more internal relationships, and one or more internal constraints (Sholingar, Para. 0031-0033, 0035-0036 – the database includes “a scenario for every possible different combination of feature values”, or relationships, where each “feature represents a variable…. that can have any variety of different values”, or axes, and each feature is “associated with two or more values and up to a dozen or more values”; a “machine learning classifier” which can “determine an importance, for example, a weight, for each feature” and statistics indicating an “algorithm performance” indicating failure of autonomous vehicle algorithm associated with each feature).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device of Nassar to include map the first parameter set to an internal parameter set using an internal parameter format, wherein the internal parameter format comprises: one or more internal axes, one or more internal relationships, and one or more internal constraints, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” (Sholingar, Para. 0018).
In regards to Claim 10, Nassar in view of Sholingar teaches the device of Claim 9, and Nassar further teaches wherein the parametric metric is based on a use case (Nassar, Para. 0031-0033 – “the analyzer 132 may evaluate the accuracy of the generated scenarios, accuracy of the system under test, and/or coverage of requirements and/or test objectives”, where “the analyzer 132 may evaluate the variations of the defined scenario” and run “additional variations of the defined scenario” based on the metric).
In regards to Claim 11, Nassar in view of Sholingar teaches the device of Claim 9, and Nassar in view of Sholingar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: select a set of axes based on the parametric metric (Sholingar, Para. 0031-0033, 0035-0038, 0048 – the database includes “a scenario for every possible different combination of feature values” where each “feature represents a variable…. that can have any variety of different values”, or axes, and “test cases are selected using a metric where the selector identifies the feature value which most often leads to failure of the algorithm”); or display a result based on the set of axes on a display device (Sholingar, Para. 0016, 0025, 0029, 0035-0037 – a “display device” capable “of displaying information to one or more users of computing device”; for example a “summary of the higher-dimensional feature space report and/or the feature report can be presented on a visualization dashboard to indicate features and how the features contributed to algorithm success/failure”, the dashboard being part of the “computer architecture”); or select a subset of the set of axes (Sholingar, Para. 0019, 0054 – where subsequent tests can “focus more coverage on scenarios that lead to algorithm failure”, or a subset, “instead of a randomized selection”); or display a subset of the result based on the subset of the set of axes on the display device (Sholingar, Para. 0016-0019, 0025 – “displaying information to one or more users” on the “display device” such as a report after “each iteration”, including “subsequent iterations” focused on “scenarios that lead to algorithm failure”, or a subset).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to further include cause the device to: select a set of axes based on the parametric metric; or display a result based on the set of axes on a display device; or select a subset of the set of axes; or display a subset of the result based on the subset of the set of axes on the display device, as taught by Sholingar, in order to improve testing accuracy and learning by focusing on a relevant subset and presenting the information to a user to “gain insight” (Sholingar, Para. 0018).
In regards to Claim 12, Nassar in view of Sholingar teaches the device of Claim 9, and Nassar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: retrieve one or more axes values for one or more testing modalities (Nassar, Para. 0036, 0067 – “the scenario determiner 124 can take the declarative description (e.g., received by the scenario designer 122) and access the domain ontology 130 to determine representatives to fill various metrics that are not explicitly defined in the scenario”, or axes values; where “the simulated environment is used for testing vehicle performance (e.g., for HIL or SIL embodiments)”); and compute one or more axes-specific metrics based on the one or more axes values (Nassar, Para. 0051 – “a scenario determiner 124 can take the declarative description (e.g., received by the scenario designer 122) and access the domain ontology 130” where the “domain ontology 130 may be used to determine the path structure information and dynamic actor information” to “fill various metrics that are not explicitly defined in the declarative description (e.g., traffic, speed of the swerve, aggressive driving, number of vehicles, types of vehicles, locations of buildings, trees, wait conditions, etc.)”).
In regards to Claim 13, Nassar in view of Sholingar teaches the device of Claim 9, and Nassar in view of Sholingar further teaches wherein the processor is further configured to execute instructions (Nassar, Para. 0024 – “a processor executing instructions stored in memory”) to cause the device to: generate a test configuration object based on one or more axes-specific metrics, wherein the one or more axes-specific metrics are based on one or more axes value selected based on the parametric metric (Sholingar, Para. 0019, 0038, 0048, 0054 – “test cases are selected using a metric where the selector identifies the feature value which most often leads to failure of the algorithm” where subsequent tests can “focus more coverage on scenarios that lead to algorithm failure instead of a randomized selection”, and “the process of selecting subsequent queries is automated using a pre-defined metric”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the device including the above limitations of Nassar in view of Sholingar to further include cause the device to: generate a test configuration object based on one or more axes-specific metrics, wherein the one or more axes-specific metrics are based on one or more axes value selected based on the parametric metric, as taught by Sholingar, in order to improve testing accuracy and learning.
Regarding Claim 14, Nassar teaches: A non-transitory computer readable storage medium (Nassar, Para. 0228-0229 – “memory 904 may include any of a variety of computer-readable media” and stores “computer-readable instructions”) including computer executable instructions that, when executed by one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), cause a vehicle simulator to:
receive scenario format data (Nassar, Para. 0027-0028 – a “scenario designer” which “may receive a declarative description” and a “scenario determiner” may “determine concrete values and/or parameters corresponding to undefined criteria in order to generate the one or more scenarios and/or observers” from an “initial declarative description of the scenario”);
generate a test configuration object using the the “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle 102, a semi-autonomous vehicle, and/or a component or feature thereof against the defined scenarios generated using the scenario generator 126”); and
test the test configuration object using a testing modality (Nassar, Para. 0063-0070, 0094-0096 – a “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle and/or actual hardware or software of the autonomous vehicle (e.g., HIL or SIL)”, different testing modalities, “against the defined scenarios generated using the scenario generator”; where the “HIL configuration may include a two-box solution” and “SIL vehicles or objects may use software to simulate or emulate the hardware from the HIL vehicles or objects”).
Nassar does not specifically teach convert the scenario format data into internal parameter format data; and generate a test configuration object using the internal parameter format data.
However, Sholingar teaches convert the scenario format data into internal parameter format data (Sholingar, Para. 0032-0033 – “convert environments defined in database 201 into test case descriptions” to create a “scenario description file”; where the database includes “a scenario for every possible different combination of feature values”, or internal parameters, the feature values include “weather, ground surface, lighting conditions, sun angle, etc.”), and generate a test configuration object using the internal parameter format data (Sholingar, Para. 0033 – “Test case description module 202 is configured to convert environments defined in database 201 into test case descriptions”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the non-transitory computer readable medium of Nassar to include convert the scenario format data into internal parameter format data; and generate a test configuration object using the internal parameter format data, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” (Sholingar, Para. 0018).
In regards to Claim 15, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 14, and Nassar further teaches wherein the instructions, when executed by the one or more processors, further cause the vehicle simulator to: format the test configuration object into a testing modality format for the testing modality (Nassar, Para. 0030, 0072, 0106 – the “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle 102, a semi-autonomous vehicle, and/or a component or feature thereof against the defined scenarios generated using the scenario generator 126”; where the “simulation system” may “generate a simulated environment 610 that may include AI objects 612 (e.g., AI objects 612A and 612B), HIL objects 614, SIL objects 616, PIL, objects 618, and/or other object types”); and send the test configuration object to the testing modality (Nassar, Para. 0063-0070, 0094-0096 – a “simulation implementer 128 may be used to test a virtual instance of an autonomous vehicle and/or actual hardware or software of the autonomous vehicle (e.g., HIL or SIL)”, different testing modalities, “against the defined scenarios generated using the scenario generator”), wherein the testing modality includes one or more of a software in the loop (SIL) simulator plugin interface (Nassar, Para. 0095, 0100 – “a software-in-the-loop configuration”; “vehicle simulator component(s) 620 (e.g., for a SIL object)”), a hardware in the loop (HIL) simulator plugin interface (Nassar, Para. 0094, 0100 – “a hardware-in-the-loop configuration”; “vehicle simulator component(s) 606 (e.g., for a HIL object)”), or a track testing plugin interface.
In regards to Claim 16, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 14, and Nassar further teaches wherein the instructions, when executed by the one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), further cause the vehicle simulator to: convert, at an internal parameter format converter plugin, testing modality format data into internal parameter format data (Sholingar, Para. 0033, 0050 – a “test case description module” configured to “convert environments defined in database 201 into test case descriptions”, for example, creating “a scenario description file (e.g., a YAML file, a JSON file, a txt file, a csv file, etc.) for each scenario” and taking “a scenario description file, such as, a YAML file, to run a simulation in a virtual environment”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the non-transitory computer readable medium including the above limitations of Nassar in view of Sholingar to include convert, at an internal parameter format converter plugin, testing modality format data into internal parameter format data, as taught by Sholingar, in order to reduce the computation effort and time to run batch testing by utilizing a standardized format.
In regards to Claim 17, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 14, and Nassar further teaches wherein the instructions, when executed by the one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), further cause the vehicle simulator to: convert axes data in a time-based metrics signal format (Nassar, Para. 0044 – “the system can scan through time series data for time variables and then evaluate for defined information from the user specified declarative description”).
Nassar does not teach convert, at an internal parameter format converter plugin, axes data into internal parameter format data; and write the axes data to a database.
However, Sholingar teaches convert, at an internal parameter format converter plugin, axes data into internal parameter format data (Sholingar, Para. 0033 – “Test case description module 202 is configured to convert environments defined in database 201 into test case descriptions”); and write the axes data to a database (Sholingar, Para. 0031-0033, 0035-0036 – the database includes “a scenario for every possible different combination of feature values” where each “feature represents a variable…. that can have any variety of different values”, or axes, and each feature is “associated with two or more values and up to a dozen or more values”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the non-transitory computer readable medium including the above limitations of Nassar in view of Sholingar to further include convert, at an internal parameter format converter plugin, axes data into internal parameter format data; and write the axes data to a database, as taught by Sholingar, in order to reduce “the computation effort and time to run batch testing” by automatic identification of features using known data (Sholingar, Para. 0018).
In regards to Claim 18, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 14, and Nassar further teaches wherein the instructions, when executed by the one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), further cause the vehicle simulator to: compute a parametric metric, wherein the parametric metric is based on one or more of a parametric performance metric (Nassar, Para. 0031-0033 – an “analyzer” used “to evaluate the variations of the defined scenario” including determining a “scenario-quality metric” and “key performance indicator (KPI) metric”) or a parametric coverage metric (Nassar, Para. 0032 – the “analyzer” may determine a “coverage”, where coverage “may be represented using a percentage of how many variations of the defined scenario meet the user specified declarative description (e.g., 900%) and/or in any other key performance indicator (KPI) metric”), wherein the parametric metric is based on a use case (Nassar, Para. 0031-0033 – “the analyzer 132 may evaluate the accuracy of the generated scenarios, accuracy of the system under test, and/or coverage of requirements and/or test objectives”, where “the analyzer 132 may evaluate the variations of the defined scenario” and run “additional variations of the defined scenario” based on the metric).
In regards to Claim 19, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 14, and Nassar further teaches wherein the instructions, when executed by the one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), further cause the vehicle simulator to: retrieve one or more axes values for the testing modality (Nassar, Para. 0036, 0067 – “the scenario determiner 124 can take the declarative description (e.g., received by the scenario designer 122) and access the domain ontology 130 to determine representatives to fill various metrics that are not explicitly defined in the scenario”, or axes values; where “the simulated environment is used for testing vehicle performance (e.g., for HIL or SIL embodiments)”); and compute one or more axes-specific metrics based on the one or more axes values (Nassar, Para. 0051 – “a scenario determiner 124 can take the declarative description (e.g., received by the scenario designer 122) and access the domain ontology 130” where the “domain ontology 130 may be used to determine the path structure information and dynamic actor information” to “fill various metrics that are not explicitly defined in the declarative description (e.g., traffic, speed of the swerve, aggressive driving, number of vehicles, types of vehicles, locations of buildings, trees, wait conditions, etc.)”).
In regards to Claim 20, Nassar in view of Sholingar teaches the non-transitory computer readable medium of Claim 18, and Nassar further teaches wherein the instructions, when executed by the one or more processors (Nassar, Para. 0024 – “a processor executing instructions stored in memory”), further cause the vehicle simulator to: generate the test configuration object based on one or more axes-specific metrics, wherein the one or more axes-specific metrics are based on one or more axes value selected based on the parametric metric (Sholingar, Para. 0019, 0038, 0048, 0054 – “test cases are selected using a metric where the selector identifies the feature value which most often leads to failure of the algorithm” where subsequent tests can “focus more coverage on scenarios that lead to algorithm failure instead of a randomized selection”, and “the process of selecting subsequent queries is automated using a pre-defined metric”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the non-transitory computer readable medium including the above limitations of Nassar in view of Sholingar to further include cause the vehicle simulator to: generate the test configuration object based on one or more axes-specific metrics, wherein the one or more axes-specific metrics are based on one or more axes value selected based on the parametric metric, as taught by Sholingar, in order to improve testing accuracy and learning.
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Nassar in view of Sholingar, and further in view of Bondor, et al. (U.S. Patent No. 10,482,003).
In regards to Claim 6, Nassar in view of Sholingar teaches the device of Claim 1, but Nassar in view of Sholingar does not specifically teach wherein the first scenario format comprises one or more of: an open scenario (OSC) definition language.
However, Bondor teaches wherein the first scenario format comprises one or more of: an open scenario (OSC) definition language (Bondor, Col. 19 Lines 11-23 – creating “a scenario file”, where the file format is for example “open file format openScenario”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified the device including the above limitations of Nassar in view of Sholingar to include wherein the first scenario format comprises one or more of: an open scenario (OSC) definition language, in order to use a scenario format that is “universal in a sense that any kind of scenario testing framework can be prepared to be able to process it and convert it into a valid scenario” (Bondor, Col. 19 Lines 11-23).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Rasche, et al. (U.S. Patent Application Pub. No. 2019/162293) teaches methods for identifying critical test cases in the context of highly automated driving, where the test cases are partially defined via the external parameter sets and used for simulating driving situations and/or environmental situations of traffic, combined and are collectively referred to as a scenario.
Hawthorne, et al. (U.S. Patent Application Pub. No. 2019/0179738) teaches a method for simulation testing an autonomy software include receiving, at processing circuitry, mission parameters indicative of a test mission, environmental parameters, and vehicle parameters and performing, by the processing circuitry, an adaptive search using a surrogate model of the autonomy software under test to selectively generate test scenarios for simulation, and clustering the plurality of test scenarios based on performance score metric values to determine performance boundaries for the autonomy software under test.
Venkatadri, et al. (U.S. Patent Application Pub. No. 2022/0194395) teaches systems and methods of the present disclosure are directed to generation and utilization of a vehicle testing knowledge structure for testing and validation of simulated autonomous vehicle performance, including obtaining a plurality of testing parameters for scenario simulation and determining a simulation accuracy value.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HELEN LI whose telephone number is (703)756-4719. The examiner can normally be reached Monday through Friday, from 9am to 5pm eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at (571) 272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.L./Examiner, Art Unit 3665
/HUNTER B LONSBERRY/Supervisory Patent Examiner, Art Unit 3665
Prosecution Insights