Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because figures 1-4 and 6-10 do not textually label the figures. The figures should be corrected to include both textual and numerical labeling for clarity and better understanding of the invention. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 14, 15, 24, and 25 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 14 and 15 mention “the one or more feasible test” while there is not antecedence for the “the one or more feasible test” in the claim it depends upon (claim 1). It is also unclear what “feasible” means in this context. The examiner will interpret this to mean more likely to occur in the real world. Therefore, correction is required.
Claims 24 and 25 mention “wherein the testing module” while there is not antecedence for the “wherein the testing module” in the claim it depends upon (claim 1). Therefore, correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2 and 14-26 are rejected under 35 U.S.C 101 because the claimed invention is directed to a judicial exception without significantly more.
Claim 1.
STEP 1: Yes. The claim recites a “method” which is a process.
STEP 2A PRONG ONE:
The claim recites mathematical abstractions and mental processes.
generating a design space comprising a plurality of test scenarios based on the variable parameter definitions, wherein the design space refers to a multidimensional combination and interaction of the variable parameters, wherein each combination of the variable parameters in the design space corresponds to a test scenario of the plurality of test scenarios;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the design space is made up of relationships of “a multidimensional combination and interaction of the variable parameters”.
generating at least one real-world scenario associated with the software-driven system based on the variable parameters, the generating of the at least one real-world scenario comprising pruning the design space by applying one or more constraints associated with the variable parameters, and wherein the pruned design corresponds to the at least one real-world scenario;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the real-world scenario is made up of relationships of “pruning the design space by applying one or more constraints associated with the variable parameters”.
identifying one or more test scenarios which are suitable for testing the software-driven system based on the at least one real-world scenario from the plurality of test scenarios by sampling the pruned design space using a trained machine learning model;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case using the machine learning model to find the suitable scenario is a series of calculations.
evaluating a behaviour of the software-driven system by applying the identified test scenarios on a model of the software-driven system in the simulated environment; and
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation to find the desired behaviour from the different scenarios.
validating the behaviour of the software-driven system in the real-world scenario based on outcome of the evaluation.
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation based on the outcomes of the scenario test.
STEP 2A PRONG TWO: The claim does not integrate the exception into a practical application.
STEP 2B: The claim does not recite an inventive concept or significantly more than the exception.
obtaining, by a processor, a plurality of test scenarios that correspond to testing of the software-driven system, the test scenarios being in a form of variable parameter definitions from a source, wherein the source is one of an input device, a user device, or a database and wherein the variable parameters include at least one of attributes and process parameters associated with the software-driven system;
MPEP 2106.05(g) – This is pre solution data gathering activity.
by a processor
MPEP 2106.05(f) – This is generic computer components used to implement the abstract idea.
generating a simulated environment representing the at least one real-world scenario in which the software-driven system is to be tested, using a simulation tool in the computer-Aided Engineering Environment, wherein simulated agents in the simulated environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate;
MPEP 2106.05(f) – This is using generic computer components as a tool to apply the abstract idea.
Conclusion: Claim 1 is directed to mental processes and mathematical abstractions, not integrated into a practical application and lacks an inventive concept. Therefore, it is ineligible under 35 U.S.C 101.
Regarding Claims 14, 15, and 17:
These claims merely narrow the abstract idea. Claims 14 and 15 add further mathematical detail to the sampling step generating samples by feeding the pruned design space to the trained machine learning model (claim 14) and generating optimal samples based on an optimization criterion (claim 15). Claim 17 narrows the validation step which remains a mental process. This does not integrate the judicial exception into a practical application. The claims do not resolve the issues from the claims they depend upon.
Regarding Claims 2 and 16:
These claims merely add generic computer implemented activity used as a tool to apply the abstract idea. Claim 2 add building the model of the system within environment and claim 16 adds instantiating and running simulation instances on that model and analyzing the resulting outputs. Building and running a model on a generic simulation tool is using a computer as a tool to carry out the abstract idea MPEP 2106.05 (f) and generally links the idea to a computer aided engineering field of use MPEP 2106.05 (h) the analysis step is itself part of the abstract mental process evaluation. This does not integrate the judicial exception into a practical application. The claims do not resolve the issues from the claims they depend upon.
Regarding Claim 18 and 25:
This claim merely adds insignificant post solution activity, of outputting a notification of the validation outcome on a GUI MPEP 2106.05 (g). This does not integrate the judicial exception into a practical application. The claims do not resolve the issues from the claims they depend upon.
Regarding Claim 24:
Claim 24 narrows the validation to a comparison of the behavior agent against an expected standard, a mental process judgment. This does not integrate the judicial exception into a practical application. The claim does not resolve the issues from the claim it depends upon.
Claim 19.
STEP 1: Yes. The claim recites a “system” which is a manufacture.
STEP 2A PRONG ONE:
The claim recites mathematical abstractions and mental processes.
generate a design space comprising a plurality of test scenarios based on the variable parameter definitions, wherein the design space refers to a multidimensional combination and interaction of the variable parameters, wherein each combination of the variable parameters in the design space corresponds to a test scenario of the plurality of test scenarios;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the design space is made up of relationships of “a multidimensional combination and interaction of the variable parameters”.
generate at least one real-world scenario associated with the software-driven system based on the variable parameters, the generating of the at least one real-world scenario comprising pruning the design space by applying one or more constraints associated with the variable parameters, and wherein the pruned design corresponds to the at least one real-world scenario;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the real-world scenario is made up of relationships of “pruning the design space by applying one or more constraints associated with the variable parameters”.
identify one or more test scenarios which are suitable for testing the software-driven system based on the at least one real-world scenario from the plurality of test scenarios by sampling the pruned design space using a trained machine learning model;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case using the machine learning model to find the suitable scenario is a series of calculations.
evaluate a behaviour of the software-driven system by applying the identified test scenarios on a model of the software-driven system in the simulated environment; and
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation to find the desired behaviour from the different scenarios.
validate the behaviour of the software-driven system in the real-world scenario based on outcome of the evaluation.
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation based on the outcomes of the scenario test.
STEP 2A PRONG TWO: The claim does not integrate the exception into a practical application.
STEP 2B: The claim does not recite an inventive concept or significantly more than the exception.
obtain, by a processor, a plurality of test scenarios that correspond to testing of the software-driven system, the test scenarios being in a form of variable parameter definitions from a source, wherein the source is one of an input device, a user device, or a database and wherein the variable parameters include at least one of attributes and process parameters associated with the software-driven system;
MPEP 2106.05(g) – This is pre solution data gathering activity.
one or more processors … a memory communicatively coupled …
MPEP 2106.05(f) – This is generic computer components used to implement the abstract idea.
generate a simulated environment representing the at least one real-world scenario in which the software-driven system is to be tested, using a simulation tool in the computer-Aided Engineering Environment, wherein simulated agents in the simulated environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate;
MPEP 2106.05(f) – This is using generic computer components as a tool to apply the abstract idea.
Conclusion: Claim 19 is directed to mental processes and mathematical abstractions, not integrated into a practical application and lacks an inventive concept. Therefore, it is ineligible under 35 U.S.C 101.
Regarding Claim 20:
This claim merely adds generic computer hardware. A user computer coupled to the processor and cloud computing system. A general purpose/cloud computer is not a particular machine MPEP 2106.05 (b) and is recited only as a tool to apply the abstract idea MPEP 2106.05 (f), generally linking it to a cloud computing field of use MPEP 2106.05 (h). This does not integrate the judicial exception into a practical application. The claim does not resolve the issues from the claim it depends upon.
Regarding Claim 21 and 22:
This claim merely narrows the abstract idea. Claims 21 and 22 add further mathematical detail to the sampling step generating samples by feeding the pruned design space to the trained machine learning model (claim 21) and generating optimal samples based on an optimization criterion (claim 22). This does not integrate the judicial exception into a practical application. The claims do not resolve the issues from the claims they depend upon.
Regarding Claim 23:
This claim merely narrows the abstract idea. Claim 23 merely adds more detail to the mental process in analyzing and identifying. This does not integrate the judicial exception into a practical application. The claim does not resolve the issues from the claim it depends upon.
Claim 26.
STEP 1: Yes. The claim recites a “non-transitory computer readable storage medium” which is a manufacture.
STEP 2A PRONG ONE:
The claim recites mathematical abstractions and mental processes.
generate a design space comprising a plurality of test scenarios based on the variable parameter definitions, wherein the design space refers to a multidimensional combination and interaction of the variable parameters, wherein each combination of the variable parameters in the design space corresponds to a test scenario of the plurality of test scenarios;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the design space is made up of relationships of “a multidimensional combination and interaction of the variable parameters”.
generate at least one real-world scenario associated with the software-driven system based on the variable parameters, the generating of the at least one real-world scenario comprising pruning the design space by applying one or more constraints associated with the variable parameters, and wherein the pruned design corresponds to the at least one real-world scenario;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case the real-world scenario is made up of relationships of “pruning the design space by applying one or more constraints associated with the variable parameters”.
identify one or more test scenarios which are suitable for testing the software-driven system based on the at least one real-world scenario from the plurality of test scenarios by sampling the pruned design space using a trained machine learning model;
This is a mathematical abstraction that can be a calculation, relationship, or equation/formula. In this case using the machine learning model to find the suitable scenario is a series of calculations.
evaluate a behaviour of the software-driven system by applying the identified test scenarios on a model of the software-driven system in the simulated environment; and
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation to find the desired behaviour on the different scenarios.
validate the behaviour of the software-driven system in the real-world scenario based on outcome of the evaluation.
This describes an observation, evaluation, judgment or opinion that can be done in the mind or with the aid of pen and paper. In this case an evaluation based on the outcomes of the scenario test.
STEP 2A PRONG TWO: The claim does not integrate the exception into a practical application.
STEP 2B: The claim does not recite an inventive concept or significantly more than the exception.
obtain a plurality of test scenarios that correspond to testing of the software-driven system, the test scenarios being in a form of variable parameter definitions from a source, wherein the source is one of an input device, a user device, or a database and wherein the variable parameters include at least one of attributes and process parameters associated with the software-driven system;
MPEP 2106.05(g) – This is pre solution data gathering activity.
a non-transitory computer readable storage medium having instructions …
MPEP 2106.05(f) – This is generic computer components used to implement the abstract idea.
generate a simulated environment representing the at least one real-world scenario in which the software-driven system is to be tested, using a simulation tool in the computer-Aided Engineering Environment, wherein simulated agents in the simulated environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate;
MPEP 2106.05(f) – This is using generic computer components as a tool to apply the abstract idea.
Conclusion: Claim 26 is directed to mental processes and mathematical abstractions, not integrated into a practical application and lacks an inventive concept. Therefore, it is ineligible under 35 U.S.C 101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 14, 16-19, 21, and 23-26 are rejected under 35 U.S.C 103 as being unpatentable over HAWTHORNE et al US20190179738A1 (2019), MARTINEZ et al US20170147719A1 (2017), and MELTZ et al WO2019234726A1 (2019).
Regarding Claim 1, HAWTHORNE teaches
A computer-implemented method for validating a software-driven system based on real-world scenarios in a Computer-Aided Engineering environment, the method comprising:
“Determining Performance of Autonomy Decision-Making Engines”. (Title).
“An example method for simulation testing an autonomy software is provided… The method may further include ranking the plurality of test scenarios based on a respective distance to a performance boundary to identify test scenarios of interest for modification of the autonomy software or real-world field testing of an autonomous vehicle.”. (Abstract).
“The real-world test implementation system 470 may be further configured to apply the subset of the plurality of test scenarios to an instance of the autonomy software operating on the real-world test implementation system to validate the autonomy software via in-field, real-world testing.”. (0086).
This shows a validation software system based on real world scenarios.
obtaining, by a processor, a plurality of test scenarios that correspond to testing of the software-driven system, the test scenarios being in a form of variable parameter definitions from a source, wherein the source is one of an input device, a user device, or a database and wherein the variable parameters include at least one of attributes and process parameters associated with the software-driven system;
“In this regard, FIG. 1 illustrates an example portion of a state space 100 in which a testing scenario may be run (or tested against). The portion of the state space 100 shown in FIG. 1 may be defined by physical, geographical parameters of the state space 100, while additional parameters that define the state space 100 may be in other dimensionalities and therefore not visualized in FIG. 1. In this regard, this portion of the state space 100, for example, may be defined with respect to geographic axes that may be referenced to indicate the position of items relative to an origin.”. (0026).
“In an autonomy test, a test scenario may be defined with respect to a set of parameters within the state space. The parameters may be fixed (i.e., the same in all scenarios) or the parameters may be dynamic (i.e., a parameter may be different in at least some scenarios). A test scenario may be defined with respect to a set of parameters that include parameters that are constant and parameters that may differ in value in other scenarios. Various parameters may be defined that may include static or fixed parameters and dynamic parameters. In this regard, for example, environmental parameters may be defined. Environmental parameters may define obstacles or conditions in the state space 100 that may impact the decisions made by the autonomy. Examples of environmental parameters may include the static or fixed locations of obstacles (e.g., buildings, trees, vehicles, pedestrians, buoys, etc.) and conditions (e.g., weather conditions, such as, temperature, rainy, windy, etc., time of day (e.g., indicating light or dark), tides, currents, etc.). For example, in the state space 100, obstacles 102, 104, and 106 may be defined by environmental parameters, with obstacle 102 being a moving obstacle and obstacles 104 and 106 being fixed position obstacles.”. (0027).
“In an example embodiment, the memory 430 may include one or more non-transitory memory devices such as, for example, volatile or non-volatile memory that may be either fixed or removable. The memory 430 may be configured to store information, data, applications, instructions or the like for enabling, for example, test scenario simulations and the like to carry out various functions in accordance with example embodiments. For example, the memory 430 could be configured to buffer input data for processing by the processing circuitry 410. Additionally or alternatively, the memory 430 could be configured to store instructions for execution by the processing circuitry 410. Among the contents of the memory 430, applications may be stored for execution by the processing circuitry 410 in order to carry out the functionality associated with each respective application.”. (0077).
This shows a processor with parameters for the test scenario from a data source that is associated with the software system.
generating a design space comprising a plurality of test scenarios based on the variable parameter definitions, wherein the design space refers to a multidimensional combination and interaction of the variable parameters, wherein each combination of the variable parameters in the design space corresponds to a test scenario of the plurality of test scenarios;
“Additionally, a scenario input state or test scenario may be defined as the vector X=[x1, x2, . . . xn] where ∀i∈n: xi∈xi. The scenario may be a specific instantiation of each parameter from their corresponding state space range. Thus, the state space may consists of all the possible scenario configurations that could be tested. A sample set of N states may be defined as XN=[X1 . . . XN]. The normalized state vector where each x i ∈[0, 1] is defined as x.”. (0046).
Hawthorne build a state space that is the full set of all possible scenario configurations, where each scenario is multidimensional vector of parameters of combinations for the test scenario.
generating at least one real-world scenario associated with the software-driven system based on the variable parameters,
“In an autonomy test, a test scenario may be defined with respect to a set of parameters within the state space. The parameters may be fixed (i.e., the same in all scenarios) or the parameters may be dynamic (i.e., a parameter may be different in at least some scenarios). A test scenario may be defined with respect to a set of parameters that include parameters that are constant and parameters that may differ in value in other scenarios. Various parameters may be defined that may include static or fixed parameters and dynamic parameters. In this regard, for example, environmental parameters may be defined. Environmental parameters may define obstacles or conditions in the state space 100 that may impact the decisions made by the autonomy. Examples of environmental parameters may include the static or fixed locations of obstacles (e.g., buildings, trees, vehicles, pedestrians, buoys, etc.) and conditions (e.g., weather conditions, such as, temperature, rainy, windy, etc., time of day (e.g., indicating light or dark), tides, currents, etc.).”. (0027).
“The states may be received by the scenario generator 208 to generate scenario files for a test scenario (or a batch of test scenarios) to be passed to the simulation manager 212.”. (0042).
Hawthorne generates test scenarios from the mission and environments of the real-world scenarios associated with the software system based on parameters.
evaluating a behaviour of the software-driven system by applying the identified test scenarios on a model of the software-driven system in the simulated environment; and
“The method may further include performing, by the processing circuitry, an adaptive search using a surrogate model of the autonomy software under test to selectively generate test scenarios for simulation, and clustering the plurality of test scenarios based on performance score metric values to determine performance boundaries for the autonomy software under test.”. (Abstract).
This shows evaluating the tested system by simulation via test scenario models.
identifying one or more test scenarios which are suitable for testing the software-driven system based on the at least one real-world scenario from the plurality of test scenarios by sampling the pruned design space using a trained machine learning model;
“For adaptive sampling, the meta-model evaluators may be used to select the subsequent batch of samples based on the set of queries with the highest expected information gain, as indicated in Algorithm 1. According to some example embodiments, the methods may retrain the meta-model evaluator at every iteration…”. (0068).
“Since the Gaussian process scales with O(n3) and the k-nearest neighbors algorithm scales with O(kn log n), these approaches may offer improved scaling as the number of dimensions and the required number of samples increases. These meta-model evaluators may be defined as
(X), where they take existing samples as inputs and return the expected information gain of a proposed query as an output”. (0065).
Hawthorne teaches identifying test scenarios by sampling the design space using a trained GPR model which is a machine learning technique.
HAWTHORNE does not explicitly teach but MARTINEZ teaches
the generating of the at least one real-world scenario comprising pruning the design space by applying one or more constraints associated with the variable parameters, and wherein the pruned design corresponds to the at least one real-world scenario;
“The present disclosure is directed, in general, to simulation software for the modeling and analysis of multi-domain systems (e.g., hydraulic, pneumatic, thermal, electric and/or mechanical systems), computer-aided design (CAD), computer-aided engineering (CAE), visualization, and manufacturing systems, product data management (PDM) systems, product lifecycle management (PLM) systems, Application Lifecycle Management systems (ALM), and similar systems, that are used to create, use, and manage data for products and other items (collectively referred to herein as product systems).”. (0001).
“Algorithm 2 may be configured to determine the behaviors (M.sub.Ci) 136 of each component model solution M.sub.Ci 116 in the collection C.sub.i 146 and prune out solutions having behaviors that are inconsistent with the behaviors (M.sub.F) 134 associated with the functional model M.sub.F 108. The output of Algorithm 2 is a pruned collection C.sub.o 132 of component model solutions.”. (0050).
“Thus if the functional model 108 specifies behavior data 134 that requires a minimum number of degrees of freedom of motion, the described design space pruning module 124 may prune out component model solutions that do not meet this required threshold to form the pruned collection 132.”. (0051).
Martinez shows pruning designs spaces by applying constraints and the resulting pruned space corresponds to a real-world scenario.
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of MARTINEZ’s application of constraints on a design space with HAWTHORNE’s software driven systems of real-world scenarios. The motivation for doing so would have been to create a more accurate scenario system via constraining data as stated by MARTINEZ “the described design space pruning module 124 may prune out component model solutions that do not meet this required threshold to form the pruned collection 132.”. (0051).
HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
generating a simulated environment representing the at least one real-world scenario in which the software-driven system is to be tested, using a simulation tool in the computer-Aided Engineering Environment, wherein simulated agents in the simulated environment are configured to represent agents in real-world conditions in which the software-driven system is expected to operate;
“The simulation framework 270 may also include a computer software of a road network 272 of, for example, a city, cities, region or country, which may correspond to real-life road network and intersections 105, 134, 135, 136. The simulation framework 270 may also include a software model 267 of various objects and obstacles, which may correspond to real-life vehicles, pedestrians and obstacles 140, 142, 144, 148, 150, among numerous others, as well as possibly including light and weather conditions 151.
”. (Pg. 28-29).
“Verification system 210 may be configured to define a particular. Math. test scenario by sending scenario configuration information 214 to the simulation framework 270. For example, it may configure the number N and type of autonomous vehicles 257, 259, the layout and characteristics of the road network 272, etc. System 210 may be configured to send to the AUT 220 mission objectives 212, e.g. endpoints for each simulated autonomous vehicle within the computer simulation (corresponding e.g. to real life endpoints 125, 127, 129). Once testing has begun, the simulation framework 270 and the AUTs may begin to interact, in a fashion similar to that in which the real-world environment 100 interacts with the AUTs”. (Pg. 29-30).
“Once testing has begun, the simulation framework 270 and the AUTs may begin to interact, in a fashion similar to that in which the real-world environment 100 interacts with the AUTs.”. (Pg. 30).
MELTZ teaches building a simulation environment (generating) with a simulation framework (CAE) and populates it with “agents” in this case vehicles that correspond to real world counterparts.
validating the behaviour of the software-driven system in the real-world scenario based on outcome of the evaluation.
“In step 720, these results of the scenario may be measured, and graded against the S-PAFs and F-PAFs. They may be measured against a set or sets of criteria indicative of algorithm performance requirements and of algorithm safety requirements, that is against a set or sets of algorithm verification test criteria. It may be determined whether the performance of the scenario meets the various scoring criteria.”. (Pg. 55).
“This process may end at step 745. It may be followed, for example, by step 450 of Fig. 4, in which the statistical results may be compared against the set or sets of statistical criteria for successful AUT, so as to determine whether the set of second results meet the set or sets of statistical criteria.”. (Pg. 56).
This shows running the scenarios and grading it against verification metrics / criterion to determine if they are met. i.e. validating the system behavior based on the outcome.
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of MELTZ’S validating method for software real-world scenarios with the combination of HAWTHORNE-MARTINEZ’s software driven systems of real-world scenarios. The motivation for doing so would have been to create a more robust system via evaluation and validation. As stated by MELTZ “…provide at least one set of criteria indicative of algorithm performance requirements and of algorithm safety requirements, constituting at least one set of algorithm verification test criteria…”. (Pg. 12).
Regarding Claim 2, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The method according to claim 1, further comprising: generating the model of the software-driven system in the Computer-Aided Engineering environment.
“A computerized simulation framework270 of the simulated environment may include a computerized model, or models, of the simulated environment in which the vehicles move, of the simulated vehicles and/or of the simulated sensors. It may contain computer software simulation models of numerous fixed sensors numbered 1 to M, 255, 256. These may correspond, for example, to real-life fixed sensors 180, 182, 184. The simulation framework 270 may also contain computer software models of autonomous vehicles numbered 1 to N, 257, 259. These may correspond, for example, to real life autonomous vehicles 110, 116, 120. Note that simulated autonomous vehicles 257, 259 may include, in some cases, both simulation models of the vehicles, as well as simulation models of sensors of each vehicle. Note also that simulation models of the vehicles may in some cases also include simulation of their communication to other autonomous vehicles and to management system 188. The simulation framework 270 may also include a computer software of a road network 272 of, for example, a city, cities, region or country, which may correspond to real-life road network and intersections 105, 134, 135, 136. The simulation framework 270 may also include a software model 267 of various objects and obstacles, which may correspond to real-life vehicles, pedestrians and obstacles 140, 142, 144, 148, 150, among numerous others, as well as possibly including light and weather conditions 151”. (Pg. 28-29).
This shows generating a model of a vehicle that is used inside the software driven simulation framework (CAE).
Regarding Claim 14, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The method according to claim 1, wherein identifying the one or more feasible test scenarios for testing the software-driven system comprises: generating samples from the pruned design space by feeding an input corresponding to the pruned design space to the trained machine learning model; and
“Note that methods 400 and 700 have been described above in terms of a use to verify a previously-developed algorithm. In some example cases, these methods, or parts of them, and the systems that run them, may also be usable in the algorithm development process itself, possibly in early stages of that process. For example, the AUT may in some cases make use of machine learning. In such cases, computerized runs of methods 400 and 700 may be used for purposes of data acquisition. For example, the first results and/or the second results may serve as inputs to the machine learning. The machine learning in the algorithm may be able to learn from both successful and failed test scenarios, and thus update and in some cases correct and/or improve the algorithm itself.”. (Pg. 56).
This shows inputs fed into the trained model which generates the next set of samples by optimizing the machine learning algorithm i.e. pruning the design space via parameter optimization.
determining the one or more feasible test scenarios based on the generated samples, wherein each of the samples comprises values of the variable parameters to be used in a specific test scenario of the plurality of test scenarios.
“Note that methods 400 and 700 have been described above in terms of a use to verify a previously-developed algorithm. In some example cases, these methods, or parts of them, and the systems that run them, may also be usable in the algorithm development process itself, possibly in early stages of that process. For example, the AUT may in some cases make use of machine learning. In such cases, computerized runs of methods 400 and 700 may be used for purposes of data acquisition. For example, the first results and/or the second results may serve as inputs to the machine learning. The machine learning in the algorithm may be able to learn from both successful and failed test scenarios, and thus update and in some cases correct and/or improve the algorithm itself. Turning now to Fig, 8, it illustrates one example of a generalized flow chart diagram of a methodology 800 for additional testing of AUT(s) 220, in accordance with certain embodiments of the presently disclosed subject matter. The additional testing of 800 is optional, and provides example methods that may be used in some example cases to provide additional verification of the AUT or AUTs 220. The process starts at step 803. The first stage of this optional additional methodology may involve a replay of a sub-set of the verification test scenario set generated in 705, to provide additional validation of the scenario results. In some examples. Replay Analyzer 368, possibly a part of verification module 340 running on processor 325, may retrieve the set of second results, including the grades of all verification test scenarios, from 374.”. (Pg. 56).
This shows getting results from testing the scenarios in order to see which pass the intended grades to iteratively improve the system.
Regarding Claim 16, HAWTHORNE teaches
The method according to claim 1, wherein evaluating the behavior of the software-driven system by applying the identified test scenarios on the model of the software-driven system in the simulated environment comprises:
generating simulation instances for testing the software-driven system based on the identified one or more test scenarios;
“The states may be received by the scenario generator 208 to generate scenario files for a test scenario (or a batch of test scenarios) to be passed to the simulation manager 212.”. (0042).
“…may take scenarios or scenario states from the test-generation system as an input and convert the scenario states into scenario files that may be read by a simulator.”. (0039).
This shows generating simulations for testing one or more scenarios.
executing the simulation instances based on the model of the software-driven system to generate simulation results; and
“The simulation manager 212 may be configured to receive and manage the simulation of the test scenarios described in the scenario files. In this regard, the simulation manager 212 may employ a computer cluster (e.g., processing circuitry) to perform the simulations on an autonomy under test. Depending on the processing power, multiple simulations of the autonomy under test may be implemented to, for example, perform parallel simulation runs to increase efficiency. The results of the simulation runs may be passed to a scoring component 210 to convert the results into desired scores (e.g., values for performance score metrics).”. (0042).
“a job scheduler may manage a transfer of scenario files, launch the simulations on a computing cluster (e.g., processing circuitry), and retrieve results from completed runs. Such jobs may be submitted in batches tailored to the size and speed of the computing cluster. After the simulations are complete the results may be scored and returned to the test-generation system to assist in further selection of the test scenarios.”. (0039).
This shows running the simulations to generate results.
analyzing the simulation results to determine the behavior of the software- driven system in the real-world scenario.
“The method may further include ranking the plurality of test scenarios based on a respective distance to a performance boundary to identify test scenarios of interest for modification of the autonomy software or real-world field testing of an autonomous vehicle.”. (Abstract).
“The simulation manager 212 may be configured to receive and manage the simulation of the test scenarios described in the scenario files. In this regard, the simulation manager 212 may employ a computer cluster (e.g., processing circuitry) to perform the simulations on an autonomy under test. Depending on the processing power, multiple simulations of the autonomy under test may be implemented to, for example, perform parallel simulation runs to increase efficiency. The results of the simulation runs may be passed to a scoring component 210 to convert the results into desired scores (e.g., values for performance score metrics).”. (0042).
“a job scheduler may manage a transfer of scenario files, launch the simulations on a computing cluster (e.g., processing circuitry), and retrieve results from completed runs. Such jobs may be submitted in batches tailored to the size and speed of the computing cluster. After the simulations are complete the results may be scored and returned to the test-generation system to assist in further selection of the test scenarios.”. (0039).
“The adaptive search 216 may also provide information about the test scenarios, including the performance scoring metrics of the simulated scenarios, for clustering by the boundary identification 218 as further described below. The boundary identification 218 may provide outputs as test scenario recommendations 222 for use in, for example, real-world application in an autonomous vehicle. Further, the test scenario recommendations 222 may include definitions of the determined performance boundaries, and ranking of the test scenarios.”. (0043).
This shows analyzing the tested scenarios to use the evaluation to determine real world behavior.
Regarding Claim 17, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The method according to claim 1, wherein validating the behavior of the software-driven system in the real-world scenario based on the outcome of the evaluation comprises: determining whether the behavior of the software-driven system in the real- world scenario meets an expected standard.
“In step 720, these results of the scenario may be measured, and graded against the S-PAFs and F-PAFs. They may be measured against a set or sets of criteria indicative of algorithm performance requirements and of algorithm safety requirements, that is against a set or sets of algorithm verification test criteria. It may be determined whether the performance of the scenario meets the various scoring criteria. Grades may be evaluated for each individual autonomous vehicle, and for the scenario as a whole. This may be done, in some examples, by performance evaluator/grader 360, which may be part of verification module 340 running on processor 325.”. (Pg. 55).
“This process may end at step 745. It may be followed, for example, by step 450 of Fig. 4, in which the statistical results may be compared against the set or sets of statistical criteria for successful AUT, so as to determine whether the set of second results meet the set or sets of statistical criteria.”. (Pg. 56).
This shows evaluating real world scenarios by grading the results to see if they meet a standard.
Regarding Claim 18, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The method according to claim 1, further comprising: generating a notification indicating an outcome of the validation on a Graphical User Interface.
“In some cases, the replaying system may output the replay results to a graphical interface. In the example architecture of Fig. 3b, replay analyzer 368 may replay the third results via user outputs module 364 to output devices 338, which may display the particular scenario of autonomous vehicle movement graphically, e.g on 2-D or 3-D map representation, possibly also with accompanying graphs and tables in the case of a graphical interface, the decision step 820 of whether the results are valid may in some examples be performed partly or entirely by a human operator, who monitors the graphic displays.”. (Pg. 59).
This shows generating a notification via a graphical interface such as a chart or some other graphical representation for the operator to do analyses.
Claim 19 recites sustainably the same limitations as claim 1 except this claim is directed to a “system”. Therefore this, claim is rejected for the same rationale as addressed above.
Claim 21 recites sustainably the same limitations as claim 14 except this claim is directed to a “system”. Therefore this, claim is rejected for the same rationale as addressed above.
Claim 23 recites sustainably the same limitations as claim 16 except this claim is directed to a “system”. Therefore this, claim is rejected for the same rationale as addressed above.
Regarding Claim 24, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The system of claim 1, wherein the testing module configures the one or more processors to, for the validation of the behavior of the software-driven system in the real-world scenario based on the outcome of the evaluation, determine when the behavior of the software-driven system in the real-world scenario meets an expected standard.
“The term “computer” should be expansively construed to cover any kind of hardware-based electronic device with data processing capabilities, including, by way of non-limiting example, a personal computer, a server, a computing system, a communication device, a processor or processing unit (e.g. digital signal processor (DSP), a microcontroller, a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), any other electronic computing device, including, by way of non-limiting example, tire processing circuitry therein, such as for example the processing circuitry 320 (further detailed herein with regard to Fig. 3A), disclosed in the present application. The operations in accordance with the teachings herein may be performed by a computer specially constructed for the desired purposes, or by a general-purpose computer specially configured for the desired purpose by a computer program stored in a non-transitory computer-readable storage medium.”. (Pg. 19-20).
“In step 720, these results of the scenario may be measured, and graded against the S-PAFs and F-PAFs. They may be measured against a set or sets of criteria indicative of algorithm performance requirements and of algorithm safety requirements, that is against a set or sets of algorithm verification test criteria. It may be determined whether the performance of the scenario meets the various scoring criteria. Grades may be evaluated for each individual autonomous vehicle, and for the scenario as a whole. This may be done, in some examples, by performance evaluator/grader 360, which may be part of verification module 340 running on processor 325.”. (Pg. 55).
“This process may end at step 745. It may be followed, for example, by step 450 of Fig. 4, in which the statistical results may be compared against the set or sets of statistical criteria for successful AUT, so as to determine whether the set of second results meet the set or sets of statistical criteria.”. (Pg. 56).
This shows evaluating real world scenarios by grading the results to see if they meet a standard. This outcome evaluation is run on microprocessors.
Regarding Claim 25, HAWTHORNE and MARTINEZ do not explicitly teach but MELTZ teaches
The system of claim 1, wherein the testing module further configures the one or more processors to generate a notification indicating an outcome of the validation on a Graphical User Interface.
“The term “computer” should be expansively construed to cover any kind of hardware-based electronic device with data processing capabilities, including, by way of non-limiting example, a personal computer, a server, a computing system, a communication device, a processor or processing unit (e.g. digital signal processor (DSP), a microcontroller, a microprocessor, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.), any other electronic computing device, including, by way of non-limiting example, tire processing circuitry therein, such as for example the processing circuitry 320 (further detailed herein with regard to Fig. 3A), disclosed in the present application. The operations in accordance with the teachings herein may be performed by a computer specially constructed for the desired purposes, or by a general-purpose computer specially configured for the desired purpose by a computer program stored in a non-transitory computer-readable storage medium.”. (Pg. 19-20).
“In some cases, the replaying system may output the replay results to a graphical interface. In the example architecture of Fig. 3b, replay analyzer 368 may replay the third results via user outputs module 364 to output devices 338, which may display the particular scenario of autonomous vehicle movement graphically, e.g on 2-D or 3-D map representation, possibly also with accompanying graphs and tables in the case of a graphical interface, the decision step 820 of whether the results are valid may in some examples be performed partly or entirely by a human operator, who monitors the graphic displays.”. (Pg. 59).
This shows generating a notification via a graphical interface such as a chart or some other graphical representation for the operator to do analyses. This system that generates a GUI with a notification is ran on a processor and a non-transitory computer-readable storage medium.
Claim 26 recites sustainably the same limitations as claim 1 except this claim is directed to a “non-transitory computer readable storage medium”. Therefore this, claim is rejected for the same rationale as addressed above.
Claims 15 and 22 are rejected under 35 U.S.C 103 as being unpatentable over HAWTHORNE et al US20190179738A1 (2019), MARTINEZ et al US20170147719A1 (2017), MELTZ et al WO2019234726A1 (2019), and CAHOON WO2018071708A1 (2018).
Regarding Claim 15, HAWTHORNE, MARTINEZ, and MELTZ do not explicitly teach but CAHOON teaches
The method according to claim 14, further comprising: generating optimal samples from the pruned design space based on at least one optimization criterion; and determining the one or more feasible test scenarios based on the generated optimal samples.
“Such a domain specific language reduces computational complexity, memory requirements, and processing time by optimizing over specific scenarios which provide useful information for validation and testing.”. (0019).
“In at least one example, variables included in the outer product can be optimized to limit which scenarios are created for validation and testing.”. (0053).
“In one example, the domain specific language can be used to define outcomes of the scenario that are useful so that scenarios that are unlikely to be useful can be filtered out. In this way, the domain specific language can be used to determine a limited number of scenarios to simulate which will provide useful information. The scenarios determined to provide useful information can then be instantiated in the simulation environment.”. (0019).
This shows choosing optimal samples that would be useful in the design space i.e. feasible (likely to occur) by pruning unlikely ones for the test scenarios.
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of CAHOON’s determination of feasible scenarios with the combination of HAWTHORNE-MARTINEZ-MELTZ’s software driven systems of real-world scenarios. The motivation for doing so would have been to create a more robust system. As stated by CAHOON “The domain specific language can be executable by a computing system to instantiate scenarios in a simulation environment. In one example, the domain specific language can be used to define outcomes of the scenario that are useful so that scenarios that are unlikely to be useful can be filtered out.”. (0019).
Claim 22 recites sustainably the same limitations as claim 15 except this claim is directed to a “system”. Therefore this, claim is rejected for the same rationale as addressed above.
Claims 20 is rejected under 35 U.S.C 103 as being unpatentable over HAWTHORNE et al US20190179738A1 (2019), MARTINEZ et al US20170147719A1 (2017), MELTZ et al WO2019234726A1 (2019), and AICHELE et al US9811074B1 (2017).
Regarding Claim 20, HAWTHOREN, MARTINEZ, and MELTZ does not explicitly teach but AICHELE teaches
The system of claim 19, further comprising: a user computer communicatively coupled to the processor, the processor comprising a cloud-computing system.
“The cloud may be formed, for example, by a network of web servers that include a plurality of computing devices, such as the computer system 600, with each server (or at least a plurality thereof) providing processor and/or storage resources. These servers may manage workloads provided by multiple users (e.g., cloud resource customers or other users). Typically, each user places workload demands upon the cloud that vary in real-time, sometimes dramatically. The nature and extent of these variations typically depends on the type of business associated with the user.”. (Pg. 16-17).
This shows a computer coupled to a processor and a cloud computing system.
It would have been obvious before the effective filing date of the claimed invention to incorporate the teachings of AICHELE’s cloud computing system with the combination of HAWTHORNE-MARTINEZ-MELTZ’s software driven systems of real-world scenarios. The motivation for doing so would have been to create a more robust system with more access to compute. As stated by AICHELE “Systems that provide cloud-based resources may be utilized exclusively by their owners or such systems may be accessible to outside users who deploy applications within the computing infrastructure to obtain the benefit of large computational or storage resources.”. (Pg. 16).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US-11030364-B2 teaches machine learning techniques to analyze performance of autonomous vehicle algorithms on real world and simulated data.
US-11436484-B2 teaches testing, and verifying autonomous machines using simulated environments.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NARCISO EDUARDO MONTES whose telephone number is (571)272-5773. The examiner can normally be reached Mon-Fri 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, REHANA PERVEEN can be reached at (571) 272-3676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.E.M./Examiner, Art Unit 2189
/REHANA PERVEEN/Supervisory Patent Examiner, Art Unit 2189
Prosecution Insights