Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Status
This instant application No. 18/339212 has Claims 1-20 pending.
Priority / Filing Date
Applicant claimed priority from U.S. provisional application No. 63/366,771. The priority filing date of this application is June 21, 2022.
Information Disclosure Statement
As required by M.P.E.P. 609(C), the Applicant’s submissions of the Information Disclosure Statements dated October 9, 2024 is acknowledged by the Examiner and the cited references have been considered in the examination of the claims now pending. As required by M.P.E.P. 609 C(2), a copy of each of the PTOL-1449s initialed and dated by the Examiner is attached to the instant Office action.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
3. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 2A Prong One:
Independent claim 1 recite:
Defining one or more test cases for a vehicle stack based on system requirements;
linking, the one or more test cases to one or more parameterized scenarios, wherein the one or more parameterized scenarios comprise a plurality of parameter permutations; and
testing the vehicle stack using the one or more test cases and the one or more parameterized scenarios.
Independent claim 9 recite:
generate a simulation environment for testing or validating a vehicle,
impose a runtime constraint on simulation properties of the simulation environment.
Independent claim 15 recite:
generate a simulation environment for testing or validating an autonomous vehicle,
sample a subset of scenarios from a group of possible scenarios to test.
All of the above limitations of Claims 1, 9 and 15 are recited in a broad manner and can be considered process steps that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Other than reciting “a memory”, “a processor operatively coupled to a memory” and “machine-learning” in the claims nothing in the claim elements precludes the steps from practically being performed in the mind. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, then it falls within the “mental processes” grouping of abstract ideas. As such claims 1, 9, and 15 recite an abstract idea.
Step 2A Prong Two:
This judicial exception is not integrated into a practical application. The claims recite the additional element of “a memory”, “a processor operatively coupled to a memory” and “machine-learning” to perform the claimed steps at a high level of generality such that it amounts to no more than mere instructions to apply the exception using a generic computer component. This additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B:
Claims 1, 9, and 15 only recite “a memory”, “a processor operatively coupled to a memory” and “machine-learning” to perform the claimed steps and therefore only recite a general purpose computer rather than a specific machine under MPEP 2106.05(b), and are directed to mere instructions to apply the exception under MPEP 2106.05(f), and do not result in anything significantly more than the judicial exception. The additional elements have been considered both individually and as an ordered combination in the significantly more consideration. The inclusion of the computer or memory and controller to perform the selecting, generating and testing steps amount to nor more than mere instructions to apply the exception using generic computer components. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Claims 1, 9, and 15 are not patent eligible.
The dependent claims include the same abstract ideas recited in the independent claims, and merely incorporate additional details that narrow the abstract ideas and fail to add significantly more to the claims.
Dependent claim 2 discloses wherein pruning at least one scenario of the one or more parameterized scenarios based on one or more of a set of possible scenarios or a set of relevant scenarios. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 3 discloses wherein filtering at least one invalid scenario out of the one or more parameterized scenarios. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 4 discloses wherein identifying at least one scenario of the one or more parameterized scenarios based on a likelihood of failure. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 5 discloses wherein selecting at least one scenario of the one or more parameterized scenarios based on a likelihood of failure to test the vehicle stack. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 6 discloses wherein determining a test coverage over a realistic scenario space. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claims 7 and 8 mentions the type of vehicle as an autonomous vehicle and the vehicle stack as an autonomous vehicle stack. These additional elements do not integrate the abstract idea into a practical application or can be considered as significantly more, because it does not impose any meaningful limits on practicing the abstract idea.
Dependent claim 10 discloses wherein the simulation properties include one or more interactions between one or more actors and one or more ego vehicles. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 11 discloses wherein detect a failure scenario using the runtime constraint. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 12 discloses wherein trigger a simulation failure based on detecting a failure scenario using the runtime constraint. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 13 discloses wherein continue a simulation within one or more boundary conditions based on detecting a failure scenario using the runtime constraint. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 14 discloses wherein impose the runtime constraint without using a mapping between the simulation properties and a scenario parameter space. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 16 discloses wherein select the subset of scenarios using auto-sampling to minimize a number of non-edge scenarios. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 17 discloses wherein generate a model mapping a scenario parameter space and one or more simulation results to identify the subset of scenarios to sample. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 18 discloses wherein search for a scenario parameter response function using auto-sampling. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 19 discloses wherein select the subset of scenarios using auto-sampling to reduce a computation time. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Dependent claim 20 discloses wherein adjust one or more of a verification workflow or validation workflow. This is a process that, under its broadest reasonable interpretation, is a process step that covers mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. Thus, the claims are directed to the abstract idea of a mental process performed in the human mind, or with the aid of pencil and paper.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
4. Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Whiteside et al., hereafter Whiteside (Pub. No.: US 2023/0234613 A1).
Regarding Claim 1, Whiteside disclose a method, comprising:
defining, using a processor operatively coupled to a memory, one or more test cases for a vehicle stack based on system requirements (Whiteside: [0058]: testing pipeline that can be used to test the performance of all or part of an autonomous vehicle (AV) runtime stack);
linking, using the processor operatively coupled to the memory, the one or more test cases to one or more parameterized scenarios, wherein the one or more parameterized scenarios comprise a plurality of parameter permutations (Whiteside: [0060]: Simulated perception inputs are derived from the simulation, which in tum feed into the stack or substack under testing, where they are processed in exactly the same way as corresponding physical perception inputs would be, so as to drive autonomous decision making within the ( sub-)stack; [0066]: the performance of the stack is assessed, at least in part, by evaluating the behaviour of the ego agent in the test oracle against a given set of performance evaluation metrics, over the course of one or more runs; [0073]: A scenario is represented as a set of scenario description parameters used by the planner 106; [0194]-0915]: scenario parametrization). and
testing, using the processor operatively coupled to the memory, the vehicle stack using the one or more test cases and the one or more parameterized scenarios (Whiteside: Figures 1B, 2, [0076]-[0082]: An ADS/ADAS stack 100, e.g. of the kind depicted in FIG. IA, is subject to repeated testing and evaluation in simulation, by running multiple scenario instances in a simulator 202, and evaluating the performance of the stack 100 (and/or individual subs-stacks thereof) in a test oracle 252) .
Regarding Claim 2, Whiteside disclose the method of claim 1, further comprising:
pruning, using the processor operatively coupled to the memory, at least one scenario of the one or more parameterized scenarios based on one or more of a set of possible scenarios or a set of relevant scenarios (Whiteside: [0117]-[0119]: rule tree implemented within the testing pipeline; Also see [0133]-[0136]: selective activation of the rules).
Regarding Claim 3, Whiteside disclose the method of claim 1, further comprising:
filtering, using the processor operatively coupled to the memory, at least one invalid scenario out of the one or more parameterized scenarios (Whiteside: [0131]-[0139]: certain rules (e.g. safety-critical rules) may result in an overall failure if the rule is failed at any time step within the scenario (that is, the rule must be passed at every time step to obtain an overall pass on the scenario)).
Regarding Claim 4, Whiteside disclose the method of claim 1, further comprising:
identifying, using the processor operatively coupled to the memory, at least one scenario of the one or more parameterized scenarios based on a likelihood of failure (Whiteside: [0131]-[0139]: certain rules (e.g. safety-critical rules) may result in an overall failure if the rule is failed at any time step within the scenario (that is, the rule must be passed at every time step to obtain an overall pass on the scenario).
Regarding Claim 5, Whiteside disclose the method of claim 1, further comprising:
selecting, using the processor operatively coupled to the memory, at least one scenario of the one or more parameterized scenarios based on a likelihood of failure to test the vehicle stack (Whiteside: [0131]-[0139]: certain rules (e.g. safety-critical rules) may result in an overall failure if the rule is failed at any time step within the scenario (that is, the rule must be passed at every time step to obtain an overall pass on the scenario; Also see [0196]-[0199]).
Regarding Claim 6, Whiteside disclose the method of claim 1, further comprising:
determining, using the processor operatively coupled to the memory, a test coverage over a realistic scenario space (Whiteside: [0131]-[0139]).
Regarding Claim 7, Whiteside further disclose the method of claim 1, wherein the vehicle is an autonomous vehicle (Whiteside: [0076]).
Regarding Claim 8, Whiteside further disclose the method of claim 1, wherein the vehicle stack is an autonomous vehicle stack (Whiteside: [0076]).
Regarding Claims 9-14 and 15-20, they are computer-implemented system claims corresponding to the method as recited claims 1-8. Therefore, they are rejected for the same reason as claims 1-8 above.
Conclusion
5. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Bordallo et al. (Pub. No.: US 2024/0256415 A1) relates to tools and techniques for testing the performance of autonomous vehicle planners and teaches a computer implemented method of evaluating the performance of a target planner for an ego robot in a scenario.
Muehlenstaedt et al. (Patent No. US 12,499,298 B2) teaches system, method, and computer program product embodiments for generating and refining simulation scenarios that includes simulating operation of an autonomous vehicle (AV) using one or more generated scenario variations.
Norden et al. (Efficient Black-box Assessment of Autonomous Vehicle Safety,2020, arXiv, pp 1-14) conceptually presents a simulation testing framework that evaluates an entire modern AV system as a black box. This framework estimates the probability of accidents under a base distribution governing standard traffic behavior. In order to accelerate rare-event probability evaluation, the framework efficiently learn to identify and rank failure scenarios via adaptive importance- sampling methods..
Horel et al. (Using Formal Conformance Testing to Generate Scenarios for Autonomous Vehicles, 2022, IEEE, pp 532-637) defines an automatic approach to generate a large number of relevant critical scenarios for autonomous driving simulators. The approach is based on the generation of behavioral conformance tests from a formal model (specifying the ground truth configuration with the range of vehicle behaviors) and a test purpose (specifying the critical feature to focus on).
6. Examiner’s Remarks: Examiner has cited particular columns and line numbers in the references applied to the claims above for the convenience of the applicant. Although the specified citations are representative of the teachings of the art and are applied to specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant in preparing responses, to fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the Examiner. In the case of amending the claimed invention, Applicant is respectfully requested to indicate the portion(s) of the specification which dictate(s) the structure relied on for proper interpretation and also to verify and ascertain the metes and bounds of the claimed invention.
Correspondence Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IFTEKHAR A KHAN whose telephone number is (571)272-5699. The examiner can normally be reached on M-F from 9:00AM-6:00PM (CST). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached on (571)272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/IFTEKHAR A KHAN/Primary Examiner, Art Unit 2187
Prosecution Insights