DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 07/02/2026 has been entered.
Response to Arguments
Applicant's arguments see Remarks, filed 07/02/2026, with respect to the rejection(s) of claims 1 and 14 under 35 U.S.C. 101 have been fully considered but they are not persuasive. As seen in MPEP 2106.04 “Second, if the specification sets forth an improvement in technology, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement. That is, the claim includes the components or steps of the invention that provide the improvement described in the specification.” Applicants’ arguments highlight paragraphs 5 and 11-14 which disclose that conventional systems do not identify causes of error or require a large number of test cases. Even though the claims now include testing a subset of data that includes identified error data, the claim does not describe how that error is identified or used in relation to the autonomous vehicles. The claim also does not reflect how the system uses fewer test cases. Furthermore, the claim merely recites calculating second characteristic value, but it is unclear how evaluation of this value is used in relation to the autonomous guidance of a motor vehicle. The claims just include the additional elements of a computer and inputting pre-recorded sensor data into the computer to calculate values. Calculating values is a mathematical concept, and using a computer, or merely providing data to a computer that loosely ties the claim to autonomous vehicles, does not integrate the claim into a practical application. Stating that the calculation describes performance or that the data comes from a vehicle environment is merely an incidental or token addition to the claim that does not alter or affect how the process steps of the calculation are performed. According to MPEP 2106.04 (C) “Claims can recite a mental process even if they are claimed as being performed on a computer. The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The Court concluded that the algorithm could be performed purely mentally even though the claimed procedures "can be carried out in existing computers long in use, no new machinery being necessary." 409 U.S at 67, 175 USPQ at 675. See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699 (concluding that concept of "anonymous loan shopping" recited in a computer system claim is an abstract idea because it could be "performed by humans without a computer").”
Lastly, the fact pattern of example 47 of the 2024 Subject Matter Eligibility Examples does not match that of the instant application. Even though it had an improvement in the specification it also reflected the improvement from the specification into the claim. Example 47 includes additional elements (d) detecting a source address associated with the one or more malicious network packets,” “(e) dropping the one or more malicious network packets,” and “(f) blocking future traffic from the source address” which enhances security by acting in real time to proactively prevent network intrusions. As seen above the instant application does not include additional elements that provide an improvement that provides a real-world action or transformation.
Applicant’s arguments, see Remarks, filed 07/02/2026, with respect to the rejection(s) of claims 1 and 14 under 35 U.S.C. 102 have been fully considered and are persuasive. Whiteside does not explicitly teach “based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors.” Therefore, the rejection has been withdrawn. The examiner concedes that a mistake was made when mapping the features of Whiteside to the features of the claim. However, upon further consideration, Whiteside does teach that a normalization value is based on a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle as seen in Para. [0179], where the robustness score is viewed as the normalization value and the impact score is viewed as the second characteristic value. A further explanation of how elements in the claim map to elements in Whiteside can be found in the rejection below. As a result, a new ground(s) of rejection is made in view of Whiteside (US 20240419572 A1) and Whiteside (2025) (US 20250123952 A1).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 3-18 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With respect to claim 1, the limitations,
“calculating at least one first characteristic value representing a performance of a tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases;
evaluating the first characteristic value of each of the plurality of test cases using a specified first criterion to form a first evaluation result;
aggregating the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case;
and evaluating the second characteristic value using a specified second criterion and a logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case;
and based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors,
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value assigned to each test case or a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle.”
are directed to abstract ideas and would fall within the “Mental Process” and “Mathematical Concept” grouping of abstract ideas. Evaluating as seen in the specification is comparing the first and second characteristic values to a threshold as seen in Para. [0038] and [0039] of the specification and can be completed in the human mind using observation, judgement, and opinion. Aggregating or collecting values can likewise be done in the human mind. MPEP 2106.04 teaches “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012) ("‘[M]ental processes and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” Calculating a first value and a second value represents a mathematical concept as it is just calculating based on a performance of a device or data representing how far a vehicle travels and from MPEP 2106.04 “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas).”
This judicial exception is not integrated into a practical application. In particular, the claim
recites the additional elements –
“A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising:
providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;”
Examiner views these limitations amount to generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
As such Examiner does NOT view that the claims
-Improve the functioning of a computer, or to any other technology or technical field
-Apply the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
-Effect a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
-Apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo.
Moreover, Examiner views the claims to be merely generally linking the use of the judicial exception to the autonomous guidance of a motor vehicle.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising: providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;” amount to using a computer as a tool and mere data.
Examiner further notes that such additional elements are viewed to be well known routine and
conventional as evidenced by
Whiteside (US 20240419572 A1)
Whiteside (2025) (US 20250123952 A1)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. Considering the claim as a whole, one of ordinary skill in the art would
not know the practical application of the present invention since the claims do not apply or use the judicial exception in some meaningful way. As currently claimed, Examiner views that the additional
elements do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit
on the judicial exception, because the claims fail to recite clearly how the judicial exception is applied in
a manner that does not monopolize the exception because the limitations regarding “A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising: providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;” can be viewed as tying the claim to a computer system and monitoring procedures or a device related to autonomous driving.
With respect to claim 14, the limitations,
“calculate at least one first characteristic value representing a performance of the tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases;
evaluate the first characteristic value of each of the plural test cases using a specified first criterion to form a first evaluation result;
aggregate the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case;
and evaluate the second characteristic value using a specified second criterion and the logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case;
and based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors,
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value assigned to each test case or a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle.”
are directed to abstract ideas and would fall within the “Mental Process” and “Mathematical Concept” grouping of abstract ideas. Evaluating as seen in the specification is comparing the first and second characteristic values to a threshold as seen in Para. [0038] and [0039] of the specification and can be completed in the human mind using observation, judgement, and opinion. Aggregating or collecting values can likewise be done in the human mind. MPEP 2106.04 teaches “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012) ("‘[M]ental processes and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” Calculating a first value and a second value represents a mathematical concept as it is just calculating based on a performance of a device or data representing how far a vehicle travels and from MPEP 2106.04 “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas).”
This judicial exception is not integrated into a practical application. In particular, the claim
recites the additional elements –
“A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator”
Examiner views these limitations amount to generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
As such Examiner does NOT view that the claims
-Improve the functioning of a computer, or to any other technology or technical field
-Apply the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
-Effect a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
-Apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo.
Moreover, Examiner views the claims to be merely generally linking the use of the judicial exception to the autonomous guidance of a motor vehicle.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator” amount to using a computer as a tool and mere data.
Examiner further notes that such additional elements are viewed to be well known routine and
conventional as evidenced by
Whiteside (US 20240419572 A1)
Whiteside (2025) (US 20250123952 A1)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. Considering the claim as a whole, one of ordinary skill in the art would
not know the practical application of the present invention since the claims do not apply or use the judicial exception in some meaningful way. As currently claimed, Examiner views that the additional
elements do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit
on the judicial exception, because the claims fail to recite clearly how the judicial exception is applied in
a manner that does not monopolize the exception because the limitations regarding “A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator” can be viewed as tying the claim to a computer system and monitoring procedures or a device related to autonomous driving.
Dependent claims 3-13, and 15-18 when analyzed as a whole are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitation(s) fail(s) to establish that the claims are not directed to an abstract idea, as detailed below:
They amount to further limiting the claims with using well known mathematical concepts like statistical analysis, normalization, and comparing data to thresholds. Claim 15 includes a computer which is being used as tool which is further seen as adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Claims 17 and 18 include functions that are monitored and the conditions that the test cases are recorded. These also amount to adding insignificant extra-solution activity to the judicial exception as they are well known and are only nominally or tangentially related to the invention. There are no additional element(s) in the dependent claims that adds a meaningful limitation to the abstract idea to make the claim significantly more than the judicial exception (abstract idea).
Dependent claims 3-13, and 15-18 further limit the abstract idea with an abstract idea
and thus, the claims are still directed to an abstract idea without significantly more.
Claim 15 is further rejected under 35 U.S.C. 101 as the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because a computer program with program code is not a process, machine, manufacture, or composition of matter. Examiner suggests amending the claim to read “A non-transitory computer readable medium comprising a computer program with program code to carry out the method according to claim 1 when the computer program is executed on a computer.”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3-17 are rejected under 35 U.S.C. 103 as being unpatentable over Whiteside (US 20240419572 A1) in view of Whiteside (2025) (US 20250123952 A1).
With respect to claim 1,
Whiteside teaches,
providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; (Para. [0087] teaches “highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed {i.e. providing pre-recorded sensor data} to a ‘ground-truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and/or perception outputs captured/generated on board one or more vehicles {i.e. sensor data of a vehicle environment})
calculating at least one first characteristic value representing a performance of a tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases; (Para. [0001] teaches “Such planners are capable of autonomously planning ego trajectories for fully/semi-autonomous vehicles or other forms of mobile robot. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing.” Para. [0032] teaches “determining a scenario parameter set for the scenario and a likelihood of the set of scenario parameters;). {i.e. Scenario parameters are seen as first characteristic values} )
evaluating the first characteristic value of each of the plurality of test cases using a specified first criterion to form a first evaluation result; (Para. [0032] teaches “applying one or more performance evaluation rules to the traces, thereby obtaining a set of performance evaluation results for the trajectory planner;” {i.e. evaluation rules viewed as first criterion})
aggregating the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case; (Para. [0032] teaches “computing an impact score based on the performance evaluation results;” Para. [0030] teaches “As another example, the impact score may be computed for multiple rules, e.g. by summing or otherwise aggregating the importance values of any rules that are failed.”)
and evaluating the second characteristic value using a specified second criterion and a logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case. (Para. [0032] teaches “and computing a significance score for the instance of the scenario based on the impact score and the likelihood of the set of scenario parameters.” {i.e. significance score is viewed as second evaluation result} Para. [0134] teaches “The above examples consider simple logical predicates evaluated on results or signals at a single time instance, such as OR, AND, Gt etc. However, in practice, it may be desirable to formulate certain rules in terms of temporal logic.” {i.e. logical predicates are seen as logically linking.})
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value, based on a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle, assigned to each test case. (Para. [0157] teaches “An impact score can be computed for a failure event (or near failure event) between agents in various ways, e.g. based on the robustness score of a failed (or near-failed) rule(s), some parameter(s) relating to the failure (or near-failed) event and/or an importance value assigned to a failed (or near-failed) rule(s).” {i.e. using first characteristic values by using parameters related to failure as they are scenario parameters as defined above.} Para. [0179] further teaches “The robustness score may be normalized, so that zero represents the boundary between pass and fail (a score of zero means the rule has only ‘just’ been failed), with the score preferably normalized to a predetermined range, e.g. [−1,1], with −1 denoting maximum failure and 1 denoting maximum pass. As an example, for a “no collision rule” defined between a pair of agents, the robustness score could be in terms of distance and/or intersection. {i.e. Robustness core is viewed as normalized value representing distance.})
Whiteside does not explicitly teach,
based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors,
Whiteside 2025 teaches,
based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors, (Para. [0126] teaches “FIG. 8 shows an architecture for evaluating perception errors. A triage tool 152 comprising a perception oracle 1108 is used to extract and evaluate perception errors for both real and simulated driving scenarios, and outputs the results to be rendered in a GUI 500 alongside results from a test oracle 252.” Fig. 8 further shows multiple characteristic values feed the perception oracle which is a collection of rules used to evaluate the data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside with based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors, such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside (2025), because as seen in Para. [0052] of Whiteside (2025) evaluating errors would allow a user to determine the causes of said errors and address the causes to prevent further issues in the future. Furthermore, Whiteside mentions using error models in Para(s). [0112] and [0113]. Therefore, one would be motivated to combine the prior art to build upon the error analysis in Whiteside in order to fix the cause of the errors in the data.
With respect to claim 3,
Whiteside further teaches,
The computer-implemented method according to claim 2, wherein the second characteristic value is calculated by minimum, maximum or median formation of the plurality of first characteristic values of each of the test cases and the normalization value assigned to each test case. (Para. [0176] teaches “For example, the overall robustness score for each rule may be equal to the minimum robustness score on that rule across all time steps of the run (the minimum robustness score denoting the worst failure point on that rule if negative, or the closest the ego came to failing that rule if positive or zero). In such cases, the impact score may, for example, be defined as zero when the minimum robustness score is zero or positive (positive robustness score means the rule is passed, thus no failure event on that rule), and equal to the magnitude of robustness score (or some transformation thereof) when the minimum robustness score is negative.” {i.e. minimum formation para. [0179] shows normalization})
With respect to claim 4,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first criterion is specified by a first threshold, wherein the first characteristic value is compared with the first threshold for each of the plurality of test cases, and wherein a determination of a fulfillment or non-fulfillment a requirement addressed to the device and/or function for at least partial autonomous guidance of the motor vehicle is made using a comparative result. (Para. [0121] teaches “In addition, each assessor node 304 derives a time-varying numerical signal from the output(s) of its child node(s), which is related to the categorical results by a threshold condition” {i.e. shows threshold} Para. [0123] teaches “The results 314 are correlated with the derived signal 312, in that a pass result is returned when (and only when) the derived signal exceeds a failure threshold 316. As will be appreciated, this is merely one example of a threshold condition that relates a time-sequence of results to a corresponding signal.” {i.e. pass or failure is analogous to fulfillment or non-fulfillment})
With respect to claim 5,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the second criterion is specified by a second threshold, wherein the second characteristic value for the meta-test case is compared with the second threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the meta-test case is made using a comparative result and the logical linking of the first evaluation results of the plurality of test cases. (Para. [0179] teaches “The principles could also be extended to ‘near failure’ events, whose impact is assessed and influences the risk score. A near failure event may be defined, for example, in terms of a second threshold applied to the robustness score (e.g. with a score<0 defined as failure, and a score between 0 and +0.1 defined as near failure). In this case, a near failure event may give rise to a non-zero impact score.” {i.e. second threshold})
With respect to claim 6,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions.
Whiteside (2025) teaches,
wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions. (Para. [0006] teaches “These rules could include driving rules that evaluate the behaviour of the ego agent based on some model of expected safe driving behaviour in similar driving scenarios, and/or perception rules, that evaluate the accuracy of the ego's perception of its surroundings.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it allows the system to give interpretable results as seen in Para. [0006] “One way to provide interpretable results on the scenario level is to provide a graphical user interface for displaying results for each given scenario instance (or ‘run’) of an ego agent driving in a given set of conditions (real or simulated).” Furthermore, it would be harder to compare or simulate data from dissimilar conditions as the results would be less meaningful.
With respect to claim 7,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the tested device and/or function for at least partially autonomous guidance of the motor vehicle is object recognition and/or vehicle guidance. (Para. [0001] teaches “The present disclosure pertains to methods for evaluating the performance of trajectory planners in real or simulated scenarios, and computer programs and systems for implementing the same. Such planners are capable of autonomously planning ego trajectories for fully/semi-autonomous vehicles or other forms of mobile robot. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing.” {i.e. Trajectory planning is equivalent to vehicle guidance})
With respect to claim 8,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases.
Whiteside (2025) teaches,
The computer-implemented method according to claim 1, wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases. (Para. [0034] teaches “A set of rules can then be applied together to a given driving scenario by defining a perception error specification which includes all the rules to be applied. Typical perception rules that may be included in a specification define thresholds on longitudinal and lateral translation errors (measuring mean error of the detection with respect to ground truth in the longitudinal and lateral directions, respectively), orientation error (defining a minimum angle that one needs to rotate the detection to line it up with the corresponding ground truth), size error (error on each dimension of the detected bounding box, or an intersection over union on the aligned ground truth and detected boxes to get a volume delta). Further rules may be based on vehicle dynamics, including errors in the velocity and acceleration of the agents, and errors in classifications, for example defining penalty values for misclassifying a car as a pedestrian or lorry. Rules may also include false positives or missed detections, as well as detection latency.” {i.e. total perception error is calculated using other errors from a plurality of characteristic values making it a third characteristic value.})
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it would help to determine if the system is safe to drive and incorporate complex dependencies as seen in Para. [0035] of Whiteside (2025).
With respect to claim 9,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 8, wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases.
Whiteside (2025) teaches,
wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases. (Para. [0034] teaches “A set of rules can then be applied together to a given driving scenario by defining a perception error specification which includes all the rules to be applied. Typical perception rules that may be included in a specification define thresholds on longitudinal and lateral translation errors (measuring mean error of the detection with respect to ground truth in the longitudinal and lateral directions, respectively), orientation error (defining a minimum angle that one needs to rotate the detection to line it up with the corresponding ground truth), size error (error on each dimension of the detected bounding box, or an intersection over union on the aligned ground truth and detected boxes to get a volume delta). Further rules may be based on vehicle dynamics, including errors in the velocity and acceleration of the agents, and errors in classifications, for example defining penalty values for misclassifying a car as a pedestrian or lorry. Rules may also include false positives or missed detections, as well as detection latency.” {i.e. total perception error is calculated using other errors from a plurality of characteristic values making it a third characteristic value. The third threshold would be the error threshold.}
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it would help to determine if the system is safe to drive and incorporate complex dependencies as seen in Para. [0035] of Whiteside (2025).
With respect to claim 10,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first characteristic value is determined by comparing ground truth data with the pre-recorded sensor data of the vehicle environment and/or vehicle-related parameters representing a plurality of test cases. (Para. [0066] teaches “the performance of the stack is assessed, at least in part, by evaluating the behaviour of the ego agent in the test oracle against a given set of performance evaluation rules, over the course of one or more runs. The rules are applied to “ground truth” of the (or each) scenario run which, in general, simply means an appropriate representation of the scenario run (including the behaviour of the ego agent) that is taken as authoritative for the purpose of testing. Ground truth is inherent to simulation; a simulator computes a sequence of scenario states, which is, by definition, a perfect, authoritative representation of the simulated scenario run. In a real-world scenario run, a “perfect” representation of the scenario run does not exist in the same sense; nevertheless, suitably informative ground truth can be obtained in numerous ways, e.g. based on manual annotation of on-board sensor data, automated/semi-automated annotation of such data (e.g. using offline/non-real time processing), and/or using external information sources (such as external sensors, maps etc.) etc.” {i.e. ground truth is compared to real world scenario run.)
With respect to claim 11,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first characteristic value of the pre-recorded sensor data of the vehicle environment and/or vehicle-related parameters representing a plurality of test cases is determined with a fourth threshold. (Para. [0131] teaches “Additionally, the rule tree includes a longitudinal distance branch, and a top-level OR predicate (safe distance node, is_d_safe) to implement a safe distance metric. Similar to the lateral distance branch, the longitudinal distance brand extracts longitudinal distance and longitudinal distance threshold signals from the scenario data (extractor nodes lond and lonsd respectively), and a longitudinal safety assessor node (is_lond_safe) returns TRUE when the longitudinal distance is above the safe longitudinal distance threshold.”{i.e. the safe distance metric is considered a parameter representing a plurality of test cases since the distance signals are time varying as seen in Para. [0129]. The fourth threshold is the safe longitudinal distance threshold.})
With respect to claim 12,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein each test case has a plurality of test sequences, individual images, and/or measurement time series. (Para. [0108] teaches “The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass/fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later.” {i.e. measurement time series} Para. [0074] teaches “The on-board sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices” {i.e. images})
With respect to claim 13,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein a parameter space is determined for a virtual test of the device and/or function for at least partially autonomous guidance of a motor vehicle using the evaluation of the second characteristic value of the plurality of test cases. (Para. [0068] teaches “The configurable parameter(s) define a parameter space (also referred to as the scenario space), and the parameterization corresponds to a point in the parameter space” Para. [0173] teaches “Here, the risk score is stored and visualized as a function of the underlying scenario parameters. FIGS. 8 and 9 plot the risk score over a portion of the scenario parameter space. Each shows their respective visualizations 800, 900 in plan and perspective views.”)
With respect to claim 14,
Whiteside teaches,
a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; (Para. [0087] teaches “highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed {i.e. providing pre-recorded sensor data} to a ‘ground-truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and/or perception outputs captured/generated on board one or more vehicles {i.e. sensor data of a vehicle environment})
a calculation device to calculate at least one first characteristic value representing a performance of the tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases; (Para. [0001] teaches “Such planners are capable of autonomously planning ego trajectories for fully/semi-autonomous vehicles or other forms of mobile robot. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing.” Para. [0032] teaches “determining a scenario parameter set for the scenario and a likelihood of the set of scenario parameters;). {i.e. Scenario parameters are seen as first characteristic values} Para. [0085] teaches “The improved stack 100 typically includes program instructions (software) executed in one or more computer processors of an on-board computer system of the vehicle 101 (not shown).”)
a first evaluator to evaluate the first characteristic value of each of the plural test cases using a specified first criterion to form a first evaluation result; (Para. [0032] teaches “applying one or more performance evaluation rules to the traces, thereby obtaining a set of performance evaluation results for the trajectory planner;” {i.e. evaluation rules viewed as first criterion})
an aggregator to aggregate the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case; (Para. [0032] teaches “computing an impact score based on the performance evaluation results;” Para. [0030] teaches “As another example, the impact score may be computed for multiple rules, e.g. by summing or otherwise aggregating the importance values of any rules that are failed.”)
and a second evaluator to evaluate the second characteristic value using a specified second criterion and the logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case. (Para. [0032] teaches “and computing a significance score for the instance of the scenario based on the impact score and the likelihood of the set of scenario parameters.” {i.e. significance score is viewed as second evaluation result} Para. [0134] teaches “The above examples consider simple logical predicates evaluated on results or signals at a single time instance, such as OR, AND, Gt etc. However, in practice, it may be desirable to formulate certain rules in terms of temporal logic.” {i.e. logical predicates are seen as logically linking.})
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value based on a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle, assigned to each test case. (Para. [0157] teaches “An impact score can be computed for a failure event (or near failure event) between agents in various ways, e.g. based on the robustness score of a failed (or near-failed) rule(s), some parameter(s) relating to the failure (or near-failed) event and/or an importance value assigned to a failed (or near-failed) rule(s).” {i.e. using first characteristic values by using parameters related to failure} Para. [0179] further teaches “The robustness score may be normalized, so that zero represents the boundary between pass and fail (a score of zero means the rule has only ‘just’ been failed), with the score preferably normalized to a predetermined range, e.g. [−1,1], with −1 denoting maximum failure and 1 denoting maximum pass. As an example, for a “no collision rule” defined between a pair of agents, the robustness score could be in terms of distance and/or intersection. {i.e. Robustness core is viewed as normalized value representing distance.})
Whiteside does not explicitly teach,
based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors,
Whiteside 2025 teaches,
based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors, (Para. [0126] teaches “FIG. 8 shows an architecture for evaluating perception errors. A triage tool 152 comprising a perception oracle 1108 is used to extract and evaluate perception errors for both real and simulated driving scenarios, and outputs the results to be rendered in a GUI 500 alongside results from a test oracle 252.” Fig. 8 further shows multiple characteristic values feed the perception oracle which is a collection of rules used to evaluate the data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside with based on said evaluating the second characteristic value, subsequently testing a subset of said sensor data including identified errors, such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside (2025), because as seen in Para. [0052] of Whiteside (2025) evaluating errors would allow a user to determine the causes of said errors and address the causes to prevent further issues in the future. Furthermore, Whiteside mentions using error models in Para(s). [0112] and [0113]. Therefore, one would be motivated to combine the prior art to build upon the error analysis in Whiteside in order to fix the cause of the errors in the data.
With respect to claim 15,
Whiteside further teaches,
A computer program with program code to carry out the method according to claim 1 when the computer program is executed on a computer. (Para. [0206] teaches “The subsystems 102-108 of the runtime stack FIG. 1A may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like.”)
With respect to claim 16,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the at least one first characteristic value is a key performance indicator, which is a quantifiable measure of performance over time for a specific objective. (Para. [0015] teaches “A first aspect herein is directed to a computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a scenario, in which the trajectory planner is used to control an ego agent of the scenario responsive to at least one other agent of the scenario, the method comprising: determining a scenario parameter set (set of one or more scenario parameters) for the scenario and a likelihood of the set of scenario parameters; computing an impact score for a dynamic interaction (failure event or near failure event) between the mobile robot and the other agent occurring in the scenario, the impact score quantifying severity of the dynamic interaction; and computing a risk score for the instance of the scenario based on the impact score and the likelihood of the set of scenario parameters.”{i.e. impact score or risk score represent key performance indicators.})
With respect to claim 17,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the function for at least partially autonomous guidance of a motor vehicle comprises object recognition, adaptive cruise control, lane departure warning, active brake assist and / or park assist. (Para. [0100] teaches “In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC).”
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Whiteside (US 20240419572 A1) and Whiteside (2025) (US 20250123952 A1) as applied to claim 1 above, and further in view of Venkatadri (US 20220194395 A1).
With respect to claim 18,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions.
Venkatadri teaches,
wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions. (Para. [0031] “In some implementations, the data can include sensor data from an autonomous vehicle that operated under an evaluated operating condition that is substantially similar and/or equivalent to the unevaluated operating condition.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions such as that of Venkatadri.
One of ordinary skill would have been motivated to modify Whiteside, because it ensures an accurate comparison between test cases. If the test cases are not tested in similar conditions it would lead to data that is wildly variable and lead to inaccurate results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to JOSHUA L FORRISTALL whose telephone number is 703-756-4554. The examiner
can normally be reached Monday-Friday 8:30 AM- 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Schechter can be reached on 571-272-2302. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/JOSHUA L FORRISTALL/Examiner, Art Unit 2857
/ANDREW SCHECHTER/Supervisory Patent Examiner, Art Unit 2857
Prosecution Insights