DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments to the claims, filed 03/03/2026, are accepted and appreciated by the
Examiner.
Response to Arguments
Applicant’s arguments, see Remarks, filed 03/03/2026, with respect to the 35 U.S.C. 112(a) rejection of claims 1-16 have been fully considered and are persuasive. The application as originally filed defines that the first characteristic value is a key performance indicator, which is a quantifiable measure of performance over time and is well within the understanding of a person of ordinary skill in the art. Therefore, the disclosure shows possession of the claimed invention. The 35 U.S.C. 112(a) rejection of claims 1-18 has been withdrawn.
Applicant’s arguments, see Remarks, filed 03/03/2026, with respect to the 35 U.S.C. 112(b) rejections of claims 1-13 have been fully considered and are persuasive in light of the amendments. The 35 U.S.C. 112(b) rejections of claims 1-13 have been withdrawn.
Applicant's arguments filed 03/03/2026 with respect to the 35 U.S.C. 101 and 102 rejections of claims 1 and 14 have been fully considered but they are not persuasive.
With respect to the 35 U.S.C. 101 rejection as seen in MPEP 2106.04 “Second, if the specification sets forth an improvement in technology, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement. That is, the claim includes the components or steps of the invention that provide the improvement described in the specification.” Applicants’ arguments highlight paragraphs 5 and 11-13 which disclose that conventional systems do not identify causes of error or require a large number of test cases. Claim 1 does not include the identification of errors or that the calculation only uses a small number of test cases. Therefore, the argued improvements are not represented in the claims.
Applicant also highlights paragraph 14 of the specification which discloses that the invention enables simplified data evaluation and more efficient test execution. However, as stated MPEP 2106.05 “It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements.” The claim discloses providing data, calculating a value based on data representing performance of a partial autonomous guidance of a motor vehicle, evaluating the calculated value for a plurality of test cases, aggregating said value, and calculating a second value based on a normalized value. Most of these step’s amount to making calculations which are seen as mathematical concepts or comparing data which is viewed as a mental process. The other steps provide data which is viewed as mere data gathering or tie the data and calculations to the performance of an autonomous vehicle or to distance traveled by a vehicle. Tying the data and calculations to the performance of an autonomous vehicle or to distance traveled by a vehicle just generally links the use of the judicial exception to a particular technological environment or field of use as seen in MPEP 2106.05. Stating that the calculation describes performance or that the data comes from a vehicle environment is merely an incidental or token addition to the claim that does not alter or affect how the process steps of the calculation are performed. Therefore, the improvement is tied to the process steps which are mathematical concepts and abstract ideas and do not integrate the claim into a practical application.
Lastly, the fact pattern of example 47 of the 2024 Subject Matter Eligibility Examples does not match that of the instant application. Example 47 includes additional elements (d) detecting a source address associated with the one or more malicious network packets,” “(e) dropping the one or more malicious network packets,” and “(f) blocking future traffic from the source address” which enhances security by acting in real time to proactively prevent network intrusions. As seen above the instant application does not include additional elements that provide an improvement that provides a real-world action or transformation.
With respect to the 35 U.S.C. 102 rejection Whiteside (US 20240419572 A1) does teach the elements included in claim 1 from cancelled claim 2. Para. [0175] teaches that the robustness score could be in terms of distance and that the robustness core is normalized. This distance would represent a distance traveled by a vehicle as the distance referred to in Whiteside is the distance the vehicle would travel to cause a collision. (Para(s). [0156 & 0175]) Therefore, claim 1 stands rejected under 35 U.S.C. 102(a)(2).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION. —The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 17 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 17 includes the limitation “wherein the function for at least partially autonomous guidance of a motor vehicle comprises objection recognition” and it is unclear and indefinite what the term objection recognition is referring to. Specification paragraph [0028] lists object recognition as a function for at least partially autonomous guidance of a motor vehicle and therefore, for the purposes of examination the limitation will be viewed as “wherein the function for at least partially autonomous guidance of a motor vehicle comprises object recognition.”
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 and 3-18 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
With respect to claim 1, the limitations,
“calculating at least one first characteristic value representing a performance of a tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases;
evaluating the first characteristic value of each of the plurality of test cases using a specified first criterion to form a first evaluation result;
aggregating the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case;
and evaluating the second characteristic value using a specified second criterion and a logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case,
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value assigned to each test case or a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle.”
are directed to abstract ideas and would fall within the “Mental Process” and “Mathematical Concept” grouping of abstract ideas. Evaluating as seen in the specification is comparing the first and second characteristic values to a threshold as seen in Para. [0038] and [0039] of the specification and can be completed in the human mind using observation, judgement, and opinion. Aggregating or collecting values can likewise be done in the human mind. MPEP 2106.04 teaches “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012) ("‘[M]ental processes and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” Calculating a first value and a second value represents a mathematical concept as it is just calculating based on a performance of a device or data representing how far a vehicle travels and from MPEP 2106.04 “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas).”
This judicial exception is not integrated into a practical application. In particular, the claim
recites the additional elements –
“A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising:
providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;”
Examiner views these limitations amount to generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
As such Examiner does NOT view that the claims
-Improve the functioning of a computer, or to any other technology or technical field
-Apply the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
-Effect a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
-Apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo.
Moreover, Examiner views the claims to be merely generally linking the use of the judicial exception to the autonomous guidance of a motor vehicle.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising: providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;” amount to using a computer as a tool and mere data.
Examiner further notes that such additional elements are viewed to be well known routine and
conventional as evidenced by
Whiteside (US 20240419572 A1)
Whiteside (2025) (US 20250123952 A1)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. Considering the claim as a whole, one of ordinary skill in the art would
not know the practical application of the present invention since the claims do not apply or use the judicial exception in some meaningful way. As currently claimed, Examiner views that the additional
elements do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit
on the judicial exception, because the claims fail to recite clearly how the judicial exception is applied in
a manner that does not monopolize the exception because the limitations regarding “A computer-implemented method for the analysis of test procedures of a device and/or a function for at least partially autonomous guidance of a motor vehicle, the method comprising: providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases;” can be viewed as tying the claim to a computer system and monitoring procedures or a device related to autonomous driving.
With respect to claim 14, the limitations,
“calculate at least one first characteristic value representing a performance of the tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases;
evaluate the first characteristic value of each of the plural test cases using a specified first criterion to form a first evaluation result;
aggregate the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case;
and evaluate the second characteristic value using a specified second criterion and the logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case,
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value assigned to each test case or a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle.”
are directed to abstract ideas and would fall within the “Mental Process” and “Mathematical Concept” grouping of abstract ideas. Evaluating as seen in the specification is comparing the first and second characteristic values to a threshold as seen in Para. [0038] and [0039] of the specification and can be completed in the human mind using observation, judgement, and opinion. Aggregating or collecting values can likewise be done in the human mind. MPEP 2106.04 teaches “The courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 (2012) ("‘[M]ental processes and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same). Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, judgments, and opinions.” Calculating a first value and a second value represents a mathematical concept as it is just calculating based on a performance of a device or data representing how far a vehicle travels and from MPEP 2106.04 “It is important to note that a mathematical concept need not be expressed in mathematical symbols, because "[w]ords used in a claim operating on data to solve a problem can serve the same purpose as a formula." In re Grams, 888 F.2d 835, 837 and n.1, 12 USPQ2d 1824, 1826 and n.1 (Fed. Cir. 1989). See, e.g., SAP America, Inc. v. InvestPic, LLC, 898 F.3d 1161, 1163, 127 USPQ2d 1597, 1599 (Fed. Cir. 2018) (holding that claims to a ‘‘series of mathematical calculations based on selected information’’ are directed to abstract ideas).”
This judicial exception is not integrated into a practical application. In particular, the claim
recites the additional elements –
“A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator”
Examiner views these limitations amount to generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h).
As such Examiner does NOT view that the claims
-Improve the functioning of a computer, or to any other technology or technical field
-Apply the judicial exception with, or by use of, a particular machine - see MPEP 2106.05(b)
-Effect a transformation or reduction of a particular article to a different state or thing - see MPEP 2106.05(c)
-Apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception - see MPEP 2106.05(e) and Vanda Memo.
Moreover, Examiner views the claims to be merely generally linking the use of the judicial exception to the autonomous guidance of a motor vehicle.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of “A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator” amount to using a computer as a tool and mere data.
Examiner further notes that such additional elements are viewed to be well known routine and
conventional as evidenced by
Whiteside (US 20240419572 A1)
Whiteside (2025) (US 20250123952 A1)
The claim does not include additional elements that are sufficient to amount to significantly
more than the judicial exception. Considering the claim as a whole, one of ordinary skill in the art would
not know the practical application of the present invention since the claims do not apply or use the judicial exception in some meaningful way. As currently claimed, Examiner views that the additional
elements do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit
on the judicial exception, because the claims fail to recite clearly how the judicial exception is applied in
a manner that does not monopolize the exception because the limitations regarding “A system to analyze test procedures of a device and/or a function for at least partial autonomous guidance of a motor vehicle, the system comprising: a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; a calculation device; a first evaluator; an aggregator; and a second evaluator” can be viewed as tying the claim to a computer system and monitoring procedures or a device related to autonomous driving.
Dependent claims 3-13, and 15-18 when analyzed as a whole are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitation(s) fail(s) to establish that the claims are not directed to an abstract idea, as detailed below:
They amount to further limiting the claims with using well known mathematical concepts like statistical analysis, normalization, and comparing data to thresholds. Claim 15 includes a computer which is being used as tool which is further seen as adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Claims 17 and 18 include functions that are monitored and the conditions that the test cases are recorded. These also amount to adding insignificant extra-solution activity to the judicial exception as they are well known and are only nominally or tangentially related to the invention. There are no additional element(s) in the dependent claims that adds a meaningful limitation to the abstract idea to make the claim significantly more than the judicial exception (abstract idea).
Dependent claims 3-13, and 15-18 further limit the abstract idea with an abstract idea
and thus, the claims are still directed to an abstract idea without significantly more.
Claim 15 is further rejected under 35 U.S.C. 101 as the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because a computer program with program code is not a process, machine, manufacture, or composition of matter. Examiner suggests amending the claim to read “A non-transitory computer readable medium comprising a computer program with program code to carry out the method according to claim 1 when the computer program is executed on a computer.”
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-5, 7, 10-17 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Whiteside (US 20240419572 A1).
With respect to claim 1,
Whiteside teaches,
providing pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; (Para. [0087] teaches “highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed {i.e. providing pre-recorded sensor data} to a ‘ground-truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and/or perception outputs captured/generated on board one or more vehicles {i.e. sensor data of a vehicle environment})
calculating at least one first characteristic value representing a performance of a tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases; (Para. [0108] teaches “The rules 254 are categorical in nature (e.g. pass/fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). {i.e. performance metrics are seen as first characteristic values})
evaluating the first characteristic value of each of the plurality of test cases using a specified first criterion to form a first evaluation result; (Para. [0121] teaches “In addition, each assessor node 304 derives a time-varying numerical signal from the output(s) of its child node(s), which is related to the categorical results by a threshold condition” {i.e. relating to a threshold is seen a specified first criterion} Para. [0123] teaches “The results 314 are correlated with the derived signal 312, in that a pass result is returned when (and only when) the derived signal exceeds a failure threshold 316. As will be appreciated, this is merely one example of a threshold condition that relates a time-sequence of results to a corresponding signal.” {i.e. results are viewed as evaluation result})
aggregating the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case; (Para. [0108] teaches “The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g. overall pass/fail).”)
and evaluating the second characteristic value using a specified second criterion and a logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case. (Para. [0134] teaches “The above examples consider simple logical predicates evaluated on results or signals at a single time instance, such as OR, AND, Gt etc. However, in practice, it may be desirable to formulate certain rules in terms of temporal logic.” {i.e. logical predicates are seen as logically linking.} Para. [0175] teaches “A set of performance evaluation rules is determined that is applicable to a scenario. A numerical performance metric is computed for each performance evaluation rule, in the form of a robustness score that quantifies the extent or success of failure. {i.e. the set of evaluation rules is viewed as the second characteristic value and robustness score is seen as a second evaluation result})
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value assigned to each test case or a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle. (Para. [0179] teaches “Robustness scores are computed by the test oracle 252 as set out above. A set of performance evaluation rules is determined that is applicable to a scenario. A numerical performance metric is computed for each performance evaluation rule, in the form of a robustness score that quantifies the extent or success of failure. {i.e. using first characteristic values} Para. [0179] further teaches “The robustness score may be normalized, so that zero represents the boundary between pass and fail (a score of zero means the rule has only ‘just’ been failed), with the score preferably normalized to a predetermined range, e.g. [−1,1], with −1 denoting maximum failure and 1 denoting maximum pass. As an example, for a “no collision rule” defined between a pair of agents, the robustness score could be in terms of distance and/or intersection. {i.e. normalized value representing distance}
With respect to claim 3,
Whiteside further teaches,
The computer-implemented method according to claim 2, wherein the second characteristic value is calculated by minimum, maximum or median formation of the plurality of first characteristic values of each of the test cases and the normalization value assigned to each test case. (Para. [0176] teaches “For example, the overall robustness score for each rule may be equal to the minimum robustness score on that rule across all time steps of the run (the minimum robustness score denoting the worst failure point on that rule if negative, or the closest the ego came to failing that rule if positive or zero). In such cases, the impact score may, for example, be defined as zero when the minimum robustness score is zero or positive (positive robustness score means the rule is passed, thus no failure event on that rule), and equal to the magnitude of robustness score (or some transformation thereof) when the minimum robustness score is negative.” {i.e. minimum formation para. [0179] shows normalization})
With respect to claim 4,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first criterion is specified by a first threshold, wherein the first characteristic value is compared with the first threshold for each of the plurality of test cases, and wherein a determination of a fulfillment or non-fulfillment a requirement addressed to the device and/or function for at least partial autonomous guidance of the motor vehicle is made using a comparative result. (Para. [0121] teaches “In addition, each assessor node 304 derives a time-varying numerical signal from the output(s) of its child node(s), which is related to the categorical results by a threshold condition” {i.e. shows threshold} Para. [0123] teaches “The results 314 are correlated with the derived signal 312, in that a pass result is returned when (and only when) the derived signal exceeds a failure threshold 316. As will be appreciated, this is merely one example of a threshold condition that relates a time-sequence of results to a corresponding signal.” {i.e. pass or failure is analogous to fulfillment or non-fulfillment})
With respect to claim 5,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the second criterion is specified by a second threshold, wherein the second characteristic value for the meta-test case is compared with the second threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the meta-test case is made using a comparative result and the logical linking of the first evaluation results of the plurality of test cases. (Para. [0179] teaches “The principles could also be extended to ‘near failure’ events, whose impact is assessed and influences the risk score. A near failure event may be defined, for example, in terms of a second threshold applied to the robustness score (e.g. with a score<0 defined as failure, and a score between 0 and +0.1 defined as near failure). In this case, a near failure event may give rise to a non-zero impact score.” {i.e. second threshold})
With respect to claim 7,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the tested device and/or function for at least partially autonomous guidance of the motor vehicle is object recognition and/or vehicle guidance. (Para. [0001] teaches “The present disclosure pertains to methods for evaluating the performance of trajectory planners in real or simulated scenarios, and computer programs and systems for implementing the same. Such planners are capable of autonomously planning ego trajectories for fully/semi-autonomous vehicles or other forms of mobile robot. Example applications include ADS (Autonomous Driving System) and ADAS (Advanced Driver Assist System) performance testing.” {i.e. Trajectory planning is equivalent to vehicle guidance})
With respect to claim 10,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first characteristic value is determined by comparing ground truth data with the pre-recorded sensor data of the vehicle environment and/or vehicle-related parameters representing a plurality of test cases. (Para. [0066] teaches “the performance of the stack is assessed, at least in part, by evaluating the behaviour of the ego agent in the test oracle against a given set of performance evaluation rules, over the course of one or more runs. The rules are applied to “ground truth” of the (or each) scenario run which, in general, simply means an appropriate representation of the scenario run (including the behaviour of the ego agent) that is taken as authoritative for the purpose of testing. Ground truth is inherent to simulation; a simulator computes a sequence of scenario states, which is, by definition, a perfect, authoritative representation of the simulated scenario run. In a real-world scenario run, a “perfect” representation of the scenario run does not exist in the same sense; nevertheless, suitably informative ground truth can be obtained in numerous ways, e.g. based on manual annotation of on-board sensor data, automated/semi-automated annotation of such data (e.g. using offline/non-real time processing), and/or using external information sources (such as external sensors, maps etc.) etc.” {i.e. ground truth is compared to real world scenario run.)
With respect to claim 11,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the first characteristic value of the pre-recorded sensor data of the vehicle environment and/or vehicle-related parameters representing a plurality of test cases is determined with a fourth threshold. (Para. [0131] teaches “Additionally, the rule tree includes a longitudinal distance branch, and a top-level OR predicate (safe distance node, is_d_safe) to implement a safe distance metric. Similar to the lateral distance branch, the longitudinal distance brand extracts longitudinal distance and longitudinal distance threshold signals from the scenario data (extractor nodes lond and lonsd respectively), and a longitudinal safety assessor node (is_lond_safe) returns TRUE when the longitudinal distance is above the safe longitudinal distance threshold.”{i.e. the safe distance metric is considered a parameter representing a plurality of test cases since the distance signals are time varying as seen in Para. [0129]. The fourth threshold is the safe longitudinal distance threshold.})
With respect to claim 12,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein each test case has a plurality of test sequences, individual images, and/or measurement time series. (Para. [0108] teaches “The scoring is also time-based: for each performance evaluation metric, the test oracle 252 tracks how the value of that metric (the score) changes over time as the simulation progresses. The test oracle 252 provides an output 256 comprising a time sequence 256a of categorical (e.g. pass/fail) results for each rule, and a score-time plot 256b for each performance metric, as described in further detail later.” {i.e. measurement time series} Para. [0074] teaches “The on-board sensor system 110 can take different forms but generally comprises a variety of sensors such as image capture devices” {i.e. images})
With respect to claim 13,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein a parameter space is determined for a virtual test of the device and/or function for at least partially autonomous guidance of a motor vehicle using the evaluation of the second characteristic value of the plurality of test cases. (Para. [0068] teaches “The configurable parameter(s) define a parameter space (also referred to as the scenario space), and the parameterization corresponds to a point in the parameter space” Para. [0173] teaches “Here, the risk score is stored and visualized as a function of the underlying scenario parameters. FIGS. 8 and 9 plot the risk score over a portion of the scenario parameter space. Each shows their respective visualizations 800, 900 in plan and perspective views.”)
With respect to claim 14,
Whiteside teaches,
a data memory to provide pre-recorded sensor data of a vehicle environment and/or vehicle-related parameters representing a plurality of test cases; (Para. [0087] teaches “highly schematic block diagram of a scenario extraction pipeline. Data 140 of a real-world run is passed {i.e. providing pre-recorded sensor data} to a ‘ground-truthing’ pipeline 142 for the purpose of generating scenario ground truth. The run data 140 could comprise, for example, sensor data and/or perception outputs captured/generated on board one or more vehicles {i.e. sensor data of a vehicle environment})
a calculation device to calculate at least one first characteristic value representing a performance of the tested device and/or function for at least partial autonomous guidance of the motor vehicle for each of the plurality of test cases; (Para. [0108] teaches “The rules 254 are categorical in nature (e.g. pass/fail-type rules). Certain performance evaluation rules are also associated with numerical performance metrics used to “score” trajectories (e.g. indicating a degree of success or failure or some other quantity that helps explain or is otherwise relevant to the categorical results). {i.e. performance metrics are seen as first characteristic values})
a first evaluator to evaluate the first characteristic value of each of the plural test cases using a specified first criterion to form a first evaluation result; (Para. [0121] teaches “In addition, each assessor node 304 derives a time-varying numerical signal from the output(s) of its child node(s), which is related to the categorical results by a threshold condition” {i.e. relating to a threshold is seen a specified first criterion} Para. [0123] teaches “The results 314 are correlated with the derived signal 312, in that a pass result is returned when (and only when) the derived signal exceeds a failure threshold 316. As will be appreciated, this is merely one example of a threshold condition that relates a time-sequence of results to a corresponding signal.” {i.e. results are viewed as evaluation result})
an aggregator to aggregate the first characteristic value calculated for each of the plurality of test cases into a second characteristic value representing a meta-test case; (Para. [0108] teaches “The test oracle 252 also provides an overall (aggregate) result for the scenario (e.g. overall pass/fail).”)
and a second evaluator to evaluate the second characteristic value using a specified second criterion and the logical linking of the first evaluation results of the plurality of test cases to form a second evaluation result of the meta-test case. (Para. [0134] teaches “The above examples consider simple logical predicates evaluated on results or signals at a single time instance, such as OR, AND, Gt etc. However, in practice, it may be desirable to formulate certain rules in terms of temporal logic.” {i.e. logical predicates are seen as logically linking.} Para. [0175] teaches “A set of performance evaluation rules is determined that is applicable to a scenario. A numerical performance metric is computed for each performance evaluation rule, in the form of a robustness score that quantifies the extent or success of failure. {i.e. the set of evaluation rules is viewed as the second characteristic value and robustness score is seen as a second evaluation result})
wherein the second characteristic value is calculated using a plurality of first characteristic values of each of the test cases and a normalization value based on a distance traveled by the motor vehicle or a factor representing a distance traveled by the motor vehicle, assigned to each test case. (Para. [0179] teaches “Robustness scores are computed by the test oracle 252 as set out above. A set of performance evaluation rules is determined that is applicable to a scenario. A numerical performance metric is computed for each performance evaluation rule, in the form of a robustness score that quantifies the extent or success of failure. {i.e. using first characteristic values} Para. [0179] further teaches “The robustness score may be normalized, so that zero represents the boundary between pass and fail (a score of zero means the rule has only ‘just’ been failed), with the score preferably normalized to a predetermined range, e.g. [−1,1], with −1 denoting maximum failure and 1 denoting maximum pass. As an example, for a “no collision rule” defined between a pair of agents, the robustness score could be in terms of distance and/or intersection. {i.e. normalized value representing distance}
With respect to claim 15,
Whiteside further teaches,
A computer program with program code to carry out the method according to claim 1 when the computer program is executed on a computer. (Para. [0206] teaches “The subsystems 102-108 of the runtime stack FIG. 1A may be implemented in programmable or dedicated processor(s), or a combination of both, on-board a vehicle or in an off-board computer system in the context of testing and the like.”)
With respect to claim 16,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the at least one first characteristic value is a key performance indicator, which is a quantifiable measure of performance over time for a specific objective. (Para. [0015] teaches “A first aspect herein is directed to a computer-implemented method of evaluating the performance of a trajectory planner for a mobile robot in a scenario, in which the trajectory planner is used to control an ego agent of the scenario responsive to at least one other agent of the scenario, the method comprising: determining a scenario parameter set (set of one or more scenario parameters) for the scenario and a likelihood of the set of scenario parameters; computing an impact score for a dynamic interaction (failure event or near failure event) between the mobile robot and the other agent occurring in the scenario, the impact score quantifying severity of the dynamic interaction; and computing a risk score for the instance of the scenario based on the impact score and the likelihood of the set of scenario parameters.”{i.e. impact score or risk score represent key performance indicators.})
With respect to claim 17,
Whiteside further teaches,
The computer-implemented method according to claim 1, wherein the function for at least partially autonomous guidance of a motor vehicle comprises objection recognition, adaptive cruise control, lane departure warning, active brake assist and / or park assist. (Para. [0100] teaches “In some contexts, this does not require any agent decision making logic 210 at all (open-loop simulation), and in other contexts useful testing can be provided using relatively limited agent logic 210 such as basic adaptive cruise control (ACC).”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 8, and 9 are rejected under 35 U.S.C. 103 as being unpatentable over Whiteside (US 20240419572 A1) as applied to claim 1 above, and further in view of Whiteside (2025) (US 20250123952 A1).
With respect to claim 6,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions.
Whiteside (2025) teaches,
wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions. (Para. [0006] teaches “These rules could include driving rules that evaluate the behaviour of the ego agent based on some model of expected safe driving behaviour in similar driving scenarios, and/or perception rules, that evaluate the accuracy of the ego's perception of its surroundings.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein the meta-test case represents a plurality of test cases of a similar nature or represents test cases recorded under similar environmental conditions and/or similar traffic conditions such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it allows the system to give interpretable results as seen in Para. [0006] “One way to provide interpretable results on the scenario level is to provide a graphical user interface for displaying results for each given scenario instance (or ‘run’) of an ego agent driving in a given set of conditions (real or simulated).” Furthermore, it would be harder to compare or simulate data from dissimilar conditions as the results would be less meaningful.
With respect to claim 8,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases.
Whiteside (2025) teaches,
The computer-implemented method according to claim 1, wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases. (Para. [0034] teaches “A set of rules can then be applied together to a given driving scenario by defining a perception error specification which includes all the rules to be applied. Typical perception rules that may be included in a specification define thresholds on longitudinal and lateral translation errors (measuring mean error of the detection with respect to ground truth in the longitudinal and lateral directions, respectively), orientation error (defining a minimum angle that one needs to rotate the detection to line it up with the corresponding ground truth), size error (error on each dimension of the detected bounding box, or an intersection over union on the aligned ground truth and detected boxes to get a volume delta). Further rules may be based on vehicle dynamics, including errors in the velocity and acceleration of the agents, and errors in classifications, for example defining penalty values for misclassifying a car as a pedestrian or lorry. Rules may also include false positives or missed detections, as well as detection latency.” {i.e. total perception error is calculated using other errors from a plurality of characteristic values making it a third characteristic value.})
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein a third characteristic value is calculated using a plurality of second characteristic values of each of the meta-test cases such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it would help to determine if the system is safe to drive and incorporate complex dependencies as seen in Para. [0035] of Whiteside (2025).
With respect to claim 9,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 8, wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases.
Whiteside (2025) teaches,
wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases. (Para. [0034] teaches “A set of rules can then be applied together to a given driving scenario by defining a perception error specification which includes all the rules to be applied. Typical perception rules that may be included in a specification define thresholds on longitudinal and lateral translation errors (measuring mean error of the detection with respect to ground truth in the longitudinal and lateral directions, respectively), orientation error (defining a minimum angle that one needs to rotate the detection to line it up with the corresponding ground truth), size error (error on each dimension of the detected bounding box, or an intersection over union on the aligned ground truth and detected boxes to get a volume delta). Further rules may be based on vehicle dynamics, including errors in the velocity and acceleration of the agents, and errors in classifications, for example defining penalty values for misclassifying a car as a pedestrian or lorry. Rules may also include false positives or missed detections, as well as detection latency.” {i.e. total perception error is calculated using other errors from a plurality of characteristic values making it a third characteristic value. The third threshold would be the error threshold.}
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein a third criterion is specified by a third threshold, wherein the third characteristic value for a test suite comprising a plurality of meta-test cases is compared with the third threshold, and wherein a determination of a fulfillment or non-fulfillment of a requirement addressed to the test suite is made using a comparative result and the logical linking of the second evaluation results of the plurality of meta-test cases such as that of Whiteside (2025).
One of ordinary skill would have been motivated to modify Whiteside, because it would help to determine if the system is safe to drive and incorporate complex dependencies as seen in Para. [0035] of Whiteside (2025).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Whiteside (US 20240419572 A1) as applied to claim 1 above, and further in view of Venkatadri (US 20220194395 A1).
With respect to claim 18,
Whiteside does not explicitly teach,
The computer-implemented method according to claim 1, wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions.
Venkatadri teaches,
wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions. (Para. [0031] “In some implementations, the data can include sensor data from an autonomous vehicle that operated under an evaluated operating condition that is substantially similar and/or equivalent to the unevaluated operating condition.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Whiteside wherein the meta- test case comprises a plurality of concrete test cases recorded under similar conditions such as that of Venkatadri.
One of ordinary skill would have been motivated to modify Whiteside, because it ensures an accurate comparison between test cases. If the test cases are not tested in similar conditions it would lead to data that is wildly variable and lead to inaccurate results.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to JOSHUA L FORRISTALL whose telephone number is 703-756-4554. The examiner
can normally be reached Monday-Friday 8:30 AM- 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use
the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Schechter can be reached on 571-272-2302. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or
571-272-1000.
/JOSHUA L FORRISTALL/Examiner, Art Unit 2857
/ANDREW SCHECHTER/Supervisory Patent Examiner, Art Unit 2857