DETAILED ACTION
Claims 1-20 are presented for examination.
This Office Action is in response to submission of documents on 3/26/2025.
Rejection of claims 1-20 under 35 U.S.C. 101 for being directed to unpatentable subject matter.
Rejection of claims 1-8, 11-12, and 15-19 under 35 U.S.C. 102(a)(2) as being anticipated by Ashayer.
Rejection of claims 9-10 and 20 under 35 U.S.C. 103 as being obvious over Ashayer in view of Brogle.
Rejection of claims 13-14 under 35 U.S.C. 103 as being obvious over Ashayer in view of Maleki.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to judicial exceptions without significantly more. The claims recite mental processes. This judicial exception is not integrated into a practical application because the additional elements that are recited in the claims are extra-solution activities that do not integrate the judicial exceptions into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because courts have found that the steps of receiving data and recitations of generic computer components are not significantly more than a judicial exception.
Claim 1
Step 1: The claim is directed to a system, falling under one of the four statutory categories of invention.
Step 2A, Prong 1: The claim 1 limitations include (bolded for abstract idea identification):
Claim 1
Mapping Under Step 2A Prong 1
A computer-implemented system, comprising: one or more processing units; and one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
receiving first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
receiving second data collected from a driving simulation that generates a simulated driving scene and operations of a second vehicle located in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
determining a fidelity of the driving simulation based on a comparison between the first data and the second data.
Abstract Idea: Mental Process
The limitation does not limit what is included in “a fidelity,” which could be a judgment on the accuracy. The judgment could be performed in the human mind by observing the two data and, based on opinion, determine whether the similarity is acceptable or not. See e.g., MPEP 2106.04(a)(2).
Step 2A, Prong 2: The claim 1 limitations recite (bolded for additional element identification):
Claim 1
Mapping Under Step 2A Prong 2
A computer-implemented system, comprising: one or more processing units; and one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
receiving first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
receiving second data collected from a driving simulation that generates a simulated driving scene and operations of a second vehicle located in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
determining a fidelity of the driving simulation based on a comparison between the first data and the second data.
Reciting generic computer components is the additional element of instructions to apply the recited judicial exception, which courts have found does not integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The limitation is directed to the extra-solution activity of data gathering (i.e., “receiving…data”). The limitation does not impose meaningful limits on the claim and thus is minimally or tangentially related to the invention. See MPEP 2106.05(g).
The limitation is directed to the extra-solution activity of data gathering (i.e., “receiving…data”). The limitation does not impose meaningful limits on the claim and thus is minimally or tangentially related to the invention. See MPEP 2106.05(g).
Step 2B: Regarding Step 2B, the inquiry is whether any of the additional elements (i.e., the elements that are not the judicial exception) amount to significantly more than the recited judicial exception. Recitations of generic computer components and steps of receiving data are both additional elements that courts have found do not incorporate the recited judicial exception into a practical application. See MPEP 2106.05(f), Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014), Gottschalk v. Benson, 409 U.S. 63, 70, 175 USPQ 673, 676 (1972), Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 112 USPQ2d 1750 (Fed. Cir. 2014); Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 119 USPQ2d 1739 (Fed. Cir. 2016). See also, e.g., In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); In re Meyers, 688 F.2d 789, 794; 215 USPQ 193, 196-97 (CCPA 1982); OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93; CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011).
Accordingly, claim 1 is rejected for being directed to unpatentable subject matter.
Claim 2
Claim 2 recites wherein the reference driving scene is a real-world driving scene. The claim does not add additional elements but instead further specifies additional elements that are already present in claim 1. Accordingly, claim 2 is rejected for being directed to unpatentable subject matter.
Claim 3
Claim 3 recites wherein the determining the fidelity of the driving simulation comprises: comparing the driving behavior of the first vehicle in the reference driving scene at a first time instant to the driving behavior of the second vehicle in the simulated driving scene at a second time instant, the second time instant based on a correlation between a first timestamp in the first data and a second timestamp in the second data. Comparing driving behavior is a mental process that can include observing a first driving behavior and noting differences between the first driving behavior and observations of a second driving behavior. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 3 is rejected for being directed to unpatentable subject matter.
Claim 4
Claim 4 recites wherein: the first data includes an indication of a pose of the first vehicle with respect to the reference driving scene; the second data includes an indication of a pose of the second vehicle with respect to the simulated driving scene; and the determining the fidelity of the driving simulation comprises: comparing the pose of the first vehicle with respect to the reference driving scene to the pose of the second vehicle with respect to the simulated driving scene. Comparing pose (e.g., location) of a vehicle is a mental process that can include observing a first vehicle and noting differences between the location of the first vehicle and observations of a second vehicle location. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 4 is rejected for being directed to unpatentable subject matter.
Claim 5
Claim 5 recites wherein: the first data includes an indication of a vehicle dynamic of the first vehicle; the second data includes an indication of a vehicle dynamic of the second vehicle; and the determining the fidelity of the driving simulation comprises: comparing the vehicle dynamic of the first vehicle to the vehicle dynamic of the second vehicle. Comparing vehicle dynamics is a mental process (or, if the dynamic is a numerical value, a mathematical concept) that can include observing a first vehicle and noting (or calculating) differences between the first vehicle and observations of a second vehicle. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 5 is rejected for being directed to unpatentable subject matter.
Claim 6
Claim 6 recites wherein: the first data includes an indication of a placement of a first object in the reference driving scene; the second data includes an indication of a placement of a second object in the simulated driving scene; and the determining the fidelity of the driving simulation comprises: comparing the placement of the first object in the reference driving scene to the placement of the second object in the simulated driving scene. Comparing locations of vehicles is a mental process that can include observing a first vehicle and noting differences between the location of the first vehicle and observations of a second vehicle location. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 6 is rejected for being directed to unpatentable subject matter.
Claim 7
Claim 7 recites wherein: the first data includes an indication of an attribute of a first road in the reference driving scene; the second data includes an indication of an attribute of a second road in the simulated driving scene; and the determining the fidelity of the driving simulation comprises: comparing the attribute of the first road in the reference driving scene to the attribute of the second road in the simulated driving scene. Comparing attributes between a real environment and a simulated environment is a mental process that includes observing both road attributes and evaluating the attributes to determining similarities and differences. Accordingly, claim 7 is rejected for being directed to unpatentable subject matter.
Claim 8
Claim 8 recites wherein: the first data includes a classification of a first object in the reference driving scene determined by the first vehicle; the second data includes a classification of a second object in the simulated driving scene determined by the second vehicle; and the determining the fidelity of the driving simulation comprises: comparing the classification of the first object to the classification of the second object. Comparing classification of objects in a real environment and a simulated environment is a mental process that includes observing both classifications and evaluating to determine similarities and differences. According, claim 8 is rejected for being directed to unpatentable subject matter.
Claim 9
Claim 9 recites wherein: the first data includes first sensor data collected from the reference driving scene; the second data includes second sensor data collected from the simulated driving scene; and the determining the fidelity of the driving simulation comprises: comparing the first sensor data to the second sensor data. Sensor data can be qualitatively compared (a mental process) or numerically compared (a mathematical concept). In either instance, the claim recites an abstract idea and is therefore a judicial exception to patentable subject matter. Accordingly, claim 9 is directed to unpatentable subject matter.
Claim 10
Claim 10 recites wherein: the first sensor data is collected from a sensor of the first vehicle; the second sensor data is collected from a simulated sensor in the simulated driving scene; and the sensor of the first vehicle and the simulated sensor are of the same sensor modality. The claim does not include additional elements because it further specifies elements already present in parent claims. Instead, the claim further specifies a generator of sensor data. Accordingly, claim 10 is directed to unpatentable subject matter.
Step 1: The claim is directed to a system, falling under one of the four statutory categories of invention.
Step 2A, Prong 1: The claim 1 limitations include (bolded for abstract idea identification):
Claim 11
Mapping Under Step 2A Prong 1
A computer-implemented system, comprising: one or more processing units; and one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
executing a simulation that simulates operations of a simulated vehicle driving in a simulated driving scene;
collecting, from the simulation, first data associated with the simulated driving scene and a driving behavior of the simulated vehicle in the simulated driving scene;
determining a plurality of differences between measurements of the first data and second data, the second data associated with a real-world driving scene and a driving behavior of a real-world vehicle in the real-world driving scene; and
processing the plurality of differences using a simulation integrity validation model to generate a simulation integrity score.
Abstract Idea: Mathematical Concepts
A simulation is comprised of performing mathematical calculations to mimic real-world physical behavior via a model. See MPEP § 2106.04(a)(2), Subsection I.
Abstract Idea: Mathematical Concepts
Comparing numerical values is a mathematical concept. See e.g., MPEP 2106.04(a)(2), Subsection III.
Abstract Idea: Mathematical Calculations
Using a model and/or simulation to predict the validity of a system is a mathematical concept that includes performing one or more operations according to functions that describe the system. Thus, a simulation is the judicial exception of a mathematical concepts. See MPEP 2106.04(a)(2), Subsection I.
Step 2A, Prong 2: The claim 1 limitations recite (bolded for additional element identification):
Claim 11
Mapping Under Step 2A Prong 2
A computer-implemented system, comprising: one or more processing units; and one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
executing a simulation that simulates operations of a simulated vehicle driving in a simulated driving scene;
collecting, from the simulation, first data associated with the simulated driving scene and a driving behavior of the simulated vehicle in the simulated driving scene;
Determining a plurality of differences between measurements of the first data and second data, the second data associated with a real-world driving scene and a driving behavior of a real-world vehicle in the real-world driving scene; and
processing the plurality of differences using a simulation integrity validation model to generate a simulation integrity score.
Reciting generic computer components is the additional element of instructions to apply the recited judicial exception, which courts have found does not integrate the judicial exception into a practical application. See MPEP 2106.05(f).
The limitation is directed to the extra-solution activity of data gathering. The limitation does not impose meaningful limits on the claim and thus is minimally or tangentially related to the invention. See MPEP 2106.05(g).
Step 2B: Regarding Step 2B, the inquiry is whether any of the additional elements (i.e., the elements that are not the judicial exception) amount to significantly more than the recited judicial exception. Courts have found that the extra-solution activity of data gathering is insignificantly more than the recited judicial exception. See, e.g., In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); In re Meyers, 688 F.2d 789, 794; 215 USPQ 193, 196-97 (CCPA 1982); OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93; CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011).
Accordingly, claim 11 is rejected for being directed to unpatentable subject matter.
Claim 12
Claim 12 recites wherein an individual difference of the plurality of differences corresponds to a difference between a component of a plurality of components in the real-world driving scene and a respective one of a plurality of simulated components used by the simulation. Comparing numerical values is a mathematical concept. See e.g., MPEP 2106.04(a)(2), Subsection III. Accordingly, claim 12 is rejected for being directed to unpatentable subject matter.
Claim 13
Claim 13 recites wherein the simulation integrity validation model is a statistical model. The claim further details the mathematical concepts of the validation model. Accordingly, claim 13 is rejected for being directed to unpatentable subject matter.
Claim 14
Claim 14 recites wherein the simulation integrity validation model is a generalized linear model (GLM). The claim further details the mathematical concepts of the validation model. Accordingly, claim 14 is rejected for being directed to unpatentable subject matter.
Claim 15
Claim 15 recites wherein the simulation integrity score includes an indication of a pass or a failure. The claim further details a mental process, whereby a score can be observed and, through judgment and evaluation, be classified as a pass or fail. Accordingly, claim 15 is rejected for being directed to unpatentable subject matter.
Claim 16
Step 1: The claim is directed to a process, falling under one of the four statutory categories of invention.
Step 2A, Prong 1: The claim 1 limitations include (bolded for abstract idea identification):
Claim 16
Mapping Under Step 2A Prong 1
A method comprising: receiving, by a computer-implemented system, first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
receiving, by the computer-implemented system, second data collected from an end-to- end (E2E) vehicle simulation of a simulated driving scene and operations of a second vehicle in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
determining, by the computer-implemented system, a fidelity of the E2E simulation based on a comparison between the first data and the second data.
Abstract Idea: Mental Process
The limitation does not limit what is included in “a fidelity,” which could be a judgment on the accuracy. The judgment could be performed in the human mind by observing the two data and, based on opinion, determine whether the similarity is acceptable or not. See e.g., MPEP 2106.04(a)(2).
Step 2A, Prong 2: The claim 1 limitations recite (bolded for additional element identification):
Claim 16
Mapping Under Step 2A Prong 2
A method comprising: receiving, by a computer-implemented system, first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
receiving, by the computer-implemented system, second data collected from an end-to- end (E2E) vehicle simulation of a simulated driving scene and operations of a second vehicle in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
determining, by the computer-implemented system, a fidelity of the E2E simulation based on a comparison between the first data and the second data.
The limitation is directed to the extra-solution activity of data gathering. The limitation does not impose meaningful limits on the claim and thus is minimally or tangentially related to the invention. See MPEP 2106.05(g).
The limitation is directed to the extra-solution activity of data gathering. The limitation does not impose meaningful limits on the claim and thus is minimally or tangentially related to the invention. See MPEP 2106.05(g).
Regarding Step 2B, the inquiry is whether any of the additional elements (i.e., the elements that are not the judicial exception) amount to significantly more than the recited judicial exception. Courts have found that the extra-solution activity of data gathering is insignificantly more than the recited judicial exception. See, e.g., In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); In re Meyers, 688 F.2d 789, 794; 215 USPQ 193, 196-97 (CCPA 1982); OIP Technologies, 788 F.3d at 1363, 115 USPQ2d at 1092-93; CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011).
Claim 17
Claim 17 recites wherein the determining the fidelity of the E2E simulation comprises: comparing the driving behavior of the first vehicle in the reference driving scene at a first time instant to the driving behavior of the second vehicle in the simulated driving scene at a second time instant corresponding to the first time instant. Comparing driving behavior is a mental process that can include observing a first driving behavior and noting differences between the first driving behavior and observations of a second driving behavior. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 17 is rejected for being directed to unpatentable subject matter.
Claim 18
Claim 18 recites wherein: the first data includes an indication of a pose of the first vehicle with respect to the reference driving scene; the second data includes an indication of a pose of the second vehicle with respect to the simulated driving scene; and the determining the fidelity of the E2E simulation comprises: comparing the pose of the first vehicle with respect to the reference driving scene to the pose of the second vehicle with respect to the simulated driving scene. Comparing pose (e.g., location) of a vehicle is a mental process that can include observing a first vehicle and noting differences between the location of the first vehicle and observations of a second vehicle location. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 18 is rejected for being directed to unpatentable subject matter.
Claim 19
Claim 19 recites wherein: the first data includes an indication of a placement of a first object in the reference driving scene; the second data includes an indication of a placement of a second object in the simulated driving scene; and the determining the fidelity of the E2E simulation comprises: comparing the placement of the first object in the reference driving scene to the placement of the second object in the simulated driving scene. Comparing locations of vehicles is a mental process that can include observing a first vehicle and noting differences between the location of the first vehicle and observations of a second vehicle location. See MPEP 2106.04(a)(2), Subsection III. The claim does not include additional limitations beyond the recited judicial exception. Accordingly, claim 19 is rejected for being directed to unpatentable subject matter.
Claim 20
Claim 20 recites wherein: the first data includes first sensor data collected from the reference driving scene; the second data includes second sensor data collected from the simulated driving scene; and the determining the fidelity of the E2E simulation comprises: comparing the first sensor data to the second sensor data. Sensor data can be qualitatively compared (a mental process) or numerically compared (a mathematical concept). In either instance, the claim recites an abstract idea and is therefore a judicial exception to patentable subject matter. Accordingly, claim 20 is rejected for being directed to unpatentable subject matter.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-8, 11-12, and 15-19 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ashayer, et al., (U.S. Pat. Pub. No. 2022/0283055, hereinafter “Ashayer”).
Claim 1
Ashayer discloses:
A computer-implemented system, comprising: one or more processing units; and one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
The memory 620 computing device(s) 604 and the memory 636 of the computing device(s) 632 are examples of non-transitory computer-readable media. Ashayer at [0090].
receiving first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
FIG. 1 is a pictorial flow diagram illustrating an example data flow 100 in which vehicle(s) 102 generate log data 104 associated with an environment 106 and transmit the log data to computing device(s) 108 that determine associations 110 between locations of the vehicle(s) 102 and object data captured by the vehicle(s) 102 at different instances in time. Ashayer at [0028].
The environment 106 is analogous to the “reference driving scene.”
The locations of the vehicles 102 is analogous to “driving behavior of a first vehicle.”
receiving second data collected from a driving simulation that generates a simulated driving scene and operations of a second vehicle located in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and
FIG. 3 illustrates an example data conversion 300 that is performed at least partially by a vehicle perception system 302. The vehicle perception system 302 may output different representations of an environment (e.g., environment 106) in which the vehicle (e.g., vehicle 102) is operating based on different perspectives of the environment as perceived by the vehicle from different locations at different instances of time, as described herein. Ashayer at [0046].
wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
The prediction component 626 may generate one or more probability maps representing prediction probabilities of possible locations of one or more objects in an environment. For example, the prediction component 626 may generate one or more probability maps for vehicles, pedestrians, animals, and the like within a threshold distance from the vehicle 602. Ashayer at [0073].
The one or more objects in a simulated environment includes vehicles, which is a second vehicle that is in the simulation. The location (i.e., “threshold distance”) is analogous to “driving behavior of the second vehicle.”
determining a fidelity of the driving simulation based on a comparison between the first data and the second data.
The techniques discussed herein can improve the accuracy of simulations for testing and/or validating vehicle controllers. For instance, by determining a closest prior location of a log vehicle to a simulated vehicle's current location and instantiating static objects in the simulated environment that the log vehicle would have perceived from that prior location, the dynamic aspects of static objects may be realistically simulated. Ashayer at [0016].
The “accuracy of simulations” is analogous to a “fidelity of the driving simulation.”
Claim 2
Ashayer discloses:
wherein the reference driving scene is a real-world driving scene.
Additionally, the simulated object may be positioned in the simulated environment at a location that corresponds with where the object is positioned in the real-world environment. Ashayer at [0022].
Claim 3
Ashayer discloses:
wherein the determining the fidelity of the driving simulation comprises: comparing the driving behavior of the first vehicle in the reference driving scene at a first time instant to the driving behavior of the second vehicle in the simulated driving scene at a second time instant, the second time instant based on a correlation between a first timestamp in the first data and a second timestamp in the second data.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
The “the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment” is analogous to the driving behavior of the first and second vehicles.
The “first time” in the environment is correlated with the logs of the simulation, thus correlating a time in the environment with a time in the simulation.
Claim 4
Ashayer discloses:
wherein: the first data includes an indication of a pose of the first vehicle with respect to the reference driving scene;
...a real pose of the real vehicle in the real environment… Ashayer at [0103].
the second data includes an indication of a pose of the second vehicle with respect to the simulated driving scene; and
…a simulated pose of the simulated vehicle in the simulated environment… Ashayer at [0103].
the determining the fidelity of the driving simulation comprises: comparing the pose of the first vehicle with respect to the reference driving scene to the pose of the second vehicle with respect to the simulated driving scene.
…the simulated vehicle may be oriented on a heading of 90 degrees from North, while the real vehicle is oriented on a heading of 93 degrees from the North… Ashayer at [0103].
Claim 5
Ashayer discloses:
wherein: the first data includes an indication of a vehicle dynamic of the first vehicle; the second data includes an indication of a vehicle dynamic of the second vehicle; and the determining the fidelity of the driving simulation comprises: comparing the vehicle dynamic of the first vehicle to the vehicle dynamic of the second vehicle.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
The “real location” and “simulated location” are analogous to “vehicle dynamics.”
Claim 6
Ashayer discloses:
wherein: the first data includes an indication of a placement of a first object in the reference driving scene; the second data includes an indication of a placement of a second object in the simulated driving scene; and the determining the fidelity of the driving simulation comprises: comparing the placement of the first object in the reference driving scene to the placement of the second object in the simulated driving scene.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
Claim 7
Ashayer discloses:
wherein: the first data includes an indication of an attribute of a first road in the reference driving scene;
In contrast, “static objects” may be objects that are associated with the environment 106 such as, for example, buildings/structures, road surfaces, road markers, signage, barriers, trees, sidewalks, streetlights, parked vehicles, etc. In some instances, the computing device associated with the vehicle(s) 102 may determine information about objects in the environment, such as bounding boxes, classifications, segmentation information, and the like. Ashayer at [0031].
A “road surface” is an attribute of the road in the environment.
the second data includes an indication of an attribute of a second road in the simulated driving scene; and the
Accordingly, this disclosure describes techniques for improving simulations based on vehicle logs by determining a closest location of the log vehicle to the current, simulated vehicle's location and instantiating static objects in the simulated environment that the log vehicle would have perceived at that time and/or from that location. Ashayer at [0015].
determining the fidelity of the driving simulation comprises: comparing the attribute of the first road in the reference driving scene to the attribute of the second road in the simulated driving scene.
The techniques discussed herein can improve the accuracy of simulations for testing and/or validating vehicle controllers. For instance, by determining a closest prior location of a log vehicle to a simulated vehicle's current location and instantiating static objects in the simulated environment that the log vehicle would have perceived from that prior location, the dynamic aspects of static objects may be realistically simulated. Ashayer at [0016].
“Static objects” includes “road surface” and by comparing the actual surface with the simulated surface in the manner disclosed, the accuracy (i.e., “fidelity”) of the simulation is improved.
Claim 8
Ashayer discloses:
wherein: the first data includes a classification of a first object in the reference driving scene determined by the first vehicle; the second data includes a classification of a second object in the simulated driving scene determined by the second vehicle; and the
At operation 712, the method 700 includes obtaining, from the log data, object data representing a real object in the real environment that was perceived by the real vehicle at the first time. For instance, the computing device(s) 108 may obtain object data from the log data associations 130 based at least in part on which prior location of the vehicle 102 that the simulated vehicle is closest to. At operation 716, the method 700 includes generating, in the simulated environment, a simulated object representing the real object such that the simulated vehicle perceives the simulated object. For instance, the computing device(s) 108 may instantiate the simulated object within the simulated environment at the location in which the vehicle 102 perceived the object. Ashayer at [0099].
determining the fidelity of the driving simulation comprises: comparing the classification of the first object to the classification of the second object.
In some examples, whether or not the simulated object is generated in the simulated environment may be based at least in part on determining a classification of the real object. That is, if the real object is a dynamic object, then the simulated object may not be generated. However, if the real object is a static object, then the simulated object may be generated and/or instantiated within the simulated environment.. Ashayer at [0099].
Claim 11
Ashayer discloses:
A computer-implemented system, comprising: one or more processing units; and
The processor(s) 618 of the computing device(s) 604 and the processor(s) 634 of the computing device(s) 632 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, the processor(s) 618 and 634 may comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that may be stored in registers and/or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors in so far as they are configured to implement encoded instructions. Ashayer at [0089].
one or more non-transitory computer-readable media storing instructions, when executed by the one or more processing units, cause the one or more processing units to perform operations comprising:
The memory 620 computing device(s) 604 and the memory 636 of the computing device(s) 632 are examples of non-transitory computer-readable media. The memory 620 and 636 may store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems. Ashayer at [0090].
executing a simulation that simulates operations of a simulated vehicle driving in a simulated driving scene;
At operation 704, the method 700 includes generating, using the log data, a simulated scenario for testing an autonomous vehicle controller, the simulated scenario including a simulated environment representing at least a portion of the real environment. Ashayer at [0096].
collecting, from the simulation, first data associated with the simulated driving scene and a driving behavior of the simulated vehicle in the simulated driving scene;
FIG. 3 illustrates an example data conversion 300 that is performed at least partially by a vehicle perception system 302. The vehicle perception system 302 may output different representations of an environment (e.g., environment 106) in which the vehicle (e.g., vehicle 102) is operating based on different perspectives of the environment as perceived by the vehicle from different locations at different instances of time, as described herein. Ashayer at [0046].
The prediction component 626 may generate one or more probability maps representing prediction probabilities of possible locations of one or more objects in an environment. For example, the prediction component 626 may generate one or more probability maps for vehicles, pedestrians, animals, and the like within a threshold distance from the vehicle 602. Ashayer at [0073].
determining a plurality of differences between measurements of the first data and second data, the second data associated with a real-world driving scene and a driving behavior of a real-world vehicle in the real-world driving scene; and
The techniques discussed herein can improve the accuracy of simulations for testing and/or validating vehicle controllers. For instance, by determining a closest prior location of a log vehicle to a simulated vehicle's current location and instantiating static objects in the simulated environment that the log vehicle would have perceived from that prior location, the dynamic aspects of static objects may be realistically simulated. Ashayer at [0016].
processing the plurality of differences using a simulation integrity validation model to generate a simulation integrity score.
The techniques discussed herein can improve the accuracy of simulations for testing and/or validating vehicle controllers. For instance, by determining a closest prior location of a log vehicle to a simulated vehicle's current location and instantiating static objects in the simulated environment that the log vehicle would have perceived from that prior location, the dynamic aspects of static objects may be realistically simulated. Ashayer at [0016].
Claim 12
Ashayer discloses:
wherein an individual difference of the plurality of differences corresponds to a difference between a component of a plurality of components in the real-world driving scene and a respective one of a plurality of simulated components used by the simulation.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
Claim 15
Ashayer discloses:
wherein the simulation integrity score includes an indication of a pass or a failure.
Based at least in part on determining that the autonomous controller performed consistent with the predetermined outcome (that is, the autonomous controller did everything it was supposed to do) and/or determining that a rule was not broken or an assertion was not triggered, the simulation component 644 may determine that the autonomous controller succeeded. Ashayer at [0088].
Claim 16
Ashayer discloses:
A method comprising: receiving, by a computer-implemented system, first data collected from a reference driving scene, the first data associated with the reference driving scene and a driving behavior of a first vehicle in the reference driving scene;
FIG. 1 is a pictorial flow diagram illustrating an example data flow 100 in which vehicle(s) 102 generate log data 104 associated with an environment 106 and transmit the log data to computing device(s) 108 that determine associations 110 between locations of the vehicle(s) 102 and object data captured by the vehicle(s) 102 at different instances in time. Ashayer at [0028].
The environment 106 is analogous to the “reference driving scene.”
The locations of the vehicles 102 is analogous to “driving behavior of a first vehicle.”
receiving, by the computer-implemented system, second data collected from an end-to- end (E2E) vehicle simulation of a simulated driving scene and operations of a second vehicle in the simulated driving scene, wherein the simulated driving scene is a simulation of the reference driving scene, and wherein the second data is associated with the simulated driving scene and a driving behavior of the second vehicle in the simulated driving scene; and
FIG. 3 illustrates an example data conversion 300 that is performed at least partially by a vehicle perception system 302. The vehicle perception system 302 may output different representations of an environment (e.g., environment 106) in which the vehicle (e.g., vehicle 102) is operating based on different perspectives of the environment as perceived by the vehicle from different locations at different instances of time, as described herein. Ashayer at [0046].
determining, by the computer-implemented system, a fidelity of the E2E simulation based on a comparison between the first data and the second data.
The techniques discussed herein can improve the accuracy of simulations for testing and/or validating vehicle controllers. For instance, by determining a closest prior location of a log vehicle to a simulated vehicle's current location and instantiating static objects in the simulated environment that the log vehicle would have perceived from that prior location, the dynamic aspects of static objects may be realistically simulated. Ashayer at [0016].
The “accuracy of simulations” is analogous to a “fidelity of the driving simulation.”
Claim 17
Ashayer discloses:
wherein the determining the fidelity of the E2E simulation comprises: comparing the driving behavior of the first vehicle in the reference driving scene at a first time instant to the driving behavior of the second vehicle in the simulated driving scene at a second time instant corresponding to the first time instant.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
Claim 18
Ashayer discloses:
wherein: the first data includes an indication of a pose of the first vehicle with respect to the reference driving scene;
...a real pose of the real vehicle in the real environment… Ashayer at [0103].
the second data includes an indication of a pose of the second vehicle with respect to the simulated driving scene; and
…a simulated pose of the simulated vehicle in the simulated environment… Ashayer at [0103].
the determining the fidelity of the E2E simulation comprises: comparing the pose of the first vehicle with respect to the reference driving scene to the pose of the second vehicle with respect to the simulated driving scene.
…the simulated vehicle may be oriented on a heading of 90 degrees from North, while the real vehicle is oriented on a heading of 93 degrees from the North… Ashayer at [0103].
Claim 19
Ashayer discloses:
wherein: the first data includes an indication of a placement of a first object in the reference driving scene; the second data includes an indication of a placement of a second object in the simulated driving scene; and the determining the fidelity of the E2E simulation comprises: comparing the placement of the first object in the reference driving scene to the placement of the second object in the simulated driving scene.
At operation 710, the method 700 includes determining, based at least in part on the log data, a first time during the period of time in which a real location of the real vehicle in the real environment corresponded closest to the simulated location of the simulated autonomous vehicle in the simulated environment. For instance, the computing device(s) 108 may determine a first time (e.g., to 112) in which the prior location 206(1) of the vehicle 102 was closest to the location of the simulated vehicle 508. Additionally, or alternatively, the computing device(s) 108 may determine the first time based at least in part on the log data associations 130. Ashayer at [0098].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 9-10 and 20 are rejected under 35 U.S.C. 103 as being obvious over Ashayer in view of Brogle, et al., (Hardware-in-the-Loop Autonomous Driving Simulation Without Real-Time Constraints, hereinafter “Brogle”).
Claim 9
Ashayer discloses:
wherein: the first data includes first sensor data collected from the reference driving scene;
[T]he log data may have been generated by a perception system of the vehicle based at least in part on sensor data captured by one or more sensors of the vehicle. In at least one example, the log data includes multiple instances of log data representing the environment in which the vehicle and/or another vehicle were operating during multiple periods of time…..In some examples, the method may include generating, using the log data, a simulated scenario for testing a vehicle controller. Ashayer at [0017]-[0018].
Ashayer does not appear to disclose:
the second data includes second sensor data collected from the simulated driving scene;
and the determining the fidelity of the driving simulation comprises: comparing the first sensor data to the second sensor data.
Brogle, which is analogous art, discloses:
the second data includes second sensor data collected from the simulated driving scene;
This resulted in the interface consisting of a single ROS node written in Python which retrieves sensor and environment data from the CARLA application programming interface (specifically, two camera feeds, a LiDAR point cloud and pose and velocity information of the vehicle), and published this information to the topics expected by the video processing and LiDAR processing nodes. Brogle at pg. 377, col. 2.
and the determining the fidelity of the driving simulation comprises: comparing the first sensor data to the second sensor data.
The inclusion of HIL allows a much narrower “reality gap” than a purely software-based system, as all vehicle responses are “real” in a sense that they come from the actual drive computer at the correct “real” vehicle timing conditions. Brogle at pg. 377, cols. 1-2.
Brogle is analogous prior art because both are related to generating a simulation that is realistic and simulates sensors of the simulated vehicle. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to combine the sensor data logs generated during the real life scenario, as disclosed in Ashayer, with the simulate sensor data from the same scene, as disclosed in Brogle, to allow the simulation to be evaluated based on similarity between the data. Motivation to combine includes a safer sensor system by allowing sensors to be tested in a simulation environment that is realistic compared to a real environment, thereby improving the accuracy of the sensor design.
Claim 10
Ashayer discloses:
wherein: the first sensor data is collected from a sensor of the first vehicle;
[T]he log data may have been generated by a perception system of the vehicle based at least in part on sensor data captured by one or more sensors of the vehicle. In at least one example, the log data includes multiple instances of log data representing the environment in which the vehicle and/or another vehicle were operating during multiple periods of time…..In some examples, the method may include generating, using the log data, a simulated scenario for testing a vehicle controller. Ashayer at [0017]-[0018].
Ashayer does not appear to disclose:
the second sensor data is collected from a simulated sensor in the simulated driving scene; and
the sensor of the first vehicle and the simulated sensor are of the same sensor modality.
Brogle discloses:
the second sensor data is collected from a simulated sensor in the simulated driving scene; and
This resulted in the interface consisting of a single ROS node written in Python which retrieves sensor and environment data from the CARLA application programming interface (specifically, two camera feeds, a LiDAR point cloud and pose and velocity information of the vehicle), and published this information to the topics expected by the video processing and LiDAR processing nodes. Brogle at pg. 377, col. 2..
the sensor of the first vehicle and the simulated sensor are of the same sensor modality.
In order for simulated testing to be effective, a high degree of similarity in vehicle dynamics and sensor outputs must be maintained between the simulated and real systems. As a result, validation of these aspects of the HIL simulation system against a physical autonomous driving platform was undertaken. Brogle at pg. 380, col. 2.
Claim 20
Ashayer discloses:
wherein: the first data includes first sensor data collected from the reference driving scene;
[T]he log data may have been generated by a perception system of the vehicle based at least in part on sensor data captured by one or more sensors of the vehicle. In at least one example, the log data includes multiple instances of log data representing the environment in which the vehicle and/or another vehicle were operating during multiple periods of time…..In some examples, the method may include generating, using the log data, a simulated scenario for testing a vehicle controller. Ashayer at [0017]-[0018].
Ashayer does not appear to disclose:
the second data includes second sensor data collected from the simulated driving scene; and the determining the fidelity of the E2E simulation comprises:
comparing the first sensor data to the second sensor data.
Brogle discloses:
the second data includes second sensor data collected from the simulated driving scene; and the determining the fidelity of the E2E simulation comprises:
This resulted in the interface consisting of a single ROS node written in Python which retrieves sensor and environment data from the CARLA application programming interface (specifically, two camera feeds, a LiDAR point cloud and pose and velocity information of the vehicle), and published this information to the topics expected by the video processing and LiDAR processing nodes. Brogle at pg. 377, col. 2.
comparing the first sensor data to the second sensor data.
The inclusion of HIL allows a much narrower “reality gap” than a purely software-based system, as all vehicle responses are “real” in a sense that they come from the actual drive computer at the correct “real” vehicle timing conditions. Brogle at pg. 377, cols. 1-2.
Claims 13-14 are rejected under 35 U.S.C. 103 as being obvious over Ashayer in view of Maleki (U.S. Pat. Pub. No 2022/0219691).
Claim 13
Ashayer does not appear to disclose:
wherein the simulation integrity validation model is a statistical model.
Maleki, which is analogous art, discloses:
wherein the simulation integrity validation model is a statistical model.
Predictive models are not equally accurate and are not at the same level of robustness and prediction accuracy for drive analytics applications. To find the best model for drive analytics purposes, a brute force approach can be utilized. Referring now to FIGS. 38A and 38B, a prediction robustness analysis algorithm, in accordance with aspects of the present technology, is shown. The most common predictive algorithms can be selected to generate a pool of predictive models 3810. The pool can contain several models from different classes of predictive algorithms. From a class of artificial neural nets, a shallow artificial neural net (SANN) and a deep neural net (DNN) can be chosen. From a class of tree-based algorithms, classification and regression tree (CART) and random forest (RF) can be selected. The pool can also contain two lazy models including k-nearest neighbor (KNN) and KStar. A support vector machine (SVM-nu) and an SVM-epsilon support vector machine can also be incorporated. From the class of statistical models, a generalized adaptive model (GAM) and generalized linear model (GLM) can be chosen. Maleki at [0331].
Maleki is analogous art to the claimed invention because both are directed to simulations for autonomous driving vehicles. It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to apply the models disclosed in Maleki to the simulation analysis of Ashayer, thus resulting in a system that uses additional types of statistical models to validate a simulation. Motivation to do so includes improves analysis techniques which improve the evaluation process for a simulation, thus resulting in better simulations being selected for further use in testing of sensors and autonomous driving test.
Claim 14
Ashayer does not appear to disclose:
wherein the simulation integrity validation model is a generalized linear model (GLM).
Predictive models are not equally accurate and are not at the same level of robustness and prediction accuracy for drive analytics applications. To find the best model for drive analytics purposes, a brute force approach can be utilized. Referring now to FIGS. 38A and 38B, a prediction robustness analysis algorithm, in accordance with aspects of the present technology, is shown. The most common predictive algorithms can be selected to generate a pool of predictive models 3810. The pool can contain several models from different classes of predictive algorithms. From a class of artificial neural nets, a shallow artificial neural net (SANN) and a deep neural net (DNN) can be chosen. From a class of tree-based algorithms, classification and regression tree (CART) and random forest (RF) can be selected. The pool can also contain two lazy models including k-nearest neighbor (KNN) and KStar. A support vector machine (SVM-nu) and an SVM-epsilon support vector machine can also be incorporated. From the class of statistical models, a generalized adaptive model (GAM) and generalized linear model (GLM) can be chosen. Maleki at [0331].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPH MORRIS whose telephone number is (703)756-5735. The examiner can normally be reached M-F 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ryan Pitaro can be reached at (571) 272-4071. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JOSEPH MORRIS
Examiner
Art Unit 2188
/JOSEPH P MORRIS/Examiner, Art Unit 2188
/RYAN F PITARO/Supervisory Patent Examiner, Art Unit 2188
Prosecution Insights