DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 05/13/2026
Claims 1-29 are presented for examination
Information Disclosure Statement
The IDS dated 05/13/2026 has been reviewed. See attached.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 05/13/2026 has been entered.
Response to Arguments- 35 USC § 101
Applicant’s arguments, see page 13, filed 05/13/2026, with respect to the rejection of claims 1-29 under 35 USC § 101 have been fully considered and are persuasive. The rejection of claims 1-29 under 35 USC § 101 has been withdrawn.
Specifically, the particular way in which the machine learning model is trained successfully integrates the claims into a practical application. See Examples 38 and 39 of the 2019 Revised Patent Subject Matter Eligibility Guidance as well as the more recent guidance in view of Ex Parte Desjardins.
Response to Arguments- 35 USC § 103
Applicant's arguments filed 05/13/2026 have been fully considered but they are not persuasive.
Applicant argues that no prior art teaches generating, for each pose of the plurality of installation poses, respective ground truth data for the one or more sensor data representations of the pose based at least on mapping one or more road features of one or more roads in the virtual environment to the one or more sensor data representations of the pose,
Examiner responds by explaining that, firstly, despite the arguments to the contrary, the claims do not actually require that the ground truth be different for each pose, merely that each pose has respective associated ground truth data, regardless of the content of that ground truth. Indeed, it would not make sense for the ground truth to be different for each pose, as the purpose of the claimed invention is to enable the training of a machine learning model that can operate effectively regardless of physical sensor configuration; having this ground truth be different for each sensor (e.g. the ground truth for processing using a front camera defining an object as being a stop sign while the ground truth for processing using a side camera defines that same object as an oncoming vehicle) would lead to inaccurate training. Ground truth is meant to reflect the objective reality that the training mechanism is designed to train the model towards being able to recognize; trying to train different sensors on different objective realities makes convergence between the sensors unlikely. This convergence, i.e. the idea that the objective reality of the driving situation, such as the location of another vehicle relative to the ego vehicle, can be accurately determined regardless of sensor position, is the main purpose of the claimed invention, and the argued interpretation that the claims require different ground truths, i.e. that the training related to each sensor pose should be optimized towards each sensor simultaneously perceiving separate realities seems to go against the body of the disclosure. See [Par 26] of the specification, which discusses the generation and use of ground truth data; although this ground truth corresponds to the plurality of sensors, the actual content of this ground truth data, e.g. lane labels and annotations, are consistently applied to the respective environment-space objects. In other words, whether it is perceived by one sensor or another, the ground truth regarding a stop sign at a certain position in the environment defines it as a stop sign at that position.
As such, VESPA teaches generating, for each pose of the plurality of installation poses, respective ground truth data for the one or more sensor data representations of the pose ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”) based at least on mapping one or more road features of one or more roads in the ([Page 3 Col 1 Par 3 – Col 2 Par 1] “To quantify the performance of a sensor configuration on a vehicle being evaluated over drive cycle test cases (i.e., across various driving scenarios; see Section V), we define eight metrics (m1-m8) that are characteristic of the configuration’s ability to track and detect non-ego vehicles across various road geometries and traffic scenarios. The eight metrics are defined as follows: … The rate of late detection metric (m5) is computed as a fraction of the number of ‘late’ non ego vehicle detections made by the total number of non-ego vehicles, which matters for BW. A detection is classified as late if it is made after the non-ego vehicle crosses the minimum safe longitudinal or lateral distance defined by Intel RSS (Responsibility Sensitive Safety) models on NHTSA for pre-crash scenarios [15]. When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection. [16] Metrics 6 and 7 (m6 and m7) characterize the perception system’s ability to make a correct case for lane keep assist by taking into account the false positive and false negative lane detection rate. False positive object detection rate (m8) measures the fraction of total vehicle detections which were classified as non-ego vehicle detections but did not actually exist in ground truth in the test cases.”)
Farabet makes obvious one or more roads in the virtual environment([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 58] “The simulation system 400A may generate a simulated environment 410 that may include AI objects 412 (e.g., AI objects 412A and 412B), HIL objects 414, SIL objects 416, PIL objects 418, and/or other object types. The simulated environment 410 may include features of a driving environment, such as roads, bridges, tunnels, street signs, stop lights, crosswalks, buildings, trees and foliage, the sun, the moon, reflections, shadows, etc., in an effort to simulate a real-world environment accurately within the simulated environment 410. In some examples, the features of the driving environment within the simulated environment 410 may be more true-to-life by including chips, paint, graffiti, wear and tear, damage, etc.” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.” [Par 31] “For example, one or more autonomous vehicle (AV) perception DNNs may be trained and/or tested, where the AV perception DNNs may be used for detecting lanes and boundaries on driving surfaces” [Par 53] “As such, when the simulated environment is used for testing vehicle performance (e.g., for HIL or SIL embodiments), the bot (e.g., the pedestrian) may behave as a real-world pedestrian would (e.g., by jaywalking in rainy or dark conditions, failing to heed stop signs or traffic lights, etc.), in order to more accurately simulate a real-world environment. This method may be used for any AI bot in the simulated environment, such as vehicles, bicyclists, or motorcycles, whose AI bots may also be trained to behave as real-world objects would (e.g., weaving in and out of traffic, swerving, changing lanes with no signal or suddenly, braking unexpectedly, etc.).”)
Applicant argues that no prior art teaches updating, for each pose of the plurality of installation poses, a same set of parameters of the MLM to reduce one or more losses between the respective ground truth data for the pose and one or more outputs computed by the MLM and corresponding to the one or more roads using one or more inputs comprising the one or more sensor data representations of the pose,
Examiner responds by explaining that this is taught by the previously cited references. In particular, VESPA makes obvious ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)([Page 3 Col 1 Par 3 – Col 2 Par 1] “To quantify the performance of a sensor configuration on a vehicle being evaluated over drive cycle test cases (i.e., across various driving scenarios; see Section V), we define eight metrics (m1-m8) that are characteristic of the configuration’s ability to track and detect non-ego vehicles across various road geometries and traffic scenarios. The eight metrics are defined as follows: … The rate of late detection metric (m5) is computed as a fraction of the number of ‘late’ non ego vehicle detections made by the total number of non-ego vehicles, which matters for BW. A detection is classified as late if it is made after the non-ego vehicle crosses the minimum safe longitudinal or lateral distance defined by Intel RSS (Responsibility Sensitive Safety) models on NHTSA for pre-crash scenarios [15]. When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection. [16] Metrics 6 and 7 (m6 and m7) characterize the perception system’s ability to make a correct case for lane keep assist by taking into account the false positive and false negative lane detection rate. False positive object detection rate (m8) measures the fraction of total vehicle detections which were classified as non-ego vehicle detections but did not actually exist in ground truth in the test cases.”)
Onofrio makes obvious updating a same set of parameters of the MLM to reduce one or more losses between the ground truth data and outputs computed by the MLM ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Applicant argues that the use of a Kalman filter in VESPA makes it inapplicable to the claims.
Examiner responds by explaining that, firstly, the characterization of a Kalman filter as a deterministic mathematic algorithm that does not utilize any kind of refinement is incorrect; a Kalman filter is a statistical algorithm that requires tuning to improve its accuracy in a process that is functionally similar to the way training works with a neural network. It is clear from the specification that, given its broadest reasonable interpretation, the breadth of the claimed machine learning model includes Kalman filters. See [Par 41] of the specification which includes “lane detection algorithms, computer vision algorithms, and/or other types of machine learning models” as being considered machine learning models; as the Kalman filter discussed in VESPA is utilized for lane detection, it clearly fits the definition of a machine learning model as defined by the specification. Further, see VESPA at ([Page 3 Col 2 Par 1] “When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection. [16] Metrics 6 and 7 (m6 and m7) characterize the perception system’s ability to make a correct case for lane keep assist by taking into account the false positive and false negative lane detection rate.” [Page 5 Col 2 Par 1] “A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
Further, as noted above, even if a Kalman filter were treated as being “a deterministic mathematical algorithm that is inherently pose-agnostic; it evaluates geometric coordinate data and does not require pose-specific training to function” as argued, the claims do not require that the ground truths for the different sensor poses be different, and further requiring different ground truths for the different sensor poses seems to go against the purpose of the invention.
Additionally, the cited quote found on [Page 4 Col 1] in regards to the size of the design space, i.e. “The combined position and orientation exploration generates an intractably large design space…” does not refer to the use of the Kalman filter with multiple sensors within a single design, rather it refers to the number of possible sensor combinations. The following paragraphs clearly explain that any one design contains only 8 sensors ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.”)
Further, although the Kalman filter is used for sensor fusion, for this fusion to work individual data streams must first be generated from each of the sensors, i.e. sensor data representations from the sensors at each of the plurality of installation poses. The use of Kalman filter to fuse this sensor data to create a pose-agnostic representation of the combined sensor data is not evidence that VESPA is not generating sensor-specific data streams , it is rather evidence of a process similar to the one that is claimed in which pose-specific data from a number of sensors is processed to develop a combined, pose-agnostic model.
Additionally, VESPA is not relied upon to teach the actual training of the machine learning model and is primarily relied upon only for its generation of various testing scenarios and sensor configurations.
Applicant argues that VESPA only teaches the generation of a single pose.
Examiner responds by explaining that while the ultimate goal of VESPA is to find an optimal sensor configuration, it clearly arrives at this goal by considering a variety of different sensor poses and configurations ([Page 4 col 1 Par 1-2] “All of the metrics (m1 – m8) defined in section III.B represent good performance at lower values. We create a cost function that combines these metrics and frame our sensor placement and optimization problem as a minimization problem. The most important metrics are identified and grouped for each feature, as shown in Table 2, and are used to model the cost function as a weighted sum of these five metrics, where the weights are chosen on the basis of their total cardinality across all feature. By searching through the design space of sensor configurations for a minimum cost function value, a sensor configuration can thus be generated where the metrics are cumulatively minimized. … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
These sensor configurations, combined with the machine learning training using sensor data of Onofrio and the virtual sensor simulation of Farabet, teaches the claimed features of selecting a variety of sensor poses, generating virtual sensors at those positions, and training a machine learning model based on the output of those virtual sensors.
Claim Objections
Applicant is advised that should claim 13 be found allowable, claim 18 will be objected to under 37 CFR 1.75 as being a substantial duplicate thereof. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-14, 16-23, 25, 27-29 are rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1)
Claim 1. VESPA teaches([Abstract] “In emerging autonomous vehicles, perception of the environment around the vehicle depends not only on the quality and choice of sensor type, but more importantly also on the instrumented location and orientation of each of the sensors. This article explores the synthesis of heterogeneous sensor configurations towards achieving vehicle autonomy goals. We propose a novel optimization framework called VESPA that explores the design space of sensor placement locations and orientations to find the optimal sensor configuration for a vehicle.”) input data corresponding to sensor data obtained using one or more sensors of the autonomous machine having one or more installation poses of a plurality of installation poses, ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”) ([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.” based at least on the sampling, obtaining one or more sensor data representations of a ([Page 3 Col 1 Par 1] “When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection.” [Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”) generating, for each pose of the plurality of installation poses, respective ground truth data for the one or more sensor data representations of the pose ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”) based at least on mapping one or more road features of one or more roads in the ([Page 3 Col 1 Par 3 – Col 2 Par 1] “To quantify the performance of a sensor configuration on a vehicle being evaluated over drive cycle test cases (i.e., across various driving scenarios; see Section V), we define eight metrics (m1-m8) that are characteristic of the configuration’s ability to track and detect non-ego vehicles across various road geometries and traffic scenarios. The eight metrics are defined as follows: … The rate of late detection metric (m5) is computed as a fraction of the number of ‘late’ non ego vehicle detections made by the total number of non-ego vehicles, which matters for BW. A detection is classified as late if it is made after the non-ego vehicle crosses the minimum safe longitudinal or lateral distance defined by Intel RSS (Responsibility Sensitive Safety) models on NHTSA for pre-crash scenarios [15]. When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection. [16] Metrics 6 and 7 (m6 and m7) characterize the perception system’s ability to make a correct case for lane keep assist by taking into account the false positive and false negative lane detection rate. False positive object detection rate (m8) measures the fraction of total vehicle detections which were classified as non-ego vehicle detections but did not actually exist in ground truth in the test cases.”) ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)one or more outputs ([Page 3 Col 1 Par 3 – Col 2 Par 1] “To quantify the performance of a sensor configuration on a vehicle being evaluated over drive cycle test cases (i.e., across various driving scenarios; see Section V), we define eight metrics (m1-m8) that are characteristic of the configuration’s ability to track and detect non-ego vehicles across various road geometries and traffic scenarios. The eight metrics are defined as follows: … The rate of late detection metric (m5) is computed as a fraction of the number of ‘late’ non ego vehicle detections made by the total number of non-ego vehicles, which matters for BW. A detection is classified as late if it is made after the non-ego vehicle crosses the minimum safe longitudinal or lateral distance defined by Intel RSS (Responsibility Sensitive Safety) models on NHTSA for pre-crash scenarios [15]. When a lane marker is detected but there exists no ground truth lane in simulation it is classified as a false positive lane detection, conversely, if a ground truth lane exists in simulation but is not detected, it is classified as a false negative lane detection. [16] Metrics 6 and 7 (m6 and m7) characterize the perception system’s ability to make a correct case for lane keep assist by taking into account the false positive and false negative lane detection rate. False positive object detection rate (m8) measures the fraction of total vehicle detections which were classified as non-ego vehicle detections but did not actually exist in ground truth in the test cases.”)
VESPA does not explicitly teach At least one processor comprising: one or more circuits to perform one or more autonomous or semi-autonomous control operations based at least on an output computed by a machine learning model (MLM) using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine, the MLM being trained, at least in part, by: determining data from a virtual sensor on a virtual machine; obtaining one or more sensor data representations of a virtual environment from the virtual sensors based at least on instantiating the virtual sensor at the plurality of installation poses in a simulator; generating ground truth data for the one or more sensor data representations based at least on determining one or more road features of one or more roads in the virtual environment; updating a same set of parameters of the MLM to reduce one or more losses between the ground truth data and outputs computed by the MLM
Onofrio makes obvious At least one processor comprising: one or more circuits to perform one or more autonomous or semi-autonomous control operations based at least on an output computed by a machine learning model (MLM) using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine([Par 33] “In addition, the lane graphs 112A-112N generated by the DNNs 104A-104N may be largely independent as a result of the underlying training data used to train the DNNs” [Par 73] “A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from a steering actuator 756. The steering wheel may be optional for full automation (Level 5) functionality.” [Par 27] “ The perception sources may each generate one or more perception outputs that, in some non-limiting embodiments, may aid the vehicle 700 in generating an understanding of a layout or structure of the driving surface—e.g., lane locations, lane dimensions, lane curvature, etc.—and/or to determine a path through the environment along the driving surface.” [Par 24] “The process 100 may include generating and/or receiving sensor data 102 from one or more sensors of the vehicle 700. In deployment, the sensor data 102 may be used by the vehicle 700, and within the process 100, to generate any number of lane graphs” [Par 75] “Controller(s) 736, which may include one or more system on chips (SoCs) 704 (FIG. 7C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle 700. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators 748, to operate the steering system 754 via one or more steering actuators 756, to operate the propulsion system 750 via one or more throttle/accelerators 752. The controller(s) 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle 700. The controller(s) 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functionality (e.g., computer vision), a fourth controller 736 for infotainment functionality, a fifth controller 736 for redundancy in emergency conditions, and/or other controllers. In some examples, a single controller 736 may handle two or more of the above functionalities, two or more controllers 736 may handle a single functionality, and/or any combination thereof.” [Par 54] “For instance, various functions may be carried out by a processor executing instructions stored in memory. The method 400 may also be embodied as computer-usable instructions stored on computer storage media. The method 400 may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.”) the MLM being trained, at least in part, by: determining data from a ([Par 33] “For example, a first output from a first DNN(S) 104 may include a location and classification of lane edges or rails, while a second output from a second DNN(S) 104 may include a recommended trajectory along a driving surface… the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 73] “A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from a steering actuator 756. The steering wheel may be optional for full automation (Level 5) functionality.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats” [Par 30] “The object trace(s) 108 may leverage motion of the vehicle … and/or image data generated by one or more cameras, … and/or other sensor types to track and compute a trajectory of one or more other objects (e.g., vehicles) along the driving surface of the vehicle 700.”) generating ground truth data ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein.”) for the one or more sensor data representations ([Par 33] “the first DNN(S) 104 may be trained using image data having corresponding ground truth annotations generated using LIDAR data, and the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats”) based at least on determining one or more road features of one or more roads in the ([Par 33] “the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats” [Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein.”)) updating a same set of parameters of the MLM to reduce one or more losses between the ground truth data and outputs computed by the MLM ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Onofrio is analogous art because it is within the field of autonomous driving systems. It would have been obvious to combine it with VESPA before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to use the optimal sensor placement determined by VESPA to provide robust autonomous driving. While VESPA addresses the placement of the sensors for optimal coverage, it leaves use of such sensors to perform actual autonomous driving up to future works. As noted by Onofrio, path perception performed using a single data source can result in inaccuracy and dangerous conditions ([Par 4] “Conventional approaches to path perception and/or lane structure computations have relied on a single data source. For example, some conventional approaches rely on the output of a single deep neural network (DNN) that is trained to detect and compute locations of lane lines—which by extension may enable the determination of a drivable path there through. Another example includes the use of a trajectory output signal generated from a high-definition (HD) map. In either example, a single output signal corresponding to a perceived path may be relied upon for controlling an autonomous vehicle. However, obtaining a real-time metric on path perception correctness with a single path perception input signal is not feasible, since measuring and comparing the path perception accuracy of a single signal requires a volume of computing that is impractical to perform in real-time—and thus requires offline comparison to ground truth datasets. As such, because it may not be possible to verify the reliability of the path perception signal, the autonomous vehicle system may rely on inaccurate information when the path perception input fails—thereby leading to disengagement of autonomous driving functionality in some examples. The risk of failure is even further increased in challenging scenarios such as high curvature roads, poor or severe road conditions, or multi-way intersection negotiation. In addition, even where the path perception signal is not entirely inaccurate, the use of a single path perception signal may cause the system to perform poorly on other autonomous driving metrics, such as metrics for passenger comfort and/or smoothness in executing vehicle maneuvers.”) To this end, Onofrio presents a system that uses several data sources resulting in greater accuracy and redundancy ([Par 6] “In contrast to conventional systems, such as those described above, an ensemble of path perception approaches are collectively leveraged to produce a more accurate and reliable understanding of a driving surface and/or a path there through. For example, where a single path perception input may be inaccurate, an analysis of a plurality of path perception inputs provides testability and reliability for accurate and redundant lane mapping and/or path planning in real-time or near real-time. Specifically, through agreement/disagreement analyses of different path perception signal components, reliably metricizing path perception results live in an autonomous or semi-autonomous vehicle becomes possible—while also enabling a higher overall quality of path perception results.”) Overall, one of ordinary skill in the art would have recognized that combining the optimal sensor placement of VESPA with the robust, multi-source path perception of Onofrio would result in an autonomous driving system that is significantly more accurate and therefore safer.
The combination of VESPA and Onofrio does not explicitly teach determining data from a virtual sensor on a virtual machine; obtaining sensor data representations of a virtual environment from the virtual sensors based at least on instantiating the virtual sensor at the plurality of installation poses in a simulator;
Farabet makes obvious determining data from a virtual sensor on a virtual machine; obtaining sensor data representations of a virtual environment from the virtual sensors based at least on instantiating the virtual sensor at the plurality of installation poses in a simulator; ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.” [Par 31] “For example, one or more autonomous vehicle (AV) perception DNNs may be trained and/or tested, where the AV perception DNNs may be used for detecting lanes and boundaries on driving surfaces” [Par 53] “As such, when the simulated environment is used for testing vehicle performance (e.g., for HIL or SIL embodiments), the bot (e.g., the pedestrian) may behave as a real-world pedestrian would (e.g., by jaywalking in rainy or dark conditions, failing to heed stop signs or traffic lights, etc.), in order to more accurately simulate a real-world environment. This method may be used for any AI bot in the simulated environment, such as vehicles, bicyclists, or motorcycles, whose AI bots may also be trained to behave as real-world objects would (e.g., weaving in and out of traffic, swerving, changing lanes with no signal or suddenly, braking unexpectedly, etc.).”)
Farabet is analogous art because it is within the field of autonomous vehicle simulation. It would have been obvious to one of ordinary skill in the art combine it with VESPA and Onofrio before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to provide a simulated testbed on which to evaluate and improve the advanced lane-keeping system introduced by the combination of VESPA and Onofrio. As suggested, by typical methods for training and evaluating autonomous vehicle machine learning systems require lengthy data gathering from physical vehicles. Because of this, many edge cases and dangerous scenarios, particularly important scenarios such as high-speed collisions, are difficult to gather training data for, as it would require actually crashing vehicles on busy highways and streets, something that is exceedingly dangerous to say the least. ([Par 4] “For example, conventional systems often rely on data generated by physical vehicles navigating real-world environments to train the DNNs prior to deployment in working models. This approach has several limitations, however. For example, vehicles can only navigate to so many places, recreating dangerous or unique situations is difficult in the real-world environment, and testing the DNNs in these real-world environments may be dangerous. For example, especially where a DNN is used for obstacle avoidance or other safety measures, testing the DNNs in real-world environments may not be practical. On the other hand, an automaker likely will not deploy an autonomous vehicle into the real-world until an acceptable level of safety has been achieved. As a result, these competing interests make generating a sound, safe, accurate, and reliable autonomous driving system increasingly difficult.”) To this end, Farabet introduces an autonomous vehicle simulation system capable of simulating entire virtual environments in which to test simulated autonomous vehicles, allowing the evaluation of autonomous systems in scenarios that are impractical and/or dangerous to attempt to produce with real-world vehicles. ([Par 5-6] “Embodiments of the present disclosure relate training, testing, and verifying autonomous machines using simulated environments. Systems and methods are disclosed for training, testing, and/or verifying one or more features of a real-world system—such as a software stack for use in autonomous vehicles and/or robots. … In contrast to conventional systems, such as those described above, the systems of the present disclosure leverage a simulated environment to test one or more autonomous driving software stacks that include a multitude of DNNs. For example, physical sensor data, virtual sensor data, or a combination thereof may be used to train the DNNs of the software stack(s). … In any example, the simulated environment may be generated to create difficult to navigate, dangerous, unsafe, and/or otherwise unpredictable situations for the virtual object to navigate. As a result, previously untested scenarios (e.g., due to safety concerns, difficulty of reproduction, etc.) may be tested, repeated, and improved upon within the simulated environment.”) It would have been obvious to one of ordinary skill in the art that combining Farabet with VESPA and Onofrio would produce a system that allows the machine-learning based lane detection/keeping system of the combination of VESPA and Onofrio to be trained on a significantly wider set of data that can be generated at will, including data that would be dangerous to produce, such as camera footage during a high-speed collision. One of ordinary skill in the art would recognize that this wider dataset would allow the trained system to be even more accurate, maintaining tracking and autonomous system composure in the rare, high-stakes situations such as crashes where proper functioning is the most important.
Claim 12. The elements of claim 12 are substantially the same as those of claim 1. Therefore, the elements of claim 12 are rejected due to the same reasons as outlined above for claim 1.
Moreover, the additional elements of claim 12 are made obvious by the combination of VESPA, Onofrio, and Farabet. In particular,
Onofrio teaches A system comprising: one or more processors to execute operations comprising: ([Par 54] “For instance, various functions may be carried out by a processor executing instructions stored in memory. The method 400 may also be embodied as computer-usable instructions stored on computer storage media. The method 400 may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.”)
Claim 23. The elements of claim 23 are substantially the same as those of claim 1. Therefore, the elements of claim 23 are rejected due to the same reasons as outlined above for claim 1.
Claim 2. VESPA teaches wherein the sampling space limits sensor installation angle poses of the plurality of installation poses to a range of angles relative to the virtual machine. ([Page 4 Col 1 Par 2] “. The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid.”)
Claim 13. The elements of claim 13 are substantially the same as those of claim 2. Therefore, the elements of claim 13 are rejected due to the same reasons as outlined above for claim 2.
Claim 18. The elements of claim 18 are substantially the same as those of claim 2. Therefore, the elements of claim 18 are rejected due to the same reasons as outlined above for claim 2.
Claim 3. VESPA teaches ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
Onofrio makes obvious wherein the updating includes updating the same set of parameters until the MLM achieves a threshold level of accuracy with respect to the ground truth data ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Claim 4. VESPA teaches wherein the obtaining includes (([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.” [Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.” [Examiner’s note: each vehicle has a variety of sensors (4 radars and 4 cameras) that generate data i.e. several sensors on the vehicle simultaneous generate output])
Farabet makes obvious simulating, using the simulator, the virtual machine having the virtual sensor instantiated at each of a plurality of positions while driving within the virtual environment ([Par 52] “In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. . In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.). As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment. As another example, for an IMU sensor, the instance may be used for processing IMU data (e.g., representative of orientation) for the object in the simulated environment.”[Par 58] “The simulation system 400A may generate a simulated environment 410 that may include AI objects 412 (e.g., AI objects 412A and 412B), HIL objects 414, SIL objects 416, PIL objects 418, and/or other object types. The simulated environment 410 may include features of a driving environment, such as roads, bridges, tunnels, street signs, stop lights, crosswalks, buildings, trees and foliage, the sun, the moon, reflections, shadows, etc., in an effort to simulate a real-world environment accurately within the simulated environment 410. In some examples, the features of the driving environment within the simulated environment 410 may be more true-to-life by including chips, paint, graffiti, wear and tear, damage, etc.” [Par 71] “In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the HIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment for each sensor (e.g., a first instance from a field of view of a first virtual camera of the virtual vehicle, a second instance from a field of view of a second virtual camera, a third instance from a field of view of a virtual LIDAR sensor, etc.). The instances of the simulated environment may thus be used to generate sensor data for each sensor by the vehicle simulator component(s) 420.” [Par 123] “The method 1000, at block B1020, includes controlling a virtual object within a simulated environment based at least in part on the output. For example, the virtual object (e.g., virtual vehicle) may be controlled within the simulated environment based at least in part on the output. In other examples, the outputs may be used for control. For example, the outputs may be object detection, lane detection, drivable free-space detection, safety procedure determination, etc.”)
Claim 5. VESPA teaches wherein the determining of the sampling space includes defining a range in variance for one or more installation pose properties, the range in variance being relative to the ([Page 4 Col 1 Par 1-2] “Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle…. The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects.)”)
Claim 6. VESPA teaches wherein a first sensor pose of the plurality of installation poses corresponds to a first installation angle of the ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
Farabet makes obvious the virtual sensor on the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 16. The elements of claim 16 are substantially the same as those of claim 6. Therefore, the elements of claim 16 are rejected due to the same reasons as outlined above for claim 6.
Claim 7. VESPA teaches wherein at least some of the plurality of installation poses correspond to different sensor installation locations relative to the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw). ” [Page 5 Col 1 Par 3] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab.”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 8. VESPA teaches wherein at least some of the plurality of installation poses correspond to different sensor installation heights relative to the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw).” [Examiner’s note: sensor configuration is varied across 3 dimensions, i.e. including height])
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 9. Farabet teaches wherein the mapping includes projecting the one or more road features from a three-dimensional (3D) world space of the virtual environment to a two-dimensional (2D) image space of the one or more sensor data representations. ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. The global simulation may be maintained within an engine (e.g., a game engine), or other software-development environment, that may include a rendering engine (e.g., … 3D graphics), a physics engine (e.g., for collision detection, collision response, etc.), sound, scripting, animation, AI, networking, streaming, memory management, threading, localization support, scene graphs, cinematics, and/or other features. … . As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment.” [Par 170] “The neural network may take as its input at least some subset of parameters, such as bounding box dimensions, ground plane estimate obtained (e.g. from another subsystem), inertial measurement unit (IMU) sensor 1166 output that correlates with the vehicle 102 orientation, distance, 3D location estimates of the object obtained from the neural network and/or other sensors (e.g., LIDAR sensor(s) 1164 or RADAR sensor(s) 1160), among others.” [Par 69] “For example, the vehicle simulator component(s) 422 may receive (e.g., retrieve, obtain, etc.), from the global simulation (e.g., represented by the simulated environment 410) hosted by the simulator component(s) 402, data that corresponds to, is associated with, and/or is required by the vehicle simulator component(s) 422 to perform one or more operations by the vehicle simulator component(s) 422 for the PIL object. In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the PIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment corresponding to the field of view of a remote operator of the virtual vehicle controlled by the remote operator, and the portion of the simulated environment may be projected on a display (e.g., a display of a VR headset, a computer or television display, etc.) for assisting the remote operator in controlling the virtual vehicle through the simulated environment 410.” [Par 64] “The simulator component(s) 402 (e.g., simulator host device) may include one or more compute nodes of the simulation system 400A, and may host the simulation of the environment with respect to each actor (e.g., with respect to each HIL, SIL, PIL, and AI actors), as well as hosting the rendering and management of the environment or world state (e.g., the road, signs, trees, foliage, sky, sun, lighting, etc.). In some examples, the simulator component(s) 402 may include a server(s) and associated components (e.g., CPU(s), GPU(s), computers, etc.) that may host a simulator (e.g., NVIDIA's DRIVE™ Constellation AV Simulator).”) [Examiner’s note: generating an image space field of view from the point of view of a virtual camera sensor in a 3D environment is achieved by projecting features from that 3D environment to the 2D space of the virtual camera. Any features within that field of view, be it road features or otherwise, are therefore projected as well.])
Claim 10. Onofrio teaches wherein the one or more road features comprise one or more lane labels determined from a lane graph of a map corresponding to the ([Par 37] “In addition to generating the polylines that correspond to the lane graphs 112, each polyline of each lane graph 112A-112N may have an associated label or line classification assigned thereto. For example, the labels or line classifications may include one or more of: lane keep path (e.g., defined as the center of the lane of travel of the vehicle 700); fork left path, fork right path, merge from left path (e.g., defined as the vehicle 700 entering into a merge from the left); merge right path (e.g., defined as the vehicle 700 entering in a merge from the right); lane change left path; lane change right path; adjacent left path or rail; adjacent left divider or edge; adjacent right path or rail; and/or adjacent right divider or edge. As such, the labels or classifications may include paths or rails for staying in-lane, changing lanes, merging, taking a fork, etc., and/or paths or rails for adjacent lanes (e.g., immediately adjacent lanes to the vehicle 700, lanes that are two or more lanes adjacent the vehicle 700, etc.). The labels or classifications may thus belong to various sets, S: S.sub.1={ego lane, left lane, right lane}; S.sub.2={center lane or rail, divider or edge}; S.sub.3={fork left, fork right}; S.sub.4={lane change left, lane change right}; and so on. In some embodiments, the labels or classifications may include permutations between individual labels belongs to two or more sets, such as sets S.sub.1 and S.sub.2, for example.” [Par 40] “In some non-limiting embodiments, the lane graph aggregator 114 may include a label-based grouper 116, a clusterer 118, a confidence estimator 120, and/or a history tracker 122. The label-based grouper 116 may group the polylines from the various lane graphs 112A-112N according to their labels or line classifications. For example, each polyline from each of the lane graphs 112A-112N that corresponds to a given label (e.g., ego center lane, right adjacent rail, left fork path, etc.) may be retrieved, or selected and grouped together.” [Par 50] “Now with reference to FIG. 3A, FIG. 3A is an example visualization 300 of an aggregate lane representation with high confidence, in accordance with some embodiments of the present disclosure. For example, the visualization 300 may represent predictions corresponding to a structure of and/or paths through lanes 302, 304, and 306. The lane 304 may represent an ego-lane (e.g., a lane of the vehicle 700), the lane 302 may represent a left adjacent lane, and the lane 306 may represent a right adjacent lane. As such, the visualization 300 may represent an iteration of an aggregate lane graph 124 generated using sensor data 102 from the vehicle 700 and/or HD map(s) 106 as the vehicle 700 traverses the environment represented by the visualization 300. The aggregate lane graph 124 of the visualization 300 may include a left adjacent lane left edge 308, a left adjacent lane rail 310, a left adjacent lane right edge 312, an ego-lane left edge 314, an ego-lane rail 316, an ego-lane right edge 318, a right adjacent lane left edge 320, a right adjacent lane rail 322, and a right adjacent lane right edge 324. In addition, the visualization includes bounding shapes 326A-326C around vehicles leading the ego vehicle 700, which may have been used to determine object trace(s) 108 for generating the lane graph(s) 112N. As described above, the visualization 300 may represent an aggregate lane graph having generally high confidences along each of the final polylines corresponding to the various lane or path labels.”)
Farabet makes obvious the virtual environment ([Par 52] “In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. . In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.). As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment. As another example, for an IMU sensor, the instance may be used for processing IMU data (e.g., representative of orientation) for the object in the simulated environment.”[Par 58] “The simulation system 400A may generate a simulated environment 410 that may include AI objects 412 (e.g., AI objects 412A and 412B), HIL objects 414, SIL objects 416, PIL objects 418, and/or other object types. The simulated environment 410 may include features of a driving environment, such as roads, bridges, tunnels, street signs, stop lights, crosswalks, buildings, trees and foliage, the sun, the moon, reflections, shadows, etc., in an effort to simulate a real-world environment accurately within the simulated environment 410. In some examples, the features of the driving environment within the simulated environment 410 may be more true-to-life by including chips, paint, graffiti, wear and tear, damage, etc.” [Par 71] “In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the HIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment for each sensor (e.g., a first instance from a field of view of a first virtual camera of the virtual vehicle, a second instance from a field of view of a second virtual camera, a third instance from a field of view of a virtual LIDAR sensor, etc.). The instances of the simulated environment may thus be used to generate sensor data for each sensor by the vehicle simulator component(s) 420.” [Par 123] “The method 1000, at block B1020, includes controlling a virtual object within a simulated environment based at least in part on the output. For example, the virtual object (e.g., virtual vehicle) may be controlled within the simulated environment based at least in part on the output. In other examples, the outputs may be used for control. For example, the outputs may be object detection, lane detection, drivable free-space detection, safety procedure determination, etc.”)
Claim 11. Onofrio teaches wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; ([Par 22] “Systems and methods are disclosed related to path perception diversity and redundancy in autonomous machine applications.”) a system for performing simulation operations; a system for performing learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. [This claim is written in the alternative format. Unmapped elements are therefore not given patentable weight]
Claim 22. The elements of claim 22 are substantially the same as those of claim 11. Therefore, the elements of claim 22 are rejected due to the same reasons as outlined above for claim 11.
Claim 14. VESPA teaches ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
Farabet makes obvious wherein the operations further comprise instantiating the virtual machine at poses within the virtual environment, wherein the one or more sensor data representations for different sensors are generated using respective instantiations of the virtual machine. ([Par 52] “In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. . In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.). As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment. As another example, for an IMU sensor, the instance may be used for processing IMU data (e.g., representative of orientation) for the object in the simulated environment.” [Par 71] “In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the HIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment for each sensor (e.g., a first instance from a field of view of a first virtual camera of the virtual vehicle, a second instance from a field of view of a second virtual camera, a third instance from a field of view of a virtual LIDAR sensor, etc.). The instances of the simulated environment may thus be used to generate sensor data for each sensor by the vehicle simulator component(s) 420.”)
Claim 17. VESPA teaches wherein the sampling includes generating, using the sampling space, sensor locations of the ([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.”) wherein the one or more sensor data representations for one or more first poses of the plurality of installation poses are generated at a first sensor location and the one or more sensor data representations for one or more second poses of the plurality of installation poses are generated at a second sensor location different from the first sensor location. ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
Farabet makes obvious sensor data representations of the virtual sensors on the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 28. The elements of claim 28 are substantially the same as those of claim 17. Therefore, the elements of claim 28 are rejected due to the same reasons as outlined above for claim 17.
Claim 19. Onofrio teaches wherein the MLM is trained to compute locations of one or more lane boundaries, the one or more lane boundaries including at least one of a left lane boundary, a right lane boundary, or a lane rail. ([Par 33] “In addition, the lane graphs 112A-112N generated by the DNNs 104A-104N may be largely independent as a result of the underlying training data used to train the DNNs” [Par 36] “To generate the lane graphs 112, produced perception output may be converted into a set of polylines—which may be labeled or classified according to their associated lane and/or lane line type, such as left edge, right edge, lane rail, etc.”)
Claim 20. VESPA makes obvious wherein the respective ground truth data for at least one pose of the plurality of installation poses ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
Onofrio teaches wherein the ground truth data ([Par 62] “In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein”) for at least one pose represents a trajectory along a lane, and the MLM is trained to compute locations of one or more trajectory points. ([Par 33] “For example, a first output from a first DNN(S) 104 may include a location and classification of lane edges or rails, while a second output from a second DNN(S) 104 may include a recommended trajectory along a driving surface—which may be independent of any actual lanes on the driving surface.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats—“)
Claim 21. Onofrio teaches wherein, in deployment, the MLM computes at least one of locations of one or more lane boundaries or locations of one or more trajectory points using sensor data obtained using one or more real-world sensors of a real-world vehicle. ([Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats—“ [Par 32] “In such an example, the lane graph 112N-1 may be generated using data from an external offline mapping system while the lane graph 112A may be generated via live perception of the DNN(S) 104A. Moreover, in the example of the lane graph 112N-2, since the sensor data 102 may be combined across multiple sensors (e.g., RADAR, IMU, GNSS, camera, etc.) rather than just a camera,” [Par 81] “One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.”)
Claim 25. VESPA teaches wherein the sampling space corresponds to a plurality of different vehicle ([Par 4 Col 1 Par 2] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro”)
Farabet makes obvious wherein the system considers a plurality of different vehicle makes ([Par 114] “The number of virtual sensors used may be dynamically configurable such that one sensor may be used in a first simulation, five in another, ten in another, etc. In some examples, the dynamic configuration may be determined based on vehicle types (e.g., a first vehicle of year X, make Y, model Z may include 20 sensors, while a second vehicle of year A, make B, model C may include 30 sensors).”)
Claim 27. VESPA teaches wherein the plurality of installation poses correspond to a plurality of installation angles relative to the ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 29. VESPA makes obvious wherein the first sensor location includes a first longitudinal location, a first lateral location, and a first vertical location on the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw). For a given set of N sensors, the number of parameters stored in each chromosomes is thus ‘6N’. Next, in the selection stage, the cost function values are computed for 100 configurations at a time, and a roulette wheel selection method is used to select which set of chromosomes will be involved in the crossover step based on their cost function probability value, computed as a fraction of the cumulative cost function sum of all chromosomes considered in the selection.” [Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
Farabet makes obvious virtual sensors on a virtual vehicle ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
(2) Claims 15 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1) as well as DeMersseman (US 20180172966 A1)
Claim 15. VESPA teaches wherein at least one installation pose of the plurality of installation poses ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”) is behind a windshield and the generating ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.” [Fig. 3] Clearly shows one of the possible sensor placement regions as being behind the windshield. Note that cameras positioned here, cameras being one of the sensor types considered, would capture field of views through the windshield])
PNG
media_image1.png
540
752
media_image1.png
Greyscale
The combination of VESPA, Onofrio, and Farabet does not explicitly teach wherein a sensor is located behind a windshield and the system includes varying one or more physical properties of the windshield
DeMersseman makes obvious wherein a sensor is located behind a windshield and the system includes varying one or more physical properties of the windshield ([Par 35] “The integrated light/rain sensor and communication antenna module 100 is generally mounted on a back (interior) surface of a windshield 92 of the vehicle 90. In various embodiments, the integrated light/rain sensor and communication antenna module 100 is attached to the windshield using a transparent adhesive material (or gasket).” [Par 50] “Referring to FIG. 11, a graph 160 is shown illustrating a relative light efficiency at varying windshield thicknesses of an integrated light/rain sensor and communication antenna module in accordance with an embodiment of the invention. The graph 160 illustrates simulation of a light/rain sensor in accordance with an embodiment of the invention for windshield thickness offsets from 4.2 mm to 6.6 mm. Simulations are shown for a flat windshield and a windshield with a radius of 1400 mm (e.g., convex as seen from outside the vehicle 90). The simulation shows the light/rain sensor in accordance with an embodiment of the invention works well for a range of thicknesses (e.g., 4.8 mm to 6.0 mm) covering a majority of windshields.”)
DeMersseman is analogous art because it is within the field of vehicular sensing. It would have been obvious to one of ordinary skill in the art to combine DeMersseman with VESPA, Onofrio, and Farabet before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to improve the sensing capabilities of the autonomous vehicle in deployment. While the combination of VESPA, Onofrio, and Farabet disclose a variety of sensors that can be included in the vehicle, a notable absence is that of a rain sensor. As noted by Farabet, rain is a condition in which machine-learning based autonomous driving systems are known to become inaccurate ([Par 49] “In some examples, pre-determined conditions or combinations of conditions (such as those described herein) that the DNNs fail to perform accurately enough may also be used to direct and focus data gathering at the edge. For example, the vehicle(s) 102 may not collect all data, but may only collect data where certain conditions or combinations of conditions are met (e.g., at night, in the rain, in certain tunnel types, etc.).”) This can prove extremely dangerous is the system loses significant accuracy but is unable to detect such rainy conditions and take countermeasures, such as self-adjustment or giving control over to a human driver in extreme conditions. (Note that the latter human handover feature is mentioned by Onofrio as a method of recourse if accuracy gets too low ([Par 45] “In an autonomous driving example, the vehicle 700 may disengage autonomous functionality and hand control back to the driver when the confidence along a desired or recommended path of the vehicle 700 is below a threshold.”)) To this end, DeMersseman presents a compact rain sensor system for vehicles ([Par 34] “Embodiments of the present invention include providing a lens for an integrated rain sensor and communication antenna that may (i) provide rain sensing, (ii) provide ambient light sensing, (iii) provide tunnel detection, (iv) provide sunload sensing, (v) take advantage of space behind a rearview mirror; (vi) integrate several features together, (vii) provide a more compact solution when compared with existing standalone solutions, (viii) offer a lower cost solution when compared with standalone solutions, (ix) enable faster assembly, (x) enable physical integration of optical rain sensing with an RF antenna, (xi) facilitate global location determination by one or more satellite constellations such as GPS-Glonass-Beidou-Gallileo, (xii) facilitate connectivity such as vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-everything (V2X) communication, (xiii) be implemented on a single printed circuit board, and/or (xiv) utilize a novel molded lens structure.”) Overall, one of ordinary skill in the would have recognized that combining DeMersseman with VESPA, Onofrio, and Farabet would result in a vehicle system that is more aware of its surroundings and potential hazards in deployment, allowing for adjustments and countermeasures to be taken if conditions become too poor, ultimately resulting in a much safer autonomous driving system.
Claim 26. VESPA teaches wherein at least one installation pose of the plurality of installation poses ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”) is located behind a windshield of the ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.” [Fig. 3] Clearly shows one of the possible sensor placement regions as being behind the windshield. Note that cameras positioned here, cameras being one of the sensor types considered, would capture field of views through the windshield])
PNG
media_image1.png
540
752
media_image1.png
Greyscale
Farabet makes obvious sensors of the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
The combination of VESPA, Onofrio, and Farabet does not explicitly teach one or more physical properties of the windshield are varied for simulation
DeMersseman makes obvious one or more physical properties of the windshield are varied for simulation ([Par 35] “The integrated light/rain sensor and communication antenna module 100 is generally mounted on a back (interior) surface of a windshield 92 of the vehicle 90. In various embodiments, the integrated light/rain sensor and communication antenna module 100 is attached to the windshield using a transparent adhesive material (or gasket).” [Par 50] “Referring to FIG. 11, a graph 160 is shown illustrating a relative light efficiency at varying windshield thicknesses of an integrated light/rain sensor and communication antenna module in accordance with an embodiment of the invention. The graph 160 illustrates simulation of a light/rain sensor in accordance with an embodiment of the invention for windshield thickness offsets from 4.2 mm to 6.6 mm. Simulations are shown for a flat windshield and a windshield with a radius of 1400 mm (e.g., convex as seen from outside the vehicle 90). The simulation shows the light/rain sensor in accordance with an embodiment of the invention works well for a range of thicknesses (e.g., 4.8 mm to 6.0 mm) covering a majority of windshields.”)
DeMersseman is analogous art because it is within the field of vehicular sensing. It would have been obvious to one of ordinary skill in the art to combine DeMersseman with VESPA, Onofrio, and Farabet before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to improve the sensing capabilities of the autonomous vehicle in deployment. While the combination of VESPA, Onofrio, and Farabet disclose a variety of sensors that can be included in the vehicle, a notable absence is that of a rain sensor. As noted by Farabet, rain is a condition in which machine-learning based autonomous driving systems are known to become inaccurate ([Par 49] “In some examples, pre-determined conditions or combinations of conditions (such as those described herein) that the DNNs fail to perform accurately enough may also be used to direct and focus data gathering at the edge. For example, the vehicle(s) 102 may not collect all data, but may only collect data where certain conditions or combinations of conditions are met (e.g., at night, in the rain, in certain tunnel types, etc.).”) This can prove extremely dangerous is the system loses significant accuracy but is unable to detect such rainy conditions and take countermeasures, such as self-adjustment or giving control over to a human driver in extreme conditions. (Note that the latter human handover feature is mentioned by Onofrio as a method of recourse if accuracy gets too low ([Par 45] “In an autonomous driving example, the vehicle 700 may disengage autonomous functionality and hand control back to the driver when the confidence along a desired or recommended path of the vehicle 700 is below a threshold.”)) To this end, DeMersseman presents a compact rain sensor system for vehicles ([Par 34] “Embodiments of the present invention include providing a lens for an integrated rain sensor and communication antenna that may (i) provide rain sensing, (ii) provide ambient light sensing, (iii) provide tunnel detection, (iv) provide sunload sensing, (v) take advantage of space behind a rearview mirror; (vi) integrate several features together, (vii) provide a more compact solution when compared with existing standalone solutions, (viii) offer a lower cost solution when compared with standalone solutions, (ix) enable faster assembly, (x) enable physical integration of optical rain sensing with an RF antenna, (xi) facilitate global location determination by one or more satellite constellations such as GPS-Glonass-Beidou-Gallileo, (xii) facilitate connectivity such as vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-everything (V2X) communication, (xiii) be implemented on a single printed circuit board, and/or (xiv) utilize a novel molded lens structure.”) Overall, one of ordinary skill in the would have recognized that combining DeMersseman with VESPA, Onofrio, and Farabet would result in a vehicle system that is more aware of its surroundings and potential hazards in deployment, allowing for adjustments and countermeasures to be taken if conditions become too poor, ultimately resulting in a much safer autonomous driving system.
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1) as well as Wyrwas (US 20210286925 A1)
Claim 24. VESPA teaches wherein one or more sensor data representations for one or more poses of the plurality of installation poses ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
The combination of VESPA, Onofrio, and Farabet does not explicitly teach wherein sensor data correspond to a plurality of lateral machine poses along a lane and wherein one lateral machine pose of the plurality of lateral machine poses is laterally offset with respect to at least one other lateral machine pose of the plurality of lateral machine poses.
Wyrwas makes obvious wherein sensor data correspond to a plurality of lateral machine poses along a lane, and wherein one lateral machine pose of the plurality of lateral machine poses is laterally offset with respect to at least one other lateral machine pose of the plurality of lateral machine poses. ([Par 76] “The augmentation engine 204 samples the actor states and the ego-vehicle state and generates augmented data. In some implementations, the augmentation engine 204 manipulates the identified actors and actor states (e.g., actor types, and actor motion behavior characteristics, such as the trajectory) to generate variations…. Some examples of manipulations that may be performed by the augmentation engine 202 include changing a speed or acceleration of an actor, changing the actor type or size, changing an offset position (e.g., a lateral or longitudinal offset)” [Par 96] “Other examples of manipulations that may be performed by the augmentation engine 202 include manipulating the path of an actor. For example, a selectable offset 1002, 1052 in position (e.g., a lateral or longitudinal offset) may be introduced in to the trajectory of an actor. … Again, FIGS. 10A and 10B illustrate examples representations 1000, 1050 of mutated simulation data produced by the augmentation engine 202 from the simulation data of FIG. 7. FIG. 10A shows a representation 1000 of mutated simulation data produced from the simulation data of FIG. 7 in which the actor vehicle XX 704 is mutated to have a lateral offset 1002 of the path 706a, 706b to a new path 1004.” [Fig. 7] Shows the original lateral position of the vehicle while [Fig. 10A] shows the laterally offset position of the vehicle)
PNG
media_image2.png
548
406
media_image2.png
Greyscale
PNG
media_image3.png
585
386
media_image3.png
Greyscale
Wyrwas is analogous art because it is within the field of autonomous vehicle simulation. It would have been obvious to one of ordinary skill in the art to combine it with VESPA, Onofrio, and Farabet before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to generate an even greater variance of testing/training scenarios, including scenarios based on real data. Wyrwas notes the disadvantages of both pure-simulation and pure-recorded data approaches, noting how many simulators lack accuracy while using real-world recorded data can make developing a sufficient quantity of data for training difficult. ([Par 2] “A challenge to autonomous vehicle technology arises in acquiring a sufficient quantity and quality of training data to accurately represent a wide variety of driving conditions and scenarios. This training data is used to train the machine learning models used for different systems in the autonomous vehicle, for example, the perception, planning, and control subsystems. One problem is that training of the machine learning models requires a very large amount of data, and just capturing sensor data from operation of autonomous vehicles does not provide enough data. Some approaches have tried to use simulation data for training the machine learning models to address this data quantity issue. For example, some have used simulation data obtained from the execution of simulators that operate similar to video games. However, the problem with that approach is that the data provided by such simulators is not of high enough quality and does not provide an accurate representation of real-world driving conditions.”) To this end, Wyrwas presents a method for augmenting real-word vehicle data to generate highly accurate simulations that are also sufficiently numerous to perform effective training of AV systems ([Par 69] “Generating simulation scenarios based on logged data 214 has an advantage in that the simulation scenarios may be highly realistic because they are based off of logged data 214. Additionally, as described below in more detail, many variations on the simulation scenarios may be generated to increase the variety and quantity of training data. The simulation scenarios generated from logged data 214 may generally be used to simulate an encounter between the autonomous vehicle 100, its surrounding environment, and other entities (i.e., other actors) in the surrounding environment. In some implementations, the logged data 214 may be used to generate variations in simulation scenarios. In some implementations, the variations in the simulation scenarios are generated algorithmically. For example, the process starts with a scenario, changes the types of actors, adding/subtracting them, and using different parameters for models that control the actors. The variations of the simulation scenarios can also be generated algorithmically by varying geometry of the world, traffic lights, paint, geometric models of the models, their light, any other properties of the autonomous vehicle's environment. The simulation scenarios may provide a dataset that includes information to instantiate a three-dimensional world that mimics the motion behavior and sensor configuration of the autonomous vehicle 100, other vehicles (autonomous and/or non-autonomous), and pedestrians, among other things.”) Overall, one of ordinary skill in the art would have recognized that combining Wyrwas with VESPA, Onofrio, and Farabet would produce a system that resulted in more accurate training through the use of higher quality, more numerous training data examples.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached on (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./ Examiner, Art Unit 2187
/EMERSON C PUENTE/ Supervisory Patent Examiner, Art Unit 2187
Prosecution Insights