DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Responsive to the communication dated 12/08/2025
Claims 1-29 are presented for examination
Information Disclosure Statement
The IDS dated 12/21/2021 and 05/18/2022 has been reviewed.
Drawings
The drawings dated 10/08/2021 have been reviewed. They are accepted.
Abstract
The abstract dated 10/08/2021 has been reviewed. It has 140 words, and contains no legal phraseology. It is accepted.
Finality
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Response to Arguments - 35 USC § 112
Applicant’s arguments, see page 1, filed 12/08/2025, with respect to the rejection of claims 1-29 under 112 have been fully considered and are persuasive. The rejection of claims 1-29 under 112 has been withdrawn.
Response to Arguments - 35 USC § 101
Applicant's arguments filed 12/08/2025 have been fully considered but they are not persuasive.
Applicant argues that the disclosure describes a method to avoid using viewpoint transformations provides an improvement that integrates the claims into a practical application/provides significantly more.
Examiner responds by explaining that this avoidance of viewpoint transformations is merely the result of the abstract idea itself. Particularly, such generation of training data that avoids viewpoint transformation is merely the result of generating the virtual sensor output, which is a mental process, in a certain way. For example, generating “virtual” sensor outputs of a “virtual” vehicle in a “virtual” environment in such a manner is a mental process equivalent to drawing, with a pencil and paper, a depiction of the vehicle in an environment, and a depiction of what the output of the sensor should be. For example, if the vehicle is drawn facing outwards in a driveway, the depiction of the output of the backup camera could include the house and garage. This could be repeated over and over again to depict different sensor positions and therefore slightly different camera perspectives. Avoiding the transformation to artificially warp the image in this case merely means generated images, by drowning them, in a way that is already warped in the desired way.
With this in mind, it is clear that this alleged improvement is solely the result of the abstract idea itself; as such, this alleged improvement is not sufficient to integrate the claims into a practical application nor provide significantly more.
(MPEP 2106.05(a)(I): An inventive concept "cannot be furnished by the unpatentable law of nature (or natural phenomenon or abstract idea) itself." Genetic Techs. Ltd. v. Merial LLC, 818 F.3d 1369, 1376, 118 USPQ2d 1541, 1546 (Fed. Cir. 2016))
Response to Arguments - 35 USC § 103
Applicant's arguments filed 12/08/2025 have been fully considered but they are not persuasive.
Applicant argues that no prior art teaches determining sample space poses for a virtual sensor or updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
Examiner responds by explaining that the combination of Vespa, Onofrio, and Farabet teaches these limitations. Particularly:
Vespa teaches determining a sampling space of sensor installation poses for a the ;([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.”)
generating ground truth data corresponding to the one or more sensor outputs from each of the plurality of installation poses ([Page 5 Col 2 Par 1] “Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
Further, passages like the above cited Col 1 Par 3 – Col 2 Par 1 of VESPA clearly the describe the generation of sensor output before comparing that output with ground truth data.
Onofrio teaches the explicit use of ground truth and MLM output data to determine losses and update MLM parameters
Onofrio makes obvious ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Finally, Farabet teaches the explicit use of virtual sensors on virtual machines within a virtual environment, that environment including lanes.
Farabet makes obvious a virtual sensor on a virtual machine; a virtual sensor with a virtual environment; ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
lanes corresponding to the virtual environment; ([Par 31] “For example, one or more autonomous vehicle (AV) perception DNNs may be trained and/or tested, where the AV perception DNNs may be used for detecting lanes and boundaries on driving surfaces” [Par 53] “This method may be used for any AI bot in the simulated environment, such as vehicles, bicyclists, or motorcycles, whose AI bots may also be trained to behave as real-world objects would (e.g., weaving in and out of traffic, swerving, changing lanes with no signal or suddenly, braking unexpectedly, etc.).”)
Applicant argues that Vespa only teaches considering a single configuration.
Examiner responds by explaining that although a final best configuration is the goal of the project of VESPA, it clearly arrives at this goal by considering a variety of different sensor poses and configurations ([Page 4 col 1 Par 1-2] “All of the metrics (m1 – m8) defined in section III.B represent good performance at lower values. We create a cost function that combines these metrics and frame our sensor placement and optimization problem as a minimization problem. The most important metrics are identified and grouped for each feature, as shown in Table 2, and are used to model the cost function as a weighted sum of these five metrics, where the weights are chosen on the basis of their total cardinality across all feature. By searching through the design space of sensor configurations for a minimum cost function value, a sensor configuration can thus be generated where the metrics are cumulatively minimized. … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
These sensor configurations, combined with the machine learning training using sensor data of Onofrio and the virtual sensor simulation of Farabet, teaches the claimed features of selecting a variety of sensor poses, generating virtual sensors at those positions, and training a machine learning model based on the output of those virtual sensors.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-29 are rejected under 35 U.S.C. 101 because they are directed to an abstract idea without significantly more.
Claim 1 (Statutory Category – Machine)
Step 2A – Prong 1: Judicial Exception Recited?
Yes, the claim recites a mental process, specifically:
MPEP 2106.04(a)(2)(Ill): “Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental processes include observations, evaluations, Judgments, and opinions.”
Further, the MPEP recites “The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation.”
one or more circuits to perform one or more autonomous or semi-autonomous control operations corresponding to an autonomous machine based at least on an output computed by one or more machine learning models (MLMs) using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine, the one or more MLMs being trained, at least in part, by …
Performing “control operations” corresponding to a vehicle is a mental process equivalent to driving said vehicle such as a car, a process done by millions of people every day. Doing so using sensor data from sensors of the vehicle is an extension of this process equivalent to the use of driver assistance features, such as the use of backup cameras or blind spot warning indicators.
Doing so using “one or more circuits” and “one or more machine learning models (MLMs)” amounts to no more than mere instructions to apply the judicial exception using a general-purpose computer.
Finally, obtaining the sensor data is also an example of mere data gathering.
Should it be found that performing the control operations themselves is not a mental process, it is also an example of insignificant post-solution activity.
determining a sampling space of sensor installation poses for a virtual sensor on a virtual machine; sampling a plurality of installation poses from the sampling space;
Determining this sampling space is a mental process equivalent to determining the set of possible positions that a sensor can be placed. For example, a person could observe a vehicle and a sensor and judge where the sensor could fit. This is functionally equivalent to the mental process performed when deciding to put a sticker or magnet on a vehicle.
Sampling from this sampling space merely consists of making a judgment on how the larger set should be narrowed and choosing preferred positions from the larger set. Returning to the sticker/magnet example, this is equivalent to deciding that a sticker should go in a position on the rear bumper rather than considering every available position on the entire vehicle.
Doing this on a computerized “virtual” machine with a “virtual” sensor amounts to no more than mere instructions to apply the judicial exception using a general-purpose computer.
based at least on the sampling, generating one or more sensor outputs from the virtual sensor at each of the plurality of installation poses within a virtual environment; generating ground truth data corresponding to the one or more sensor outputs from each of the plurality of installation poses based at least on determining lane labels from a lane graph corresponding to the virtual environment; and
Generating “virtual” sensor outputs of a “virtual” vehicle in a “virtual” environment in such a manner is a mental process equivalent to drawing, with a pencil and paper, a depiction of the vehicle in an environment, and a depiction of what the output of the sensor should be. For example, if the vehicle is drawn facing outwards in a driveway, the depiction of the output of the backup camera could include the house and garage. This could be repeated over and over again to depict different sensor positions and therefore slightly different camera perspectives.
Generating the ground truth data from these outputs is equivalent to labeling the objects and features detected by the sensors. For example, if there is a dog in view of the sensor, a person could label it as “dog.”
Determining a label for a lane is equivalent to observing the road and identifying qualities about the lane, something commonly performed by people when driving. For example, in the US when someone sees that the lane lines are dashed, they recognize them as denoting that this is a passing zone.
Further, should it be found that generating the ground truth data is not a mental process, it is also an example of mere data gathering, as analyzed below.
Doing this with a computerized system with a “virtual” sensor on a “virtual” machine within a “virtual” environment amounts to no more than mere instructions to apply the judicial exception using a general-purpose computer.
The claims also recite a mathematic concept, particularly:
updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
It is clear that this limitation is merely a textual placeholder for a mathematic calculation, particularly the evaluation of a loss function to determine how the parameters should be updated. See the specification ([Par 56] “The ground truth data 122 and the outputs 128 may be used by the training engine 124 to determine the accuracy of the outputs 128, and to update parameters - e.g., weights and biases - of the machine learning model using one or more loss functions.”)
With this in mind, the use of a mathematic function such as a loss function is merely the use of a mathematic concept.
Should it be found that this is not a mathematic process, it is also an example of mere data gathering and mere instructions to apply.
Step 2A – Prong 2: Integrated into a Practical Solution?
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and
post solution activity to be insignificant extra-solution activity.
Data gathering:
generating one or more sensor outputs from the virtual sensor at each of the plurality of installation poses within a virtual environment;
When recited at such a high level of generality, without an explanation of how the simulation works to model the virtual environment, machine and sensor or how the outputs from the virtual sensors are actually generated, obtaining these virtual sensor outputs amounts to no more than merely gathering data representative of those outputs.
using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine… generating ground truth data corresponding to the one or more sensor outputs from each of the plurality of installation poses based at least on determining lane labels from a lane graph corresponding to the virtual environment; and
Generating sensor data is equivalent to gathering data from those sensors, and is thus no more than an example of mere data gathering
updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
Updating the parameters, when recited at such a high level of generality without explaining how the new values are obtained other than to “reduce one or more losses,” merely amounts to gathering data representative of those new parameter values and subsequently an updated version of the MLM.
Should it be found that this is not an example of mere data gathering, it is also an example of mere instructions to apply.
Post-solution activity:
perform one or more autonomous or semi-autonomous control operations corresponding to an autonomous machine based at least on an output computed by one or more machine learning models
Performing such control operations is merely equivalent to acting on the results of the abstract idea, functionally equivalent to a step of cutting hair with a tool after a mental process, to which the claims are directed, of designing a hairstyle. Therefore, this element amounts to no more than insignificant post-solution activity.
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
an output computed by one or more machine learning models (MLMs)… the one or more MLMs being trained, at least in part, by: … updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
Applying a computer to perform generic machine learning operations at a high level of generality is simply the act of instructing a computer to perform generic functions to perform the operations of that machine learning model, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the machine learning model generates an “output,” “control operations” are performed, and the parameters of the model are “updated” without reciting how this is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations; for additional clarity see ([Par 41] “For example, and without limitation, the machine learning model 126 may include any type of machine learning model, such as a machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, long/short term memory/LSTM, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), lane detection algorithms, computer vision algorithms, and/or other types of machine learning models.”)
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Step 2B: Claim provides an Inventive Concept?
No, as discussed with respect to Step 2A, the additional limitations are mere data gathering or post solution activity (Insignificant Extra-Solution Activity), Well-Understood, Routine, Conventional Activity, or a general purpose computer and do not impose any meaningful limits on practicing the abstract idea and therefore the claim does not provide an inventive concept in Step 2B.
Insignificant Extra-Solution Activity (MPEP 2106.05(g)) has found mere data gathering and
post solution activity to be insignificant extra-solution activity.
Data gathering:
generating one or more sensor outputs from the virtual sensor at each of the plurality of installation poses within a virtual environment;
When recited at such a high level of generality, without an explanation of how the simulation works to model the virtual environment, machine and sensor or how the outputs from the virtual sensors are actually generated, obtaining these virtual sensor outputs amounts to no more than merely gathering data representative of those outputs.
The courts have found that claim elements equivalent to merely gathering data are not indicative of integration into a practical application nor evidence of an Inventive concept (MPEP 2106.05(g)(Mere Data Gathering)(i) Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989);)
using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine… generating ground truth data corresponding to the one or more sensor outputs from each of the plurality of installation poses based at least on determining lane labels from a lane graph corresponding to the virtual environment; and
Generating sensor data is equivalent to gathering data from those sensors, and is thus no more than an example of mere data gathering.
A claim element that amounts to merely gathering data is not indicative of integration into a
practical solution nor evidence that the claim provides an inventive concept or significantly more, as exemplified by ((MPEP 2106.05)(g)(Mere Data Gathering) i. Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011);
updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
Updating the parameters, when recited at such a high level of generality without explaining how the new values are obtained other than to “reduce one or more losses,” merely amounts to gathering data representative of those new parameter values and subsequently an updated version of the MLM.
A claim element that amounts to merely gathering data is not indicative of integration into a
practical solution nor evidence that the claim provides an inventive concept or significantly more, as exemplified by ((MPEP 2106.05)(g)(Mere Data Gathering) i. Performing clinical tests on individuals to obtain input for an equation, In re Grams, 888 F.2d 835, 839-40; 12 USPQ2d 1824, 1827-28 (Fed. Cir. 1989); iv. Obtaining information about transactions using the Internet to verify credit card transactions, CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011);
Should it be found that this is not an example of mere data gathering, it is also an example of mere instructions to apply.
Post-solution activity:
perform one or more autonomous or semi-autonomous control operations corresponding to an autonomous machine based at least on an output computed by one or more machine learning models
Performing such control operations is merely equivalent to acting on the results of the abstract idea, functionally equivalent to a step of cutting hair with a tool after a mental process, to which the claims are directed, of designing a hairstyle. Therefore, this element amounts to no more than insignificant post-solution activity.
This element merely acts on the results of the previous abstract steps. A claim element that merely acts on a series of previous abstract steps is not indicative of integration into a practical solution nor evidence that the claim provides an inventive concept, as exemplified by ((MPEP 2106.05)(g)(Insignificant application) i. Cutting hair after first determining the hair style, In re Brown, 645 Fed. App'x 1014, 1016-1017 (Fed. Cir. 2016) and ii. Printing or downloading generated menus, Ameranth, 842 F.3d at 1241-42, 120 USPQ2d at 1854-55.)
Mere Instructions to Apply (MPEP 2106.05(f)) has found that merely applying a judicial exception such as an abstract idea, as by performing it on a computer, does not integrate the claim into a practical solution.
Mere Instructions to Apply:
an output computed by one or more machine learning models (MLMs)… the one or more MLMs being trained, at least in part, by: … updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels within field of views captured by the one or more sensor outputs from each of the plurality of installation poses.
Applying a computer to perform generic machine learning operations at a high level of generality is simply the act of instructing a computer to perform generic functions to perform the operations of that machine learning model, which is merely an instruction to apply a computer to the judicial exception. The claim only recites the idea of a solution or outcome, i.e. that the machine learning model generates an “output,” “control operations” are performed, and the parameters of the model are “updated” without reciting how this is actually accomplished. Further, the computer elements claimed are cited as merely generic tools to perform the operations; for additional clarity see ([Par 41] “For example, and without limitation, the machine learning model 126 may include any type of machine learning model, such as a machine learning model(s) using linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbor (Knn), K means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., auto-encoders, convolutional, recurrent, perceptrons, long/short term memory/LSTM, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), lane detection algorithms, computer vision algorithms, and/or other types of machine learning models.”)
The courts have found that such mere instructions to apply are not indicative of integration into a practical application nor recitation of significantly more than the judicial exception (MPEP 2106.05(f) “Another consideration when determining whether a claim integrates a judicial exception into a practical application in Step 2A Prong Two or recites significantly more than a judicial exception in Step 2B is whether the additional elements amount to more than a recitation of the words "apply it" (or an equivalent) or are more than mere instructions to implement an abstract idea or other exception on a computer. As explained by the Supreme Court, in order to make a claim directed to a judicial exception patent-eligible, the additional element or combination of elements must do "‘more than simply stat[e] the [judicial exception] while adding the words ‘apply it’". Alice Corp. v. CLS Bank, 573 U.S. 208, 221, 110 USPQ2d 1976, 1982-83 (2014) (quoting Mayo Collaborative Servs. V. Prometheus Labs., Inc., 566 U.S. 66, 72, 101 USPQ2d 1961, 1965). Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible. Alice Corp., 573 U.S. at 223, 110 USPQ2d at 1983”)
Moreover, Mere Instructions To Apply An Exception (MPEP 2106.05(f)) has found that simply adding a general purpose computer or computer components after the fact to an abstract idea (e.g., a fundamental economic practice or mathematical equation) does not integrate a judicial exception into a practical application or provide significantly more. In light of this, the additional generic computer component elements of “At least one processor comprising: one or more circuits to perform … one or more machine learning models (MLMs), a virtual sensor on a virtual machine, the virtual sensor … within a virtual environment; ” are not sufficient to integrate a judicial exception into a practical application nor provide evidence of an inventive concept.
The additional elements have been considered both individually and as an ordered combination in the consideration of whether they constitute significantly more, and have been determined not to constitute such.
The claim is ineligible.
Claim 12 The elements of claim 12 are substantially the same as those of claim 1. Therefore, the elements of claim 12 are rejected due to the same reasons as outlined above for claim 1.
Claim 23 The elements of claim 23 are substantially the same as those of claim 1. Therefore, the elements of claim 23 are rejected due to the same reasons as outlined above for claim 1.
Claim 2 recites “wherein the sampling space limits sensor installation angle poses of the installation poses to a range of angles relative to the virtual machine.”
This merely clarifies the data contained in the sampling space and what sensor poses are represented therein and is therefore merely an extension of the mental process and mere instructions to apply.
Claim 18 The elements of claim 18 are substantially the same as those of claim 2. Therefore, the elements of claim 18 are rejected due to the same reasons as outlined above for claim 2.
Claim 3 recites “wherein updating includes updating the one or more parameters until the outputs computed by the one or more MLMs achieve a threshold level of accuracy with respect to the ground truth data.”
This merely clarifies how the updating is repeated, and is therefore merely an extension of the mental process, mathematic concept, mere data gathering, and mere instructions to apply.
Additionally, determining whether the accuracy is above or below a threshold is a mental process equivalent to comparing the result of the loss function used during the updating step to arbitrary threshold for that accuracy by determining which number is larger, the loss function output or the threshold value.
Claim 4 recites “wherein the one or more MLMs are further trained at least in part by: instantiating the virtual sensor at each of the plurality of installation poses on the virtual machine within the virtual environment; and simulating the virtual machine, having the virtual sensor instantiated at each of the plurality of installation poses, driving within the virtual environment to simultaneously generate the one or more sensor outputs.”
Specifying that the sensor outputs are instantiated in such a way merely clarifies additional information about how the outputs are generated, and therefore is merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Further, simulating the output of the vehicles driving through the environment is a mental process equivalent to drawing, with a pencil and paper, the output of what such a plurality of sensors would be. For example, a person could draw the output of a forward-facing camera and a rear camera by drawing an imagined representation of the road in front of and behind the car while it drives.
Doing this with a computerized system with a “virtual” sensor on a “virtual” machine within a “virtual” environment amounts to no more than mere instructions to apply the judicial exception using a general-purpose computer.
Claim 5 recites “, wherein the determining of the sampling space includes defining a range in variance for one or more installation pose properties, the range in variance being relative to the virtual machine.”
Determining a range of variance is a mental process that involves deciding a range within which data is acceptable. For example, a person could come up with an arbitrary range for sensor placement, for example that sensors must be at least 2 inches from each other, or that they cannot be installed at an angle above 45 degrees relative to the surface of the outside of the vehicle.
Claim 6 recites “wherein a first sensor pose of the plurality of installation poses corresponds to a first installation angle of the virtual sensor on the virtual machine and a second sensor pose of the plurality of installation poses corresponds to a second installation angle of the virtual sensor on the virtual machine.”
This merely clarifies the position of the sensors relative to the vehicle they are mounted on, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 16 The elements of claim 16 are substantially the same as those of claim 6. Therefore, the elements of claim 16 are rejected due to the same reasons as outlined above for claim 6.
Claim 7 recites “wherein at least some of the sensor installation poses correspond to different sensor installation locations relative to the virtual machine.”
This merely clarifies the position of the sensors relative to the vehicle they are mounted on, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 8 recites “wherein at least some of the sensor installation poses correspond to different sensor installation heights relative to the virtual machine.”
This merely clarifies the position of the sensors relative to the vehicle they are mounted on, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 9 recites “wherein the output corresponds to at least one of a permanent lane boundary, a temporary lane boundary, or a physical lane boundary.”
This merely clarifies the output of the machine learning model and is therefore merely an extension of the mental process and mere instructions to apply.
Claim 10 recites “wherein the one or more sensor outputs are further generated based at least on varying locations of one or more real-world vehicles using real-world sensor data that captures the locations of the one or more real- world vehicles”
Obtaining sensor information from sensors mounted on a physical vehicle amounts to no more than mere data gathering. Deciding to place the sensors in varying locations on the vehicle is a mental process equivalent to observing the vehicle and choosing arbitrary locations for sensor installation.
Claim 11 recites “wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs);a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.”
This element merely clarifies the form of machine and processor, and is therefore merely an extension of the mere instructions to apply.
Claim 22 The elements of claim 22 are substantially the same as those of claim 11. Therefore, the elements of claim 22 are rejected due to the same reasons as outlined above for claim 11.
Claim 13 recites “wherein the plurality of sensor outputs correspond to a plurality of lateral machine poses within or outside of the lane, and wherein at least one lateral machine pose of the plurality of lateral machine poses is laterally offset with respect to at least one other machine pose of the plurality of lateral machine poses.”
This merely clarifies the nature of the sensor outputs and their physical relationship to each other, and is therefore merely an extension of the mental process and data gathering steps.
Claim 24 The elements of claim 24 are substantially the same as those of claim 13. Therefore, the elements of claim 24 are rejected due to the same reasons as outlined above for claim 13.
Claim 14 recites “wherein the operations further comprise instantiating the virtual machine at poses within the virtual environment, wherein each sensor output of the plurality of sensor outputs is generated using a respective instantiation of the virtual machine.”
This element merely clarifies that the data and sensor outputs are first split into individual data elements that are processed independently before being used to train the model, and is thereby merely an extension of the mental process and mere instructions to apply an exception.
Claim 15 recites “wherein at least one installation pose of the plurality of installation poses is behind a windshield and the generating the plurality of sensor outputs includes varying one or more physical properties of the windshield to capture a plurality of the field of views through the windshield.”
Firstly, specifying that one of the sensor installation poses is located behind a windshield merely clarifies the location if the sensors, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Varying the physical properties of the windshield to produce different sensor outputs is a mental process equivalent to drawing representations of that sensor output with a pen and paper. For example, a person could draw a first image depicting the view of a dashboard-mounted camera out through the windshield with low tint, and a second, much darker depiction of the camera view out of a highly tinted windshield.
Claim 26 The elements of claim 26 are substantially the same as those of claim 15. Therefore, the elements of claim 26 are rejected due to the same reasons as outlined above for claim 15.
Claim 17 recites “wherein the sampling includes generating, using the sampling space, sensor locations of the virtual sensor relative to the virtual machine, wherein at least a first sensor output of the plurality of sensor outputs is generated at a first sensor location and at least a second sensor output of the plurality of sensor outputs is generated at a second sensor location different from the first sensor location”
Generating sensor locations is a mental process equivalent to observing a vehicle and making arbitrary judgements about where a series of sensors should be installed.
Generating “virtual” sensor outputs of a “virtual” vehicle in a “virtual” environment in such a manner is a mental process equivalent to drawing, with a pencil and paper, a depiction of the vehicle in an environment, and a depiction of what the output of the sensor should be. For example, if the vehicle is drawn facing outwards in a driveway, the depiction of the output of the backup camera could include the house and garage. This could be repeated over and over again to depict different sensor positions and therefore slightly different camera perspectives.
Claim 19 recites “wherein the one or more MLMs are trained to compute locations of one or more lane boundaries, the one or more lane boundaries including at least one of a left lane boundary, a right lane boundary, or a lane rail.”
Differentiating a left lane boundary from a right lane boundary, or vice versa, is a mental process equivalent to observing the road ahead and determining which side each lane marking is on, a process performed mentally by hundreds of millions of people each day when driving a vehicle.
Claim 20 recites “wherein the ground truth data further represents a trajectory along the lane, and the one or more MLMs are trained to compute locations of one or more trajectory points.”
This merely clarifies what data is contained in the ground truth data, and is therefore merely an extension of the data gathering step.
Moreover, determining a trajectory of a vehicle is a mental process equivalent to observing where the car is and its current heading to determine where it is going, a process mentally performed every day by people driving vehicles.
Claim 21 recites “wherein, in deployment, the one or more MLMs compute at least one of locations of one or more lane boundaries or locations of one or more trajectory points using sensor data obtained using one or more real-world sensors of a real-world vehicle.”
Determining the location of a lane boundary is a mental process that involves observing the road to find the lane lines, as is typically done when driving a car to maintain your position between lane lines. Similarly, determining the trajectory of the vehicle through the environment merely consists of looking ahead to see where the vehicle will go next on its current path. This is also frequently done by people when driving a vehicle.
Additionally, obtaining the sensor data is such a generic manner amounts to no more than mere data gathering.
Claim 25 recites “wherein the sampling space corresponds to a plurality of different vehicle makes and models.”
This merely clarifies additional details of about the sampling space, and therefore is merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 27 recites “wherein the installation poses correspond to a plurality of installation angles relative to the virtual machine.”
This merely clarifies the rotation of the sensors relative to the vehicle they are mounted on, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim 28 recites “wherein the sampling includes generating, using the sampling space, sensor locations of the virtual sensor on the virtual machine, wherein at least a first output of the plurality of sensor outputs is generated at a first sensor location and at least a second output of the plurality of sensor outputs is generated at a second sensor location different from the first sensor location.”
This merely clarifies that sensor data is obtained at multiple different locations relative to the machine the sensor is mounted on, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Generating “virtual” sensor outputs of a “virtual” vehicle in a “virtual” environment in such a manner is a mental process equivalent to drawing, with a pencil and paper, a depiction of the vehicle in an environment, and a depiction of what the output of the sensor should be. For example, if the vehicle is drawn facing outwards in a driveway, the depiction of the output of the backup camera could include the house and garage. This could be repeated over and over again to depict different sensor positions and therefore slightly different camera perspectives.
Claim 29 recites “wherein the first sensor location includes a first longitudinal location, a first lateral location, and a first vertical location on the virtual machine and the second sensor location includes a second longitudinal location, a second lateral location, and second vertical location on the virtual machine, further wherein at least one of the first longitudinal location is different from the second longitudinal location, the first lateral location is different from the second lateral location, or the first vertical location is different from the second vertical location.”
This merely clarifies that the first and second sensors are mounted on the machine at different xyz locations, and is therefore merely an extension of the mental process, mere data gathering, and mere instructions to apply.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-12, 14, 16-23, 25, 27-29 are rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1)
Claim 1. VESPA teaches ([Abstract] “In emerging autonomous vehicles, perception of the environment around the vehicle depends not only on the quality and choice of sensor type, but more importantly also on the instrumented location and orientation of each of the sensors. This article explores the synthesis of heterogeneous sensor configurations towards achieving vehicle autonomy goals. We propose a novel optimization framework called VESPA that explores the design space of sensor placement locations and orientations to find the optimal sensor configuration for a vehicle.”) ([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.”)
based at least on the sampling, generating, one or more sensor outputs from the ([Page 5 Col 2 Par 1] “Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”)
generating ground truth data corresponding to the one or more sensor outputs from each of the plurality of installation poses ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.”) ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
VESPA does not explicitly teach At least one processor comprising: one or more circuits to perform one or more autonomous or semi-autonomous control operations based at least on an output computed by one or more machine learning models (MLMs) using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine, the one or more MLMs being trained, at least in part, by: determining data from a virtual sensor on a virtual machine; generating one or more sensor outputs from the virtual sensor within a virtual environment; generating ground truth data corresponding to the one or more sensor outputs based at least on determining lane labels from a lane graph corresponding to the virtual environment; and updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels
Onofrio makes obvious At least one processor comprising: one or more circuits to perform one or more autonomous or semi-autonomous control operations based at least on an output computed by one or more machine learning models (MLMs) using input data corresponding to sensor data obtained using one or more sensors of the autonomous machine, ([Par 33] “In addition, the lane graphs 112A-112N generated by the DNNs 104A-104N may be largely independent as a result of the underlying training data used to train the DNNs” [Par 73] “A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from a steering actuator 756. The steering wheel may be optional for full automation (Level 5) functionality.” [Par 27] “ The perception sources may each generate one or more perception outputs that, in some non-limiting embodiments, may aid the vehicle 700 in generating an understanding of a layout or structure of the driving surface—e.g., lane locations, lane dimensions, lane curvature, etc.—and/or to determine a path through the environment along the driving surface.” [Par 24] “The process 100 may include generating and/or receiving sensor data 102 from one or more sensors of the vehicle 700. In deployment, the sensor data 102 may be used by the vehicle 700, and within the process 100, to generate any number of lane graphs” [Par 75] “Controller(s) 736, which may include one or more system on chips (SoCs) 704 (FIG. 7C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle 700. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators 748, to operate the steering system 754 via one or more steering actuators 756, to operate the propulsion system 750 via one or more throttle/accelerators 752. The controller(s) 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle 700. The controller(s) 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functionality (e.g., computer vision), a fourth controller 736 for infotainment functionality, a fifth controller 736 for redundancy in emergency conditions, and/or other controllers. In some examples, a single controller 736 may handle two or more of the above functionalities, two or more controllers 736 may handle a single functionality, and/or any combination thereof.” [Par 54] “For instance, various functions may be carried out by a processor executing instructions stored in memory. The method 400 may also be embodied as computer-usable instructions stored on computer storage media. The method 400 may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.”) the one or more MLMs being trained, at least in part, by: determining data from a ([Par 33] “For example, a first output from a first DNN(S) 104 may include a location and classification of lane edges or rails, while a second output from a second DNN(S) 104 may include a recommended trajectory along a driving surface… the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 73] “A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (e.g., along a desired path or route) when the propulsion system 750 is operating (e.g., when the vehicle is in motion). The steering system 754 may receive signals from a steering actuator 756. The steering wheel may be optional for full automation (Level 5) functionality.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats” [Par 30] “The object trace(s) 108 may leverage motion of the vehicle … and/or image data generated by one or more cameras, … and/or other sensor types to track and compute a trajectory of one or more other objects (e.g., vehicles) along the driving surface of the vehicle 700.”) generating ground truth data ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein.”) corresponding to the one or more sensor outputs ([Par 33] “the first DNN(S) 104 may be trained using image data having corresponding ground truth annotations generated using LIDAR data, and the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats”) based at least on determining lane labels from a lane graph corresponding to the ([Par 33] “the second DNN(S) 104 may be trained using video data of trajectories driven by human drivers through varying environments and along varying driving surfaces.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats” [Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein.”)) and updating one or more parameters of the one or more MLMs to reduce one or more losses between the ground truth data and outputs computed by the one or more MLMs using inputs respectively depicting one or more lanes corresponding to the lane labels([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Onofrio is analogous art because it is within the field of autonomous driving systems. It would have been obvious to combine it with VESPA before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to use the optimal sensor placement determined by VESPA to provide robust autonomous driving. While VESPA addresses the placement of the sensors for optimal coverage, it leaves use of such sensors to perform actual autonomous driving up to future works. As noted by Onofrio, path perception performed using a single data source can result in inaccuracy and dangerous conditions ([Par 4] “Conventional approaches to path perception and/or lane structure computations have relied on a single data source. For example, some conventional approaches rely on the output of a single deep neural network (DNN) that is trained to detect and compute locations of lane lines—which by extension may enable the determination of a drivable path there through. Another example includes the use of a trajectory output signal generated from a high-definition (HD) map. In either example, a single output signal corresponding to a perceived path may be relied upon for controlling an autonomous vehicle. However, obtaining a real-time metric on path perception correctness with a single path perception input signal is not feasible, since measuring and comparing the path perception accuracy of a single signal requires a volume of computing that is impractical to perform in real-time—and thus requires offline comparison to ground truth datasets. As such, because it may not be possible to verify the reliability of the path perception signal, the autonomous vehicle system may rely on inaccurate information when the path perception input fails—thereby leading to disengagement of autonomous driving functionality in some examples. The risk of failure is even further increased in challenging scenarios such as high curvature roads, poor or severe road conditions, or multi-way intersection negotiation. In addition, even where the path perception signal is not entirely inaccurate, the use of a single path perception signal may cause the system to perform poorly on other autonomous driving metrics, such as metrics for passenger comfort and/or smoothness in executing vehicle maneuvers.”) To this end, Onofrio presents a system that uses several data sources resulting in greater accuracy and redundancy ([Par 6] “In contrast to conventional systems, such as those described above, an ensemble of path perception approaches are collectively leveraged to produce a more accurate and reliable understanding of a driving surface and/or a path there through. For example, where a single path perception input may be inaccurate, an analysis of a plurality of path perception inputs provides testability and reliability for accurate and redundant lane mapping and/or path planning in real-time or near real-time. Specifically, through agreement/disagreement analyses of different path perception signal components, reliably metricizing path perception results live in an autonomous or semi-autonomous vehicle becomes possible—while also enabling a higher overall quality of path perception results.”) Overall, one of ordinary skill in the art would have recognized that combining the optimal sensor placement of VESPA with the robust, multi-source path perception of Onofrio would result in an autonomous driving system that is significantly more accurate and therefore safer.
The combination of VESPA and Onofrio does not explicitly teach determining data from a virtual sensor on a virtual machine; generating one or more sensor outputs from the virtual sensor within a virtual environment; lanes corresponding to the virtual environment;
Farabet makes obvious determining data from a virtual sensor on a virtual machine; generating one or more sensor outputs from the virtual sensor within a virtual environment; lanes corresponding to the virtual environment; ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.” [Par 31] “For example, one or more autonomous vehicle (AV) perception DNNs may be trained and/or tested, where the AV perception DNNs may be used for detecting lanes and boundaries on driving surfaces” [Par 53] “As such, when the simulated environment is used for testing vehicle performance (e.g., for HIL or SIL embodiments), the bot (e.g., the pedestrian) may behave as a real-world pedestrian would (e.g., by jaywalking in rainy or dark conditions, failing to heed stop signs or traffic lights, etc.), in order to more accurately simulate a real-world environment. This method may be used for any AI bot in the simulated environment, such as vehicles, bicyclists, or motorcycles, whose AI bots may also be trained to behave as real-world objects would (e.g., weaving in and out of traffic, swerving, changing lanes with no signal or suddenly, braking unexpectedly, etc.).”)
Farabet is analogous art because it is within the field of autonomous vehicle simulation. It would have been obvious to one of ordinary skill in the art combine it with VESPA and Onofrio before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to provide a simulated testbed on which to evaluate and improve the advanced lane-keeping system introduced by the combination of VESPA and Onofrio. As suggested, by typical methods for training and evaluating autonomous vehicle machine learning systems require lengthy data gathering from physical vehicles. Because of this, many edge cases and dangerous scenarios, particularly important scenarios such as high-speed collisions, are difficult to gather training data for, as it would require actually crashing vehicles on busy highways and streets, something that is exceedingly dangerous to say the least. ([Par 4] “For example, conventional systems often rely on data generated by physical vehicles navigating real-world environments to train the DNNs prior to deployment in working models. This approach has several limitations, however. For example, vehicles can only navigate to so many places, recreating dangerous or unique situations is difficult in the real-world environment, and testing the DNNs in these real-world environments may be dangerous. For example, especially where a DNN is used for obstacle avoidance or other safety measures, testing the DNNs in real-world environments may not be practical. On the other hand, an automaker likely will not deploy an autonomous vehicle into the real-world until an acceptable level of safety has been achieved. As a result, these competing interests make generating a sound, safe, accurate, and reliable autonomous driving system increasingly difficult.”) To this end, Farabet introduces an autonomous vehicle simulation system capable of simulating entire virtual environments in which to test simulated autonomous vehicles, allowing the evaluation of autonomous systems in scenarios that are impractical and/or dangerous to attempt to produce with real-world vehicles. ([Par 5-6] “Embodiments of the present disclosure relate training, testing, and verifying autonomous machines using simulated environments. Systems and methods are disclosed for training, testing, and/or verifying one or more features of a real-world system—such as a software stack for use in autonomous vehicles and/or robots. … In contrast to conventional systems, such as those described above, the systems of the present disclosure leverage a simulated environment to test one or more autonomous driving software stacks that include a multitude of DNNs. For example, physical sensor data, virtual sensor data, or a combination thereof may be used to train the DNNs of the software stack(s). … In any example, the simulated environment may be generated to create difficult to navigate, dangerous, unsafe, and/or otherwise unpredictable situations for the virtual object to navigate. As a result, previously untested scenarios (e.g., due to safety concerns, difficulty of reproduction, etc.) may be tested, repeated, and improved upon within the simulated environment.”) It would have been obvious to one of ordinary skill in the art that combining Farabet with VESPA and Onofrio would produce a system that allows the machine-learning based lane detection/keeping system of the combination of VESPA and Onofrio to be trained on a significantly wider set of data that can be generated at will, including data that would be dangerous to produce, such as camera footage during a high-speed collision. One of ordinary skill in the art would recognize that this wider dataset would allow the trained system to be even more accurate, maintaining tracking and autonomous system composure in the rare, high-stakes situations such as crashes where proper functioning is the most important.
Claim 12. The elements of claim 12 are substantially the same as those of claim 1. Therefore, the elements of claim 12 are rejected due to the same reasons as outlined above for claim 1.
Moreover, the additional elements of claim 12 are made obvious by the combination of VESPA, Onofrio, and Farabet. In particular,
Onofrio teaches A system comprising: one or more processors to execute operations comprising: ([Par 54] “For instance, various functions may be carried out by a processor executing instructions stored in memory. The method 400 may also be embodied as computer-usable instructions stored on computer storage media. The method 400 may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few.”)
Claim 23. The elements of claim 23 are substantially the same as those of claim 1. Therefore, the elements of claim 23 are rejected due to the same reasons as outlined above for claim 1.
Claim 2. VESPA teaches wherein the sampling space limits sensor installation angle poses of the installation poses to a range of angles relative to the virtual machine. ([Page 4 Col 1 Par 2] “. The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid.”)
Claim 18. The elements of claim 18 are substantially the same as those of claim 2. Therefore, the elements of claim 18 are rejected due to the same reasons as outlined above for claim 2.
Claim 3. Onofrio teaches wherein the updating includes updating the one or more parameters until the outputs computed by the one or more MLMs achieve a threshold level of accuracy with respect to the ground truth data. ([Par 62] “The DNN 502—during training, as indicated by the dashed lines—may be trained to predict the aggregate lane graph 124 using ground truth aggregate lane graphs 504 as ground truth data. In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein. As such, when the lane graphs 112A-112N are applied to the DNN 502, the aggregate lane graph 124 as predicted by the DNN 502 may be compared against the ground truth aggregate lane graph 504—using one or more loss functions 506—and the results of the comparison may be used to update parameters or coefficients (e.g., weights and biases) of the DNN 502. This process may be repeated until the predictions of the DNN 502 converge to acceptable or desirable levels of accuracy—e.g., until the loss is minimized.”)
Claim 4. VESPA teaches
(([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.” [Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions. The longitudinal and lateral ground truth were defined for non-ego vehicles and the position error was calculated from the fused sensor measurements. The deviation of sensor measurements from ground-truth was used to calculate the values of metrics m1–m8, and hence the cost function over all test cases.” [Examiner’s note: each vehicle has a variety of sensors (4 radars and 4 cameras) that generate data i.e. several sensors on the vehicle simultaneous generate output])
Farabet makes obvious wherein the one or more MLMs are further trained at least in part by:
instantiating the virtual sensor at a plurality of positions related to the virtual machine within the virtual environment; and simulating the virtual machine having the sensor instantiated thereon driving within the virtual environment. ([Par 52] “In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. . In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.). As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment. As another example, for an IMU sensor, the instance may be used for processing IMU data (e.g., representative of orientation) for the object in the simulated environment.”[Par 58] “The simulation system 400A may generate a simulated environment 410 that may include AI objects 412 (e.g., AI objects 412A and 412B), HIL objects 414, SIL objects 416, PIL objects 418, and/or other object types. The simulated environment 410 may include features of a driving environment, such as roads, bridges, tunnels, street signs, stop lights, crosswalks, buildings, trees and foliage, the sun, the moon, reflections, shadows, etc., in an effort to simulate a real-world environment accurately within the simulated environment 410. In some examples, the features of the driving environment within the simulated environment 410 may be more true-to-life by including chips, paint, graffiti, wear and tear, damage, etc.” [Par 71] “In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the HIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment for each sensor (e.g., a first instance from a field of view of a first virtual camera of the virtual vehicle, a second instance from a field of view of a second virtual camera, a third instance from a field of view of a virtual LIDAR sensor, etc.). The instances of the simulated environment may thus be used to generate sensor data for each sensor by the vehicle simulator component(s) 420.” [Par 123] “The method 1000, at block B1020, includes controlling a virtual object within a simulated environment based at least in part on the output. For example, the virtual object (e.g., virtual vehicle) may be controlled within the simulated environment based at least in part on the output. In other examples, the outputs may be used for control. For example, the outputs may be object detection, lane detection, drivable free-space detection, safety procedure determination, etc.”)
Claim 5. VESPA teaches wherein the determining of the sampling space includes defining a range in variance for one or more installation pose properties, the range in variance being relative to the ([Page 4 Col 1 Par 1-2] “Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle…. The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects.)”)
Claim 6. VESPA teaches wherein a first sensor pose of the plurality of installation poses corresponds to a first installation angle of the ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
Farabet makes obvious the virtual sensor on the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 16. The elements of claim 16 are substantially the same as those of claim 6. Therefore, the elements of claim 16 are rejected due to the same reasons as outlined above for claim 6.
Claim 7. VESPA teaches wherein at least some of the sensor installation poses correspond to different sensor installation locations relative to the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw). ” [Page 5 Col 1 Par 3] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab.”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 8. VESPA teaches wherein at least some of the sensor installation poses correspond to different sensor installation heights relative to the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw).” [Examiner’s note: sensor configuration is varied across 3 dimensions, i.e. including height])
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 9. Onofrio teaches wherein the output corresponds to at least one of a permanent lane boundary, a temporary lane boundary, or a physical lane boundary. ([Par 32] “The perception outputs from the perception sources—e.g., the (post-processed) outputs of the DNNs 104, the outputs of the HD map(s) 106, and/or the outputs of the object trace(s) 108—may be used to generate the lane graphs 112.” [Fig. 3A] shows detected permanent lane boundaries 314 and 318] [Par 50] “The aggregate lane graph 124 of the visualization 300 may include a left adjacent lane left edge 308, a left adjacent lane rail 310, a left adjacent lane right edge 312, an ego-lane left edge 314, an ego-lane rail 316, an ego-lane right edge 318, a right adjacent lane left edge 320, a right adjacent lane rail 322, and a right adjacent lane right edge 324”)
Claim 10. VESPA teaches wherein the one or more sensor outputs are further generated based at least on varying locations of one or more real-world vehicles using real-world sensor data that captures the locations of the one or more real- world vehicles ([Page 6 Col 2 Par 2] “Our last experiment involved testing the best sensor configuration from our VESPA framework and the baseline configuration for the 2019 Blazer on data from a real world drive cycle over one hour in Colorado. We focus only on assessing performance for the ACC and FCW features. Fig. 8 shows an image from the real drive cycle with data collected by the vehicle from the radar and camera sensors on it. The figure also shows a plot of the object occlusion rate (OOR). The OOR for the baseline configuration was 19.64% (it did not detect 11 out of 56 non ego vehicles), while the VESPA generated best solution had an OOR of 7.14% (it failed to detect only 4 out of 56 non ego vehicles). The results show the effectiveness of our proposed VESPA framework in generating higher quality sensor configurations.” [Fig. 5] Shows sensor placements on the physical vehicles [Figure 8] shows a snapshot from the experimental real-world drive cycle as well as associated detection/occlusion rates)
PNG
media_image1.png
255
744
media_image1.png
Greyscale
PNG
media_image2.png
341
730
media_image2.png
Greyscale
Farabet makes obvious performing operations to determine locations of the virtual machine in the virtual environment. ([Par 69] “For example, the vehicle simulator component(s) 422 may receive (e.g., retrieve, obtain, etc.), from the global simulation (e.g., represented by the simulated environment 410) hosted by the simulator component(s) 402, data that corresponds to, is associated with, and/or is required by the vehicle simulator component(s) 422 to perform one or more operations by the vehicle simulator component(s) 422 for the PIL object. In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the PIL object may be received from the simulator component(s) 402.” [Par 114] “The method 900, at block B904, includes generating virtual sensor data for each of a dynamically configurable number of virtual sensors. For example, the vehicle simulator component(s) 406, 420, and/or 422 may generate virtual sensor data using the simulation data for each of the virtual sensors of the vehicle. The virtual sensor data may be representative of the simulated environment 410 as perceived by at least one virtual sensor of a dynamically configurable number of virtual sensors of a virtual object within the simulated environment 410 (e.g., sensor data of a field of view of a virtual camera(s), sensor data of an orientation of the virtual vehicle using virtual IMU sensors, etc.).”
Claim 11. Onofrio teaches wherein the at least one processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; ([Par 22] “Systems and methods are disclosed related to path perception diversity and redundancy in autonomous machine applications.”) a system for performing simulation operations; a system for performing learning operations; a system implemented using an edge device; a system implemented using a robot; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. [This claim is written in the alternative format. Unmapped elements are therefore not given patentable weight]
Claim 22. The elements of claim 22 are substantially the same as those of claim 11. Therefore, the elements of claim 22 are rejected due to the same reasons as outlined above for claim 11.
Claim 14. Farabet teaches wherein the operations further comprise instantiating the virtual machine at poses within the virtual environment, wherein each sensor output of the plurality of sensor outputs is generated using a respective instantiation of the virtual machine. ([Par 52] “In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. . In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.). As such, an instance of the engine may be used for processing sensor data for each sensor with respect to the sensor's perception of the global simulation. As such, for a virtual camera, the instance may be used for processing image data with respect to the camera's field of view in the simulated environment. As another example, for an IMU sensor, the instance may be used for processing IMU data (e.g., representative of orientation) for the object in the simulated environment.” [Par 71] “In such an example, data (e.g., virtual sensor data corresponding to a field(s) of view of virtual camera(s) of the virtual vehicle, virtual LIDAR data, virtual RADAR data, virtual location data, virtual IMU data, etc.) corresponding to each sensor of the HIL object may be received from the simulator component(s) 402. This data may be used to generate an instance of the simulated environment for each sensor (e.g., a first instance from a field of view of a first virtual camera of the virtual vehicle, a second instance from a field of view of a second virtual camera, a third instance from a field of view of a virtual LIDAR sensor, etc.). The instances of the simulated environment may thus be used to generate sensor data for each sensor by the vehicle simulator component(s) 420.”)
Claim 17. VESPA makes obvious wherein the sampling includes generating, using the sampling space, sensor locations of the ([Page 3 Col 2 Par 2 – Page 4 Col 1 Par 4] “Fig. 2 shows an overview of our proposed VESPA framework. The physical dimensions of the vehicle model and the number and type of sensors to be considered are inputs to the framework. A design space exploration algorithm is used to generate a sensor configuration which is subsequently evaluated based on a cumulative score from the performance metrics presented in the previous section. We evaluate three design space exploration algorithms: simulated annealing with greedy randomized adaptive search (SA+GRASP), genetic algorthm (GA), and particle swarm optimization (PSO). The process of sensor configuration generation and evaluation continues until an algorithm-specific stopping criteria is met, at which point the best configuration is output. The following subsections describe our framework in more detail…. Each of the design space exploration algorithms generates sensor configurations that consider feature to field of view (FOV) zone correlations around the ego vehicle. Fig. 3(a) shows the FOV zones around the ego-vehicle. These zones of interest are defined as the most important perception areas in the environment for a particular feature. Fig. 3(b) shows the regions on the vehicle on which sensors can be mounted (in blue). Regions F and G (in yellow) are exempt from sensor placement due to the mechanical instability of placing sensors on the door of a vehicle. … For exploration of possible locations within a region, a fixed step size of 5cm in two dimensions across the surface of the vehicle is considered, which generates a 2D grid of possible positions in each zone shown in Fig. 3(b), (c). The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives… The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.”) wherein at least a first sensor output of the plurality of sensor outputs is generated at a first sensor location and at least a second sensor output of the plurality of sensor outputs is generated at a second sensor location different from the first sensor location. ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
Farabet makes obvious the virtual sensor on the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 28. The elements of claim 28 are substantially the same as those of claim 17. Therefore, the elements of claim 28 are rejected due to the same reasons as outlined above for claim 17.
Claim 19. Onofrio teaches wherein the one or more MLMS are trained to compute locations of one or more lane boundaries, the one or more lane boundaries including at least one of a left lane boundary, a right lane boundary, or a lane rail. ([Par 33] “In addition, the lane graphs 112A-112N generated by the DNNs 104A-104N may be largely independent as a result of the underlying training data used to train the DNNs” [Par 36] “To generate the lane graphs 112, produced perception output may be converted into a set of polylines—which may be labeled or classified according to their associated lane and/or lane line type, such as left edge, right edge, lane rail, etc.”)
Claim 20. Onofrio teaches wherein the ground truth data ([Par 62] “In some non-limiting embodiments, the ground truth aggregate lane graphs 504 may be generated using the lane graph aggregator 114 as described herein”) further represents a trajectory along the lane, and the one or more MLMs are trained to compute locations of one or more trajectory points. ([Par 33] “For example, a first output from a first DNN(S) 104 may include a location and classification of lane edges or rails, while a second output from a second DNN(S) 104 may include a recommended trajectory along a driving surface—which may be independent of any actual lanes on the driving surface.” [Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats—“)
Claim 21. Onofrio teaches wherein, in deployment, the one or more MLMs compute at least one of locations of one or more lane boundaries or locations of one or more trajectory points using sensor data obtained using one or more real-world sensors of a real-world vehicle. ([Par 35] “However, because each of the perception sources may generate perception outputs in different formats—e.g., locations of lane rails, locations of lane edges, classifications of the lane lines, locations of trajectories of leading vehicles, locations of recommended paths, and/or other formats—“ [Par 32] “In such an example, the lane graph 112N-1 may be generated using data from an external offline mapping system while the lane graph 112A may be generated via live perception of the DNN(S) 104A. Moreover, in the example of the lane graph 112N-2, since the sensor data 102 may be combined across multiple sensors (e.g., RADAR, IMU, GNSS, camera, etc.) rather than just a camera,” [Par 81] “One or more of the camera(s) (e.g., all of the cameras) may record and provide image data (e.g., video) simultaneously.”)
Claim 25. VESPA teaches wherein the sampling space corresponds to a plurality of different vehicle ([Par 4 Col 1 Par 2] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro”)
Farabet makes obvious wherein the system considers a plurality of different vehicle makes ([Par 114] “The number of virtual sensors used may be dynamically configurable such that one sensor may be used in a first simulation, five in another, ten in another, etc. In some examples, the dynamic configuration may be determined based on vehicle types (e.g., a first vehicle of year X, make Y, model Z may include 20 sensors, while a second vehicle of year A, make B, model C may include 30 sensors).”)
Claim 27. VESPA teaches wherein the installation poses correspond to a plurality of installation angles relative to the ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”)
Farabet makes obvious the virtual machine ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claim 29. VESPA makes obvious wherein the first sensor location includes a first longitudinal location, a first lateral location, and a first vertical location on the ([Page 4 Col 1 Par 4] “The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro. As this design space is so large that it cannot be exhaustively traversed in a practical amount of time, we explore the use of intelligent design space search algorithms that support hill climbing to escape local minima.” [Page 4 Col 2 Par 2] “The GA is adapted for our design space such that a chromosome is defined by the combined location and orientation of each sensor’s configuration (consisting of six parameters: x, y, z, roll, pitch, and yaw). For a given set of N sensors, the number of parameters stored in each chromosomes is thus ‘6N’. Next, in the selection stage, the cost function values are computed for 100 configurations at a time, and a roulette wheel selection method is used to select which set of chromosomes will be involved in the crossover step based on their cost function probability value, computed as a fraction of the cumulative cost function sum of all chromosomes considered in the selection.” [Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
Farabet makes obvious virtual sensors on a virtual vehicle ([Par 52] “The simulation system 400—e.g., represented by simulation systems 400A, 400B, 400C, and 400D, described in more detail herein—may generate a global simulation that simulates a virtual world or environment (e.g., a simulated environment) that may include artificial intelligence (AI) vehicles or other objects (e.g., pedestrians, animals, etc.), hardware-in-the-loop (HIL) vehicles or other objects, software-in-the-loop (SIL) vehicles or other objects, and/or person-in-the-loop (PIL) vehicles or other objects. … In some examples, as described herein, one or more vehicles or objects within the simulation system 400 (e.g., HIL objects, SIL objects, PIL objects, AI objects, etc.) may be maintained within their own instance of the engine. In such examples, each virtual sensor of each virtual object may include their own instance of the engine (e.g., an instance for a virtual camera, a second instance for a virtual LIDAR sensor, a third instance for another virtual LIDAR sensor, etc.).” [Par 97] “ As such, for each frame represented by the virtual sensor data, the simulator component(s) 402 may create a list of all tracked objects (e.g., trees, vehicles, pedestrians, foliage, etc.) within range of the virtual object having the virtual LIDAR sensors, and may cast virtual rays toward the tracked objects.”)
Claims 13 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1) as well as DIY Lane Keeping (hereinafter Rivera)
Claim 13. VESPA teaches wherein the plurality of sensor outputs ([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.”)
The combination of VESPA, Onofrio, and Farabet does not explicitly teach wherein data correspond to a plurality of lateral machine poses within or outside of the lane, and wherein at least one lateral machine pose of the plurality of lateral machine poses is laterally offset with respect to at least one other machine pose of the plurality of lateral machine poses.
Rivera makes obvious wherein data correspond to a plurality of lateral machine poses within or outside of the lane, ([Page 7 Par 2] “We also assume that the center of the image is lined up with the center of the car. Knowing the location of the center of the car and the center of the lane, we can calculate the deviation from the center of the lane. This deviation will be used as the error that we are trying to minimize.” [Page 8 Fig. 1, Figs Pages 16-17] Show various video snippets showing the vehicle moving within the lane lines. Since the vehicle is moving, each frame is a new pose) and wherein at least one lateral machine pose of the plurality of lateral machine poses is laterally offset with respect to at least one other machine pose of the plurality of lateral machine poses. ([Page 8 Fig. 1, Figs Pages 16-17] As can be seen in the figures, particularly in the “distance from center” indicator of each, the vehicle is moving laterally within the lanes throughout the video, i.e. one of more positions are laterally offset from one of more other positions)
Rivera is analogous art because it is within the field of autonomous vehicle operations with a particular application to lane management. It would have been obvious to one of ordinary skill in the art to combine it with VESPA, Onofrio, and Farabet before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to integrate vehicles of varying levels of autonomy into the simulation, so more realistic training data can be generated. As suggested by Rivera, there are a variety of levels of vehicle autonomy, from simple non-adaptive cruise control to full-self driving capabilities. ([Page 3 Par 1] “The Society of Automotive Engineers (SAE) has developed a list of automation levels to help us compare different self-driving systems, ranging from no automation (Level 0) to full automation with no human driver in the loop (Level 5). Waymo’s level of automation can be described as Level 4, meaning that the car is fully autonomous in a geo-fenced area.” [Page 3 Par 3] “My current daily driver is a sedan from 2003.It has a non-adaptive cruise control system that has been common since the 1970s. As a side project I wanted to see if I could use the latest technology to upgrade my car from an outdated Level 1 to a slightly less outdated Level 2 by building and retrofitting a lane keeping system. The goal of this project is not to build a full self-driving car like Waymo or Cruise, or even an Autopilot system like the one found on Teslas, but rather a lane keeping system like the ones offered by automotive manufacturers since 2014.”) It would be obvious to one of ordinary skill in the art that the environment a fully autonomous would operate within would be one in which it shares the road with the entire range, including vehicles with limited autonomous capabilities. To this end, Rivera presents and easy-to implement level 2 autonomous vehicle system capable of adaptive cruise control and lane keeping, but nothing else ([Page 4 Par 2- 3] “For this to be considered a success the systems needs to be able to steer and keep the car in the middle of the lane on a well marked road. There are 3 basic components to the systems I’m building. First, it needs to identify the lane. Second, it needs to identify how far away from the center of the lane the car is. Lastly it needs a control system that keeps the car in the center of the lane. … The only sensor used in this project is a webcam mounted on the windshield of the car and providing video input to the rest of the system.” [Page 5 Par 1] “When looking for a better approach I came across a paper that describes neural network capable of detecting lane lines called Lanenet. Don’t worry if you are not well versed in neural networks. I’ll only touch on them briefly in this write up and you don’t need to understand their inner workings to re-implement this project”) Overall, one of ordinary skill in the art would have recognized that combining Rivera with VESPA, Onofrio, and Farabet would produce a simulation system in which fully-autonomous vehicles interact with a much more diverse cast of vehicle types, producing a wider range of possibilities and thus a greater amount of useful training data.
Claim 24. The elements of claim 24 are substantially the same as those of claim 13. Therefore, the elements of claim 24 are rejected due to the same reasons as outlined above for claim 13.
(3) Claims 15 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over VESPA: A Framework for Optimizing Heterogeneous Sensor Placement and Orientation for Autonomous Vehicles (Hereinafter VESPA) in view of Onofrio (US 20200249684 A1) in further view of Farabet (US 20190303759 A1) as well as DeMersseman (US 20180172966 A1)
Claim 15. VESPA makes obvious wherein at least one installation pose of the plurality of installation poses ([Page 4 Col 1 Par 2-4] “The orientation exploration of each sensor involves rotation at a fixed step size of 1 degree between an upper and lower bounding limit for roll, pitch and yaw respectively, at each of these possible positions within the 2D grid. The orientation exploration limits were chosen with caution to the caveat that long range radars with extreme orientations increase the number of recorded false positives … The design space considered in this work uses 4 radars and 4 cameras that can be placed in any zone. With a fixed step size of 5cm in each dimensions and 1 degree rotation in orientation, the number of ways 8 sensors can be placed in all unique locations and orientations is 2.56e+23C8 for the 2019 Blazer and 6.4e+22C8 for the 2016 Camaro.”) is behind a windshield and the generating the plurality of sensor outputs([Page 5 Col 1 Par 3 – Col 2 Par 1] “Each configuration generated by the SA+GRASP, GA, and PSO algorithms was optimized on 40 test cases designed (10 test cases each for evaluating performance with ACC, FCW, LKA, and BW) using the Automated Driving Toolbox in Matlab. Half (20) of these test cases for each feature are used during the optimization phase and the remaining (20) test cases are used during the evaluation phase. Finally, the optimized configurations were evaluated on a different set of evaluation test cases. Each of the test cases was characterized by unique road geometry, variations in road elevation, curvature, banking, and different traffic densities. In some test cases, the number of lanes were varied to make the framework optimize the sensor configuration for challenging and realistic driving scenarios. A Kalman filter sensor fusion algorithm was used to combine readings from sensors in a sensor configuration being evaluated, to make predictions.” [Fig. 3] Clearly shows one of the possible sensor placement regions as being behind the windshield. Note that cameras positioned here, cameras being one of the sensor types considered, would capture field of views through the windshield])
PNG
media_image3.png
540
752
media_image3.png
Greyscale
DeMersseman makes obvious wherein a sensor is located behind a windshield and the system includes varying one or more physical properties of the windshield ([Par 35] “The integrated light/rain sensor and communication antenna module 100 is generally mounted on a back (interior) surface of a windshield 92 of the vehicle 90. In various embodiments, the integrated light/rain sensor and communication antenna module 100 is attached to the windshield using a transparent adhesive material (or gasket).” [Par 50] “Referring to FIG. 11, a graph 160 is shown illustrating a relative light efficiency at varying windshield thicknesses of an integrated light/rain sensor and communication antenna module in accordance with an embodiment of the invention. The graph 160 illustrates simulation of a light/rain sensor in accordance with an embodiment of the invention for windshield thickness offsets from 4.2 mm to 6.6 mm. Simulations are shown for a flat windshield and a windshield with a radius of 1400 mm (e.g., convex as seen from outside the vehicle 90). The simulation shows the light/rain sensor in accordance with an embodiment of the invention works well for a range of thicknesses (e.g., 4.8 mm to 6.0 mm) covering a majority of windshields.”)
DeMersseman is analogous art because it is within the field of vehicular sensing. It would have been obvious to one of ordinary skill in the art to combine DeMersseman with VESPA, Onofrio, and Farabet before the effective filing date. One of ordinary skill in the art would have been motivated to make this combination in order to improve the sensing capabilities of the autonomous vehicle in deployment. While the combination of VESPA, Onofrio, and Farabet disclose a variety of sensors that can be included in the vehicle, a notable absence is that of a rain sensor. As noted by Farabet, rain is a condition in which machine-learning based autonomous driving systems are known to become inaccurate ([Par 49] “In some examples, pre-determined conditions or combinations of conditions (such as those described herein) that the DNNs fail to perform accurately enough may also be used to direct and focus data gathering at the edge. For example, the vehicle(s) 102 may not collect all data, but may only collect data where certain conditions or combinations of conditions are met (e.g., at night, in the rain, in certain tunnel types, etc.).”) This can prove extremely dangerous is the system loses significant accuracy but is unable to detect such rainy conditions and take countermeasures, such as self-adjustment or giving control over to a human driver in extreme conditions. (Note that the latter human handover feature is mentioned by Onofrio as a method of recourse if accuracy gets too low ([Par 45] “In an autonomous driving example, the vehicle 700 may disengage autonomous functionality and hand control back to the driver when the confidence along a desired or recommended path of the vehicle 700 is below a threshold.”)) To this end, DeMersseman presents a compact rain sensor system for vehicles ([Par 34] “Embodiments of the present invention include providing a lens for an integrated rain sensor and communication antenna that may (i) provide rain sensing, (ii) provide ambient light sensing, (iii) provide tunnel detection, (iv) provide sunload sensing, (v) take advantage of space behind a rearview mirror; (vi) integrate several features together, (vii) provide a more compact solution when compared with existing standalone solutions, (viii) offer a lower cost solution when compared with standalone solutions, (ix) enable faster assembly, (x) enable physical integration of optical rain sensing with an RF antenna, (xi) facilitate global location determination by one or more satellite constellations such as GPS-Glonass-Beidou-Gallileo, (xii) facilitate connectivity such as vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), and vehicle-to-everything (V2X) communication, (xiii) be implemented on a single printed circuit board, and/or (xiv) utilize a novel molded lens structure.”) Overall, one of ordinary skill in the would have recognized that combining DeMersseman with VESPA, Onofrio, and Farabet would result in a vehicle system that is more aware of its surroundings and potential hazards in deployment, allowing for adjustments and countermeasures to be taken if conditions become too poor, ultimately resulting in a much safer autonomous driving system.
Claim 26. The elements of claim 26 are substantially the same as those of claim 15. Therefore, the elements of claim 26 are rejected due to the same reasons as outlined above for claim 15.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michael P Mirabito whose telephone number is (703)756-1494. The examiner can normally be reached M-F 10:30 am - 6:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emerson Puente can be reached on (571) 272-3652. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.P.M./ Examiner, Art Unit 2187
/EMERSON C PUENTE/ Supervisory Patent Examiner, Art Unit 2187