Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendments
Claims 1-6, 8-13, and 15-22 are presented for examination.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, 8, 13, 15, and 19 are rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 in view of Canady et al., U.S. 2021/0197859.
On claim 1, Pronovost cites except as underlined:
A perception sensor configuration optimization system comprising:
a memory having instructions stored therein;
figure 8 and [0134] Memory 818 and memory 838 are examples of non-transitory computer-readable media. The memory 818 and memory 838 may store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems.
and
a processor configured to execute the instructions stored in said memory
figure 8, Processors 818
to cause said perception sensor configuration optimization system to:
receive perception sensor input data of an autonomous vehicle, wherein the perception sensor input data corresponds to respective measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle;
figure 8 and [0035]
In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like. In various examples
(the cited GPS and ultrasonic, camera, microphone, radar, gyros, and the like are analogous to the claimed “plurality of perception sensors”).
receive ground-truth information input data of the autonomous vehicle, wherein the ground-truth information input data corresponds to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
[0027] Further, detected positions over such a period of time associated with the object may be used to determine a ground truth trajectory to associate with the object. In some examples, the vehicle computing device may provide the data to a remote computing device (i.e., computing device separate from vehicle computing device) for data analysis. In such examples, the remote computing device may analyze the sensor data to determine one or more labels for images, an actual location, yaw, speed, acceleration, direction of travel, or the like of the object at the end of the set of estimated states. In some such examples, ground truth data associated with one or more of: positions, trajectories, accelerations, directions, and so may be determined (either hand labelled or determined by another machine learned model) and such ground truth data may be used to determine a trajectory of an object.
And
[0130] In some instances, the training component 848 may be executed by the processor(s) 836 to train a machine learning model based on training data. The training data may include a wide variety of data, such as sensor data, audio data, image data, map data, inertia data, vehicle state data, historical data (log data), or a combination thereof, that is associated with a value (e.g., a desired classification, inference, prediction, etc.). Such values may generally be referred to as a “ground truth.” To illustrate, the training data may be used for determining risk associated with occluded regions and, as such, may include data representing an environment that is captured by an autonomous vehicle and that is associated with one or more classifications or determinations. In some examples, such a classification may be based on user input (e.g., user input indicating that the data depicts a specific risk) or may be based on the output of another machine learned model. In some examples, such labeled classifications (or more generally, the labeled output associated with training data) may be referred to as ground truth.
determine, based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
and output a safety-aware occupancy signal based on the safety occupancy.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
(In the above passages, and, in figure 1, several objects are highlighted has having a predicted contact with vehicle 102: object 110. In the alternative, vehicle 102 includes possible candidates such as object 108. Each respective object 110 and 108 has trajectories 116 and 114, respectively). The claimed “safety occupancy” is met by the cited objects above being in the area of influence of vehicle 102.
Regarding the excepted: receive perception sensor input data of an autonomous vehicle, wherein the perception sensor input data corresponds to respective measured location, orientation, as disclosed above, Pronovost disclosed perception data input using various sensors. However, Pronovost didn’t disclose sensors indicating the orientation of a vehicle.
In the same art of vehicle monitoring systems, Canady cites:
[0014] A data metric can be evaluated using a model generated based at least in part on empirically measured data and/or simulated data. For example, a data metric can be evaluated for sensor data captured in an environment. The sensor data can be input to a component such as a perception component to detect objects and/or to determine information about such objects. For example, image data or lidar data can be used to detect an object, such as a pedestrian, as well as a bounding box associated with the object (e.g., two-dimensional or three-dimensional bounding box), segmentation information, classification information, pose (e.g., orientation), velocity information, extent (e.g., length, width, and/or height), and the like.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost the embodiment disclosed in Canady such that the claimed invention is realized.
Canady discloses a known perception input, “orientation,” to determine the direction or posture of the vehicle. One of ordinary skill would have included this feature as an added parameter for more accurately describing the condition of the monitored vehicle.
On claim 6, Pronovost cites:
The perception sensor configuration optimization system of claim 1, wherein the plurality of perception sensors of the autonomous vehicle include a distance perception sensor configured to provide distance information input data of the at least one obstacle,
[0035] In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like.
and
wherein said processor is further configured to execute the instructions stored in said memory to cause said perception sensor configuration optimization system to determine the safety occupancy by determining, based on the distance information input data and the ground-truth information input data, the safety occupancy of the at least one obstacle within the area around the autonomous vehicle.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
On claim 8, Pronovost cites except as underlined:
A method comprising:
receiving perception sensor input data of an autonomous vehicle, the perception sensor input data corresponding to respective measured location, orientation, and type of each of a plurality of perception sensors of the autonomous vehicle;
receiving ground-truth information input data of the autonomous vehicle, the ground-truth information input data corresponding to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
figure 8 and [0035]
In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like. In various examples
(the cited GPS and ultrasonic, camera, microphone, radar, gyros, and the like are analogous to the claimed “plurality of perception sensors”).
determining, via a processor configured to execute instructions stored in a memory,
figure 8 and [0134] Memory 818 and memory 838 are examples of non-transitory computer-readable media. The memory 818 and memory 838 may store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems.
and
figure 8, Processors 818
based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
[0027] Further, detected positions over such a period of time associated with the object may be used to determine a ground truth trajectory to associate with the object. In some examples, the vehicle computing device may provide the data to a remote computing device (i.e., computing device separate from vehicle computing device) for data analysis. In such examples, the remote computing device may analyze the sensor data to determine one or more labels for images, an actual location, yaw, speed, acceleration, direction of travel, or the like of the object at the end of the set of estimated states. In some such examples, ground truth data associated with one or more of: positions, trajectories, accelerations, directions, and so may be determined (either hand labelled or determined by another machine learned model) and such ground truth data may be used to determine a trajectory of an object.
And
[0130] In some instances, the training component 848 may be executed by the processor(s) 836 to train a machine learning model based on training data. The training data may include a wide variety of data, such as sensor data, audio data, image data, map data, inertia data, vehicle state data, historical data (log data), or a combination thereof, that is associated with a value (e.g., a desired classification, inference, prediction, etc.). Such values may generally be referred to as a “ground truth.” To illustrate, the training data may be used for determining risk associated with occluded regions and, as such, may include data representing an environment that is captured by an autonomous vehicle and that is associated with one or more classifications or determinations. In some examples, such a classification may be based on user input (e.g., user input indicating that the data depicts a specific risk) or may be based on the output of another machine learned model. In some examples, such labeled classifications (or more generally, the labeled output associated with training data) may be referred to as ground truth.
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
and
outputting, via the processor, a safety-aware occupancy signal based on the safety occupancy.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
(In the above passages, and, in figure 1, several objects are highlighted has having a predicted contact with vehicle 102: object 110. In the alternative, vehicle 102 includes possible candidates such as object 108. Each respective object 110 and 108 has trajectories 116 and 114, respectively). The claimed “safety occupancy” is met by the cited objects above being in the area of influence of vehicle 102.
Regarding the excepted: orientation, Pronovost disclosed perception data input using various sensors. However, Pronovost didn’t disclose sensors indicating the orientation of a vehicle.
In the same art of vehicle monitoring systems, Canady cites:
[0014] A data metric can be evaluated using a model generated based at least in part on empirically measured data and/or simulated data. For example, a data metric can be evaluated for sensor data captured in an environment. The sensor data can be input to a component such as a perception component to detect objects and/or to determine information about such objects. For example, image data or lidar data can be used to detect an object, such as a pedestrian, as well as a bounding box associated with the object (e.g., two-dimensional or three-dimensional bounding box), segmentation information, classification information, pose (e.g., orientation), velocity information, extent (e.g., length, width, and/or height), and the like.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost the embodiment disclosed in Canady such that the claimed invention is realized.
Canady discloses a known perception input, “orientation,” to determine the direction or posture of the vehicle. One of ordinary skill would have included this feature as an added parameter for more accurately describing the condition of the monitored vehicle.
On claim 13, Pronovost cites:
The method of claim 8, wherein said determining the safety occupancy of obstacles within the area around the autonomous vehicle comprises
determining, via the processor and based on distance information input data of the at least one obstacle and the ground-truth information input data,
[0035] In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like.
the safety occupancy of obstacles within the area around the autonomous vehicle.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
On claim 15, Pronovost cites except as underlined:
A non-transitory, computer-readable media having computer-readable instructions stored thereon,
figure 8 and [0134] Memory 818 and memory 838 are examples of non-transitory computer-readable media. The memory 818 and memory 838 may store an operating system and one or more software applications, instructions, programs, and/or data to implement the methods described herein and the functions attributed to the various systems.
which, when executed across one or more processors,
figure 8, Processors 818
causes at least a portion of the one or more processors to perform operations comprising:
receiving perception sensor input data of an autonomous vehicle, the perception sensor input data corresponding to respective measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle;
figure 8 and [0035]
In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like. In various examples
(the cited GPS and ultrasonic, camera, microphone, radar, gyros, and the like are analogous to the claimed “plurality of perception sensors”).
receiving ground-truth information input data of the autonomous vehicle, the ground-truth information input data corresponding to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
[0027] Further, detected positions over such a period of time associated with the object may be used to determine a ground truth trajectory to associate with the object. In some examples, the vehicle computing device may provide the data to a remote computing device (i.e., computing device separate from vehicle computing device) for data analysis. In such examples, the remote computing device may analyze the sensor data to determine one or more labels for images, an actual location, yaw, speed, acceleration, direction of travel, or the like of the object at the end of the set of estimated states. In some such examples, ground truth data associated with one or more of: positions, trajectories, accelerations, directions, and so may be determined (either hand labelled or determined by another machine learned model) and such ground truth data may be used to determine a trajectory of an object.
And
[0130] In some instances, the training component 848 may be executed by the processor(s) 836 to train a machine learning model based on training data. The training data may include a wide variety of data, such as sensor data, audio data, image data, map data, inertia data, vehicle state data, historical data (log data), or a combination thereof, that is associated with a value (e.g., a desired classification, inference, prediction, etc.). Such values may generally be referred to as a “ground truth.” To illustrate, the training data may be used for determining risk associated with occluded regions and, as such, may include data representing an environment that is captured by an autonomous vehicle and that is associated with one or more classifications or determinations. In some examples, such a classification may be based on user input (e.g., user input indicating that the data depicts a specific risk) or may be based on the output of another machine learned model. In some examples, such labeled classifications (or more generally, the labeled output associated with training data) may be referred to as ground truth.
determining, via a processor configured to execute instructions stored in a memory, based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
and outputting, via the processor, a safety-aware occupancy signal based on the safety occupancy.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
On claim 19, Pronovost cites:
The non-transitory, computer-readable media of claim 15, wherein determining the safety occupancy of obstacles within the area around the autonomous vehicle comprises determining, via distance information input data of the at least one obstacle and the ground-truth information input data,
[0035] In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like.
the safety occupancy of obstacles within the area around the autonomous vehicle.
[0037] Generally, the prediction component 104 provides functionality to determine an object trajectory 114 associated with the pedestrian 108, and determine an object trajectory 116 associated with the vehicle 110. The prediction component 104 can also or instead predict scene data that describes a simulated environment. For instance, the prediction component 104 can output one or more scenes usable in a simulation (also referred to as a scenario or estimated states) to determine a response by the vehicle 102 to a simulated object. In some examples, the prediction component 104 can generate the output data 106 to represent one or more heat maps. In some examples, the one or more predicted trajectories may be determined or represented using a probabilistic heat map to predict object behavior, such as that described in U.S. patent application Ser. No. 15/807,521, filed Nov. 8, 2017, entitled “Probabilistic Heat Maps for Behavior Prediction,” which is incorporated herein by reference in its entirety and for all purposes.
Claim 4 is rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 (as evidenced by Read-Coop 2023) in view of Canady et al., U.S. 2021/0197859 and Roychowdhury et al., U.S. 2022/0044034 (hereinafter 034).
On claim 4, Pronovost (as evidenced by Read-Coop 2023) cites:
The perception sensor configuration optimization system of claim 2, wherein said processor is further configured to execute the instructions stored in said memory to additionally cause said perception sensor configuration optimization system to compare the safety-aware occupancy with benchmark data corresponding to predetermined thresholds of acceptability.
See the rejection of claim 1 citing Pronovost in view of Canady. The cited “ground-truth data” as indicated in Read-Coop 2003, page 1, paragraph 1, means
“In short, Ground Truth is the accurate and verified data which is used to train machine learning models.”
Pronovost discloses an embodiment in which ground truth data is used to train a machine learning model. In short, “ground truth data” is the same as the claimed “benchmark data,” and the ground truth data is used as a reference from which perception, or sensor data, is verified.
Claim 11 is rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 (as evidenced by Read-Coop 2023) in view of Canady et al., U.S. 2021/0197859 and Roychowdhury et al., U.S. 2022/0044034 (hereinafter 034) and Refaat et al., U.S. 2020/0150665.
On claim 11, Pronovost (as evidenced by Read-Coop 2023) cites:
The method of claim 9, further comprising: receiving, via the processor and from a benchmark system, benchmark data corresponding to predetermined thresholds of acceptability; and comparing, via the processor, the safety-aware occupancy with the benchmark data.
See the rejection of claim 1 citing Pronovost in view of Canady. The cited “ground-truth data” as indicated in Read-Coop 2003, page 1, paragraph 1, means
“In short, Ground Truth is the accurate and verified data which is used to train machine learning models.”
Pronovost discloses an embodiment in which ground truth data is used to train a machine learning model. In short, “ground truth data” is the same as the claimed “benchmark data,” and the ground truth data is used as a reference from which perception, or sensor data, is verified.
Claims 2, 5, 12, 16, 18, and 21 are rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 in view of Canady et al., U.S. 2021/0197859 and Roychowdhury et al., U.S. 2022/0044034 (hereinafter 034).
On claim 2, Pronovost and Canady cites except as underlined:
The perception sensor configuration optimization system of claim 1, wherein said processor is further configured to execute the instructions stored in said memory to cause said perception sensor configuration optimization system to:
determine, based on the perception sensor input data and the ground-truth information input data, a general occupancy of the at least one obstacle within the area around the autonomous vehicle by:
splitting a region around the autonomous vehicle into a plurality of voxels; and
calculating a respective likelihood of the at least one obstacle occupying each voxel of the plurality of voxels;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
determine, based on the general occupancy of the at least one obstacle and the safety occupancy of the at least one obstacle, a safety-aware occupancy; and output the safety-aware occupancy signal based on the safety-aware occupancy.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features. In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track a vehicle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels. Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle. The tracked vehicle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 5, Pronovost cites except as underlined:
The perception sensor configuration optimization system of claim 1, wherein said processor is further configured to execute the instructions stored in said memory to cause said perception sensor configuration optimization system to determine the safety occupancy by:
splitting a region around the autonomous vehicle into a plurality of voxels;
calculating, for each voxel of the plurality of voxels, a respective likelihood of an obstacle occupation; and
establishing constraints to a presence of an obstacle.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features. In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track an obstacle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels, which would fulfill the claimed “splitting.” Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle and its interaction with an obstacle. The tracked vehicle and its interaction with an obstacle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 12, Pronovost cites except:
The method of claim 8, wherein said determining the safety occupancy comprises:
splitting, via the processor, a region around the autonomous vehicle into a plurality of voxels;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
calculating, via the processor and for each voxel of the plurality of voxels, a respective likelihood of an obstacle occupation;
and establishing, via the processor, constraints to a presence of an obstacle.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features. In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track an obstacle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels, which would fulfill the claimed “splitting.” Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle and its interaction with an obstacle. The tracked vehicle and its interaction with an obstacle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 16, Pronovost cites except as underlined:
The non-transitory, computer-readable media of claim 15, wherein the operations further comprise:
determining, based on the perception sensor input data and the ground-truth information input data, a general occupancy of the at least one obstacle within the area around the autonomous vehicle by:
splitting a region around the autonomous vehicle into a plurality of voxels;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
and
calculating a respective likelihood of the at least one obstacle occupying each voxel of the plurality of voxels;
determining based on the general occupancy of the at least one obstacle and the safety occupancy of the at least one obstacle, a safety-aware occupancy; and outputting the safety-aware occupancy signal based on the safety-aware occupancy.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features. In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track a vehicle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels, which would fulfill the claimed “splitting. Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle. The tracked vehicle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 18, Pronovost cites except as underlined:
The non-transitory, computer-readable media of claim 15, wherein determining the safety occupancy comprises:
splitting a region around the autonomous vehicle into a plurality of voxels;
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
calculating, for each voxel of the plurality of voxels, a respective likelihood of an obstacle occupation;
and establishing constraints to a presence of an obstacle.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features. In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track a vehicle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels, which would fulfill the claimed “splitting. Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle. The tracked vehicle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 21, Pronovost and Canady cites except as underlined:
The perception sensor configuration optimization system of claim 2, wherein said processor is further configured to execute the instructions stored in said memory to additionally cause said perception sensor configuration optimization system to receive second perception sensor input data of the autonomous vehicle, wherein the
second perception sensor input data corresponds to respective second measured location, orientation, and type of the plurality of perception sensors of the autonomous vehicle;
receive second ground-truth information input data of the autonomous vehicle, wherein the second ground-truth information input data corresponds to respective second ideal simulated location, ideal orientation, and ideal type of the plurality of perception sensors of the autonomous vehicle;
determine, based on the second perception sensor input data and the second ground-truth information input data, a second safety occupancy of at the least one obstacle within the area around the autonomous vehicle; and
output a second safety-aware occupancy signal based on the second safety occupancy.
In the rejection of claim 1, Pronovost and Canady cited an embodiment wherein the following took place:
“receive perception sensor input data of an autonomous vehicle, wherein the perception sensor input data corresponds to respective measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle;
receive ground-truth information input data of the autonomous vehicle, wherein the ground-truth information input data corresponds to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
determine, based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
and output a safety-aware occupancy signal based on the safety occupancy.”
However, neither Pronovost nor Canady discloses “second perception sensor input data second perception sensor input data” nor “second ground-truth information input data” to “output a second safety-aware occupancy signal.”
However, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady a further embodiment wherein the excepted claim limitations are provided. While neither specifically discloses the excepted “second versions” of the claimed first limitations, it can be expected, based on the operation of an autonomous vehicle, that with each continued travel to a different location, the embodiment would operate in such a manner as to input new information wherein the perception input data and ground-truth information would be different on a subsequent trip such that the claimed invention is realized. One of ordinary skill, based on the known operation of an embodiment in one setting, would likely arrive at such a conclusion for any other trip beyond the first initial trip.
Claims 3, 9, 10, 17, and 22 are rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 in view of Canady et al., U.S. 2021/0197859 and Roychowdhury et al., U.S. 2022/0044034 (hereinafter 034) and Refaat et al., U.S. 2020/0150665.
On claim 3, Pronovost cites except as underlined:
The perception sensor configuration optimization system of claim 2, wherein said processor is further configured to execute the instructions stored in said memory to cause said perception sensor configuration optimization system to output, to a display, the safety-aware occupancy signal to display a safety-aware occupancy grid illustrating perception coverage around the autonomous vehicle based on the safety-aware occupancy as an array representing the area around the autonomous vehicle.
On claim 9, Pronovost cites except as underlined:
The method of claim 8, further comprising:
determining, via the processor, based on the perception sensor input data and the ground-truth information input data, a general occupancy of the at least one obstacle within the area around the autonomous vehicle by:
splitting, via the processor, a region around the autonomous vehicle into a plurality of voxels; and
figure 1 and [0036]
[0036] In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like. In various examples, the classification of the object may be used by a model to determine object characteristics (e.g., maximum speed, acceleration, maneuverability, etc.). In this way, potential trajectories by an object may be considered based on characteristics of the object (e.g., how the object may potentially move in the environment). As depicted in FIG. 1, the example environment 100 includes a crosswalk 112.
calculating, via the processor, a respective likelihood of the at least one obstacle occupying each voxel of the plurality of voxels;
determining, via the processor, based on the general occupancy of the at least one obstacle and the safety occupancy of the at least one obstacle, a safety-aware occupancy; and outputting, via the processor, the safety-aware occupancy signal based on the safety-aware occupancy.
Pronovost cites:
[0080] The memory 418 can further include one or more maps 438 that can be used by the vehicle 402 to navigate within the environment. For the purpose of this discussion, a map can be any number of data structures modeled in two dimensions, three dimensions, or N-dimensions that are capable of providing information about an environment, such as, but not limited to, topologies (such as intersections), streets, mountain ranges, roads, terrain, and the environment in general. In some instances, a map can include, but is not limited to: covariance data (e.g., represented in a multi-resolution voxel space), texture information (e.g., color information (e.g., RGB color information,
And
[0030] The techniques discussed herein may improve a functioning of a vehicle computing system in a number of ways. The vehicle computing system may determine an action for the autonomous vehicle to take based on a determined trajectory of the object represented by data. In some examples, using the trajectory prediction techniques described herein, a model may output object trajectories and associated probabilities that improve safe operation of the vehicle by accurately characterizing motion of the object with greater detail as compared to previous models.
Pronovost doesn’t specifically disclose the excepted claim features.
In the same art of vehicle monitoring, 034 cites:
[0024] LiDAR sensor 102 includes any available sensing technology and is capable of providing three-dimensional data in any format including point clouds—which may alternatively be referred to as “point cloud data” or “point cloud data sets” in the present disclosure. LiDAR sensor 102 may produce LiDAR sensor data including point cloud data in the format of (x,y,z,r) where x,y,z are 3-d coordinates and r is reflectivity (the intensity of the reflected laser light). LiDAR sensor 102 may generate point clouds to represent a region in three-dimensional space where each three-dimensional data cube (e.g., voxel) maps to a two-dimensional area in a camera image of the above-mentioned image data.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady the vehicle tracking feature of 034 such that the claimed invention is realized.
Pronovost, while disclosing the use of “voxels” in its mapping embodiment, Pronovost doesn’t disclose using voxels to track a vehicle. 034, on the other hand, uses LiDAR sensors as a means to track objects in a region defined by voxels, which would fulfill the claimed “splitting.” Returning to Pronovost’s embodiment, that system includes trajectory prediction techniques to determine the trajectory of a tracked vehicle. The tracked vehicle is likely tracked using the voxels disclosed in 034 wherein the “populate voxel’s” trajectory is predicted in the manner previously described in Pronovost.
One of ordinary skill would have included into Pronovost’s embodiment the voxels disclosed in 034 using the tracking features disclosed in Pronovost and the results of the modification, using these known elements, would have produced an embodiment meeting the claimed invention.
On claim 10, Pronovost cites except as underlined:
The method of claim 9, further comprising displaying, via a display, a safety-aware occupancy grid illustrating perception coverage around the autonomous vehicle based on the safety-aware occupancy as an array representing the area around the autonomous vehicle.
Pronovost, figure 2, discloses a vehicle 102 providing a trajectory 122. The figure suggest a display used to determine likely hazards to vehicle 102. Pronovost, [0120] also discloses a display. However, Pronovost doesn’t disclose the excepted claim limitations.
In the same art of vehicle tracking Rafaat, figures 6A, 6B, 7, and [0057] discloses:
This verification of predicted trajectories may be performed for all road users or only for certain types of road users with or without certain characteristics. For instance, only predicted trajectories of other vehicles may be verified and/or vehicles that are moving at a certain predetermined speed. For example, it may be especially useful to verify trajectories of other vehicles that are moving at or below 10 miles per hour as these vehicles may tend to be more unpredictable, and in addition, the grid-based approach is especially useful for short term predictions. In other words, for faster moving road users such as vehicles on a highway, the grid-based approach is less useful as the vehicle can quickly go beyond a region of the heat map.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost’s vehicle trajectory determination system the grid tracking system of Rafaat such that the claimed invention is realized. Rafaat discloses a known way to track a vehicle using a grid display scheme and one of ordinary skill would have incorporated such a feature into the display of Pronovost and the results of using these known techniques would have realized an embodiment meeting the claimed invention.
On claim 17, Pronovost cites except as underlined:
The non-transitory, computer-readable media claim 16, wherein the operations further comprise displaying, via a display, a safety-aware occupancy grid illustrating perception coverage around the autonomous vehicle based on the safety-aware occupancy as an array representing the area around the autonomous vehicle.
Pronovost, figure 2, discloses a vehicle 102 providing a trajectory 122. The figure suggest a display used to determine likely hazards to vehicle 102. Pronovost, [0120] also discloses a display. However, Pronovost doesn’t disclose the excepted claim limitations.
In the same art of vehicle tracking Rafaat, figures 6A, 6B, 7, and [0057] discloses:
This verification of predicted trajectories may be performed for all road users or only for certain types of road users with or without certain characteristics. For instance, only predicted trajectories of other vehicles may be verified and/or vehicles that are moving at a certain predetermined speed. For example, it may be especially useful to verify trajectories of other vehicles that are moving at or below 10 miles per hour as these vehicles may tend to be more unpredictable, and in addition, the grid-based approach is especially useful for short term predictions. In other words, for faster moving road users such as vehicles on a highway, the grid-based approach is less useful as the vehicle can quickly go beyond a region of the heat map.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost’s vehicle trajectory determination system the grid tracking system of Rafaat such that the claimed invention is realized. Rafaat discloses a known way to track a vehicle using a grid display scheme and one of ordinary skill would have incorporated such a feature into the display of Pronovost and the results of using these known techniques would have realized an embodiment meeting the claimed invention.
On claim 22, Pronovost and Canady cites except as underlined:
The method of claim 9, further comprising:
receiving second perception sensor input data of the autonomous vehicle, wherein the second perception sensor input data corresponds to respective second measured location,
orientation, and type of the plurality of perception sensors of the autonomous vehicle;
receiving second ground-truth information input data of the autonomous vehicle, wherein the second ground-truth information input data corresponds to respective second ideal simulated location, ideal orientation, and ideal type of the plurality of perception sensors of the autonomous vehicle determining, via the processor and based on the second perception sensor input data and the second ground-truth information input data, a second safety occupancy of at the least one obstacle within the area around the autonomous vehicle; and
outputting, via the processor, a second safety-aware occupancy signal based on the second safety occupancy.
In the rejection of claim 1, Pronovost and Canady cited an embodiment wherein the following took place:
“receive perception sensor input data of an autonomous vehicle, wherein the perception sensor input data corresponds to respective measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle;
receive ground-truth information input data of the autonomous vehicle, wherein the ground-truth information input data corresponds to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
determine, based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
and output a safety-aware occupancy signal based on the safety occupancy.”
However, neither Pronovost nor Canady discloses “second perception sensor input data second perception sensor input data” nor “second ground-truth information input data” to “output a second safety-aware occupancy signal.”
However, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost and Canady a further embodiment wherein the excepted claim limitations are provided. While neither specifically discloses the excepted “second versions” of the claimed first limitations, it can be expected, based on the operation of an autonomous vehicle, that with each continued travel to a different location, the embodiment would operate in such a manner as to input new information wherein the perception input data and ground-truth information would be different on a subsequent trip such that the claimed invention is realized. One of ordinary skill, based on the known operation of an embodiment in one setting, would likely arrive at such a conclusion for any other trip beyond the first initial trip.
Claim 20 is rejected under 35 USC 103 as being unpatentable over Pronovost, U.S. 2024/0101150 in view of Canady et al., U.S. 2021/0197859 and Khare et al., WO 2023/086669.
On claim 20, Pronovost cites except as underlined:
The non-transitory, computer-readable media of claim 15, wherein the operations further comprise enabling, via a user interface, a user to modify the perception sensor input data.
Pronovost cites:
[0035] In various examples, a vehicle computing device associated with the vehicle 102 may be configured to detect one or more objects (e.g., objects 108 and 110) in the environment 100, such as via a perception component. In some examples, the vehicle computing device may detect the objects, based on sensor data received from one or more sensors. In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras, microphones, inertial sensors (e.g., inertial measurement units, accelerometers, gyros, etc.), global positioning satellite (GPS) sensors, and the like. In various examples
Pronovost doesn’t cite the excepted claim limitations.
In the related art of sensor systems, Khare cites:
[0004] In at least one aspect, a system for intelligently selecting sensors and their associated operating parameters is provided. The system includes one or more source sensors that gather animal data from at least one targeted individual and a collecting computing device in electrical communication with the one or more source sensors. The collecting computing device is configured to utilize one or more Artificial Intelligence techniques to: (1) intelligently gather the animal data from the one or more source sensors; (2) create and/or modify and/or access one or more commands that provide one or more instructions to the one or more source sensors to perform one or more actions; and (3) intelligently transmit the one or more commands either directly (e.g., directly to the one or more source sensors) or indirectly (e.g., via another one or more sensors; via another one or more computing devices in communication with the one or more source sensors) to the one or more source sensors to create or modify one or more sensor operating parameters.
It would have been obvious to one of ordinary skill before the effective filing date of the claimed invention to include into Pronovost the sensor changing feature disclosed in Khare such that the claimed invention is realized. Khare discloses a known feature of modifying sensor parameters. One of ordinary skill would have included this feature to adjust for changing user requirements for the vehicle’s sensing system.
Response to Arguments
Claim 1 claims:
A perception sensor configuration optimization system comprising:
and
a processor configured to execute the instructions stored in said memory
to cause said perception sensor configuration optimization system to:
receive perception sensor input data of an autonomous vehicle, wherein the perception sensor input data corresponds to respective measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle;
receive ground-truth information input data of the autonomous vehicle, wherein the ground-truth information input data corresponds to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle;
determine, based on the perception sensor input data and the ground-truth information input data, a safety occupancy of at least one obstacle within an area around the autonomous vehicle;
However, the applicant’s rebuttal asserts:
“Pronovost does not disclose perception sensor input data corresponding to measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle, as recited in independent claim 1.”
As indicated in page 3 if the last Office Action of Pronovost as, “perception sensor data” is disclosed in figure 8 and [0035]. Per the examiner’s comment, “the cited GPS and ultrasonic, camera, microphone, radar, gyros, and the like are analogous to the claimed "plurality of perception sensors.” The examiner’s observation comports with the applicant’s own specification, which cites:
[0001] Autonomous vehicles (AVs) may require perception of the world in order to make intelligent control decisions. Generally, perception systems are composed of a heterogeneous set of sensors, including, e.g., cameras, ultrasonic/sonar, radar, and LiDAR.
Thus, the examiner has carefully reviewed the rebuttal and finds the applicant’s argument unpersuasive.
The rebuttal goes on to state:
“Further, Pronovost does not disclose ground-truth information input data corresponding to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle, as recited in independent claim 1. In other words, Pronovost's "ground truth" is about how objects actually moved (positions, trajectories, etc.) and labels used to train transformer model 104. On the contrary, the ground-truth information input data recited in independent claim 1 is about ideal perception sensor data as detected by an ideal sensor configuration in a simulated setting, against which the actual measured perception sensor input data from a sensor configuration is compared.”
Pronovost cites “ground truth information” as
[0130] In some instances, the training component 848 may be executed by the processor(s) 836 to train a machine learning model based on training data. The training data may include a wide variety of data, such as sensor data, audio data, image data, map data, inertia data, vehicle state data, historical data (log data), or a combination thereof, that is associated with a value (e.g., a desired classification, inference, prediction, etc.). Such values may generally be referred to as a “ground truth.” To illustrate, the training data may be used for determining risk associated with occluded regions and, as such, may include data representing an environment that is captured by an autonomous vehicle and that is associated with one or more classifications or determinations. In some examples, such a classification may be based on user input (e.g., user input indicating that the data depicts a specific risk) or may be based on the output of another machine learned model. In some examples, such labeled classifications (or more generally, the labeled output associated with training data) may be referred to as ground truth.
The applicant’s specification discloses:
[0007] Another aspect of the present disclosure is drawn to a method including: receiving perception sensor input data of an autonomous vehicle, the perception sensor input data corresponding to respective measured location, orientation, and type of each of a plurality of perception sensors of the autonomous vehicle; receiving ground-truth information input data of the autonomous vehicle, the ground-truth information input data corresponding to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle; determining, via a processor configured to execute instructions stored in a memory, based on the perception sensor input data and the ground-truth information input data
It is clearly Pronovost’s definition where “a desired classification, inference, prediction, etc.” is the same as the applicant’s specification “receiving ground-truth information input data of the autonomous vehicle, the ground-truth information input data corresponding to respective ideal simulated location, ideal orientation.” The applicant’s “ideal” is a judgment call on what data is best or “desired,” as what is shown in Provonost’s “desired classification, inference, prediction, etc. For this reason, the applicant’s arguments regarding this issue is also unpersuasive.
The applicant’s argument regarding the rejection of claim 4 has also been carefully weighed. The rebuttal states:
“Read-Coup fails to disclose, at least: perception sensor input data corresponding to measured location, orientation, and type of a plurality of perception sensors of the autonomous vehicle; and ground-truth information input data corresponding to respective ideal simulated location, ideal orientation, and ideal type of a plurality of perception sensors of the autonomous vehicle, as required in independent claim 1.”
Because the rebuttal refers back the applicant’s definition of “ground-truth,” the applicant’s argument is also unpersuasive for the same reasons disclosed in the reasons for rejected claim 1 above. Also, the applicant’s arguments regarding the rejection of claims 1,-3, 5,6, 8-13, and 15-20 have also been carefully reviewed and are rejected for the same reasons articulated under item 10 above.
Remarks
The applicant is encouraged to review figure 2A of the applicant’s specification as an embodiment more accurately reflecting the applicant’s invention. “Benchmark data” is part of the embodiment.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAL EUSTAQUIO whose telephone number is (571)270-7229. The examiner can normally be reached on 8am-5pm. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Brian Zimmerman, can be reached at (571) 272-3059. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application lnformation Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAlR only. For more information about the PAlR system, see http:/lpair-direct.uspto.gov. Should you have questions on access to the Private PAlR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-91 99 (IN USA OR CANADA) or 571-272-1000.
/CAL J EUSTAQUIO/Examiner, Art Unit 2686
/BRIAN A ZIMMERMAN/Supervisory Patent Examiner, Art Unit 2686
Prosecution Insights