DETAILED ACTION
Status of Claims
The following is a non-final, first office action in response to the communication filed 9/28/2023
Claims 1-20 have been canceled.
Claims 21-40 have been added, are currently pending, and have been examined.
Priority
The applicant’s claim for benefit of Provisional Patent Application Serial No. 62/936,438 filed 11/16/2019 has been received and acknowledged.
Information Disclosure Statement
Information Disclosure Statement received 8/31/2023 has been reviewed and considered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 27 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 27 recites “processing the second motion forecast data with a machine-learned prediction model to generate respective trajectories of the plurality of actors for the first time step and respective projected trajectories of the plurality of actors for the second time step.” However, the specification only provides support for the second motion forecast data being used to generate trajectories for the plurality of actors for the second time step. Therefore, it is unclear how the second motion forecast is being used to generate trajectories for the first time step.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 21-40 is/are rejected under 35 U.S.C. 102(a)(2)as being anticipated by Kiiski (US 12128887).
Regarding claim 21, Kiiski discloses:
21. (New) A computer-implemented method, the method comprising:
(74) col. 20, ll. 15-20 FIG. 3 is a block diagram of an example system 300 for implementing the techniques described herein. In at least one example, the system 300 may include a vehicle 302, such as vehicle 102.
(75) col. 20, ll. 20-25, The vehicle 302 may include a vehicle computing device 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312, and one or more drive modules 314.
(76) col. 20, ll. 25-50, The vehicle computing device 304 may include one or more processors 316 and memory 318 communicatively coupled with the one or more processors 316.
accessing, for a first iteration corresponding with a first time step (T=0), a first relative location embedding that describes respective relative locations of a plurality of actors (Fig. 1, objects 104, location of an object relative to the vehicle) with respect to an autonomous vehicle (Fig. 1, autonomous vehicle 102);
(28) col. 7, l. 60 - col. 8, l. 10, FIG. 1 is an illustration of an autonomous vehicle 102 (vehicle 102) in an environment 100, wherein a dynamic object relevance system may determine whether objects in the environment may be relevant to the vehicle 102. A vehicle computing device may perform the dynamic object relevance system of the vehicle 102.
(29) col 8, ll. 10-25, In various examples, the vehicle computing device may be configured to detect one or more dynamic objects 104 (e.g., objects 104) in the environment 100, such as via a perception system. The vehicle computing system may detect the object(s) 104 based on sensor data received from one or more sensors. In some examples, the sensor(s) may include sensors mounted on the vehicle 102, such as, for examples, cameras, motion detectors, lidar, radar, etc. In some examples, the sensor(s) may include one or more remote sensors, such as, for example sensors mounted on another autonomous vehicle, and/or sensors 106 mounted in the environment 100.
(43) col 11, ll. 40-55, Additionally, as an illustrative example, the vehicle computing system may determine that objects 104(1) and 104(2) may occupy the same space as the vehicle 102 (e.g., determination of relevance to the vehicle 102) during the time period. Based on an initial relevance determination (at time T=0), the vehicle computing device may store data associated with the relevant objects 104(1) and 104(2)
processing the first relative location embedding (location of autonomous vehicle 102 and relative location of objects 104 at T=0) by a machine-learned interaction transformer model (machine learning techniques) to generate first motion forecast data describing movement of the plurality of actors at the first time step (T=0, predicted trajectory from an initial position);
(11) col 2, ll. 39-68, The vehicle computing system may be configured to determine an initial position of each detected object. In various examples, the vehicle computing system may determine one or more predicted trajectories associated with each detected object, such as from an initial position associated therewith. In some examples, the one or more predicted trajectories may be determined based on the sensor data. Each predicted trajectory may represent a potential path that the detected object may travel through the environment. … In some examples, the one or more predicted trajectories may be determined utilizing machine learning techniques.
(31) col 8, ll. 40-65, In various examples, the vehicle computing system may receive the sensor data and may determine a type of object 104 (e.g., classify the type of object), such as, for example, whether the object 104 is a car, such as objects 104(1) and 104(2), a truck, motorcycle, moped, bicyclist, pedestrian, such as object 104(3), or the like. In various examples, the vehicle computing system may determine one or more predicted object trajectories 108 (trajectories 108) based on the sensor data and/or the type of object 104, such as trajectories 108(1), 108(2), and 108(3) associated with objects 104(1), 104(2), and 104(3), respectively. In some examples, the trajectories 108 may include any number of potential paths in which the object 104 may travel from a current position (e.g., at the time of perception) and/or based on a direction of travel. In some examples, a potential path for an object 104 may include remaining stationary, such as object 104(2) stopped at an intersection 110 in the environment 100. In such an example, the corresponding trajectory 108, such as 108(2), may represent little to no motion. In some examples, the number of trajectories may vary depending on a variety of factors, such as the classification of the object (e.g., type of object), other stationary and/or dynamic objects, drivable surfaces, etc.
(33) col 9, ll. 15-30, In various examples, the trajectories 108 may be determined based on a predicted motion of the object as determined by a prediction system of the vehicle.
(35) col 9, ll. 40-55, In various examples, the trajectories 108 may be determined utilizing one or more machine learning algorithms. In such examples, the vehicle computing system, such as via a prediction system, may receive the sensor data associated with the object 104 and predict a behavior of the object 104 based on the sensor data.
(51) col 13, ll. 25-45, The estimated state(s) may each represent an estimated position of the vehicle 114 and an estimated position of one or more relevant objects 116. The estimated position of the vehicle 114 and the estimated position of relevant objects 116 may be based on the movement of the vehicle 102 (e.g., vehicle trajectory 112 based on an action, etc.) and the movement of the relevant object 104 (e.g., trajectory 108), respectively. The illustrative example depicted in FIG. 1 depicts a plurality of estimated positions corresponding to a plurality of estimated states. For example, estimated positions of the vehicle 114(1), 114(2), 114(3), 114(4), and 114(5) correspond to estimated states 1 through 5, and estimated positions of the relevant object 116(1), 116(2), 116(3), and 116(4) correspond to estimated states 1 through 4.
See also (80) col 21, ll. 55-65; (117) col 30, ll. 34-65
generating, based on the first motion forecast data, a second relative location embedding for a second time step (determining an object occupies the same space as the vehicle over a pre-determined period of time, future environment, e.g., projecting the vehicle and relevant object(s) forward in the environment for the period of time (e.g., 5 seconds, 8 seconds, 12 seconds, etc.);
(12) col 3, ll. 1-15, In various examples, the vehicle computing system may determine an initial relevance of each detected object to the vehicle. The initial relevance may be based on a determination that the detected object may occupy the same space as the vehicle over a pre-determined period of time (e.g., time period) in the future (e.g., 4 seconds, 8 seconds, 10 seconds, 1 minute, etc.). In at least one example, the pre-determined period of time may be 8 seconds. The initial relevance may be based on geometric data (e.g., distance, angle, direction of travel, velocity, acceleration, trajectory, etc.), semantic data (e.g., type, class, etc. associated with the object, defined location of the object (e.g., on road, off road, on sidewalk, etc.)), or a combination thereof.
(14) col 3, ll. 55-65, In at least some examples, the system may continuously make such determinations at additional simulated time steps. In such examples, a set of future state of the environment may be predicted and, at each state a determination may be made whether or not to include one or more of the entities in subsequent time steps. In at least some such examples, such decisions may be made in accordance with similar techniques as defined above.
(17) col 4, ll. 15-50, For each applicable action and sub-action, the vehicle computing system may simulate future states (e.g., estimated states) by projecting the vehicle and relevant object(s) forward in the environment for the period of time (e.g., 5 seconds, 8 seconds, 12 seconds, etc.).
(36) col 9, l. 50 - col. 10, l. 5, In various examples, the vehicle computing system may determine an initial relevance of each detected object 104 in the environment 100 to the vehicle 102. The initial relevance may be based on a determination that a detected object 104 may occupy the same space as the vehicle 102 at an initial time (e.g., current time) and/or over a time period (e.g., 4 seconds, 8 seconds, 10 seconds, 1 minute, etc.). In some example, the time period may include a predetermined amount of time. In at least one example, the time period may be eight (8) seconds. In some examples, the time period may be based on one or more operating factors associated with the vehicle 102, such as speed, environmental factors (e.g., congested area (e.g., a number of detected objects 104), size of a road (e.g., number of lanes, width of lanes, etc.), proximity to intersection, proximity to a playground, school zone, speed limit associated with the road, etc.), weather considerations (e.g., rain, sleet, snow; ice, wind, etc.), timing factors (e.g., a time of day, day of the year, etc.), or the like.
processing the second relative location embedding (determining an object occupies the same space as the vehicle over a pre-determined period of time, future environment, 114(2)… 114(5), 116(2)…116(4)) by the machine-learned interaction transformer model to generate second motion forecast data for the plurality of actors at the second time step (determine if the relevant object will remain relevant to the vehicle based on rule(s) - relative distances (e.g., threshold distance from vehicle 102), relative locations (e.g., in front, behind, abeam the vehicle 102, etc.), comparison of trajectory 108 to vehicle trajectory 112 (e.g., speeds, and/or directions of travel between the vehicle 102 and the relevant object 104); estimated states 2…. X); and
(17) col 4, ll. 15-50, The vehicle computing system may project the relevant object(s) (e.g., estimate future positions of the object(s)) forward based on a predicted trajectory associated therewith. The vehicle computing system may project the vehicle (e.g., estimate future positions of the vehicle) forward based on a vehicle trajectory associated with an action. The estimated state(s) may represent an estimated position (e.g., estimated location) of the vehicle and an estimated position of the relevant object(s) at a time in the future. In some examples, the vehicle computing system may determine relative data between the vehicle and the relevant object(s) in the estimated state(s). In such examples, the relative data may include distances, locations, speeds, directions of travel, and/or other factors between the vehicle and the object
(19) col 5, ll. 15-45, In some examples, the vehicle computing system may apply a set of rules based on the geometric data and/or the semantic data to determine if the relevant object will remain relevant to the vehicle. The set of rules may include relative distances (e.g., threshold distance from vehicle), relative locations (e.g., in front, behind, etc.), speeds, and/or directions of travel between the vehicle and the relevant object. …
(21) col 6, ll. 1-25, In various examples, the vehicle computing system may perform the relevance verification(s) randomly throughout the period of time (e.g., at 2.5 seconds, at 7.3 seconds, with the 52.sup.nd estimated state of a set of estimated states, etc.). In some examples, the vehicle computing system may perform the relevance verification(s) periodically throughout the period of time (e.g., every 2 seconds, 4 seconds, 5 seconds, every 10 estimated states, 20 estimated states, etc.)
(51) col 13, ll. 25-45, The estimated state(s) may each represent an estimated position of the vehicle 114 and an estimated position of one or more relevant objects 116. The estimated position of the vehicle 114 and the estimated position of relevant objects 116 may be based on the movement of the vehicle 102 (e.g., vehicle trajectory 112 based on an action, etc.) and the movement of the relevant object 104 (e.g., trajectory 108), respectively. The illustrative example depicted in FIG. 1 depicts a plurality of estimated positions corresponding to a plurality of estimated states. For example, estimated positions of the vehicle 114(1), 114(2), 114(3), 114(4), and 114(5) correspond to estimated states 1 through 5, and estimated positions of the relevant object 116(1), 116(2), 116(3), and 116(4) correspond to estimated states 1 through 4.
(53) col 13, l. 55- col. 14, l. 15, In various examples, the vehicle computing system may perform one or more relevance verifications during a set of estimated states. The relevance verification(s) may include a determination that an object previously determined to be relevant to the vehicle 102 remains relevant to the vehicle (e.g., may occupy the same space as the vehicle 102 for the remainder of the set of estimated states). In various examples, …
(54) col 14, ll. 15-40, In various examples, the vehicle computing system may utilize machine learning techniques to verify relevance of a relevant object 104. In such examples, the machine learning algorithms may be trained to determine, based on geometric and/or semantic data, that an object 104 is or is irrelevant to the vehicle 102 at a particular time during the set of estimated states (e.g., time period). In some examples, the vehicle computing system may apply one or more rules (e.g., set of rules) based on the geometric data and/or the semantic data to determine if the relevant object 104 remains relevant to the vehicle 102. In some examples, the rule(s) may be generated utilizing machine learning techniques. In such examples, machine learning algorithms may be trained to generate one or more rules for determining relevance of an object. As discussed above with regard to the initial relevance determination, the rule(s) may be based on a classification associated with the relevant object 104. In some examples, the rule(s) may be based on predictability of the classification associated with the relevant object 104. The rule(s) may include relative distances (e.g., threshold distance from vehicle 102), relative locations (e.g., in front, behind, abeam the vehicle 102, etc.), comparison of trajectory 108 to vehicle trajectory 112 (e.g., speeds, and/or directions of travel between the vehicle 102 and the relevant object 104).
See also (80) col 21, ll. 55-65, , (117) col 30, ll. 34-65,
controlling motion of the autonomous vehicle based on at least one of the first motion forecast data or the second motion forecast data.
(15) col 3, ll. 60 - col. 4, l. 10, In various examples, the vehicle computing system may be configured to determine actions to take while operating (e.g., control planning) based on the relevant objects. The actions may include a reference action (e.g., one of a group of maneuvers the vehicle is configured to perform in reaction to a dynamic operating environment) such as a right lane change, a left lane change, staying in a lane, going around an obstacle (e.g., double-parked vehicle, traffic cones, etc.), or the like. The actions may additionally include sub-actions, such as speed variations (e.g., maintain velocity, accelerate, decelerate, etc.), positional variations (e.g., changing a position in a lane), or the like. For example, an action may include staying in a lane (action) and adjusting a position of the vehicle in the lane from a centered position to operating on a left side of the lane (sub-action).
(44) col 11, ll. 55 - col. 12, l. 5, In various examples, the vehicle computing system may be configured to determine actions to take while operating (e.g., control planning) based on the relevant objects 104(1) and 104(2). The actions may include reference actions (e.g., one of a group of maneuvers the vehicle is configured to perform in reaction to a dynamic operating environment) such as a right lane change, a left lane change, staying in a lane, going around an obstacle (e.g., double-parked vehicle, traffic cones, etc.), or the like. The actions may additionally include sub-actions, such as speed variations (e.g., maintain velocity, accelerate, decelerate, etc.), positional variations (e.g., changing a position in a lane), or the like. For example, an action may include staying in a lane (action) and adjusting a position of the vehicle in the lane from a centered position to operating on a left side of the lane (sub action).
(118) col 30, ll. 65 - col. 31, l. 15, At operation 404, the process may include determining an action that the vehicle may take while operating in the environment. In some examples, the action may be based on the detected object and/or the object trajectory associated therewith. The action may include a reference action such as a right lane change, a left lane change, staying in a lane, going around an obstacle (e.g., double-parked vehicle, traffic cones, etc.), or the like. The action may additionally include one or more sub actions, such as speed control (e.g., maintain velocity, accelerate, decelerate, etc.), positional variations (e.g., changing a position in a lane), or the like. For example, an action may include staying in a lane (action) and adjusting a position of the vehicle in the lane from a centered position to operating on a left side of the lane (sub action).
Regarding claim 22, Kiiski discloses the limitations of claim 21, and further discloses: processing the first relative location embedding by a machine-learned interaction transformer model to generate first motion forecast data describing movement of the plurality of actors at the first time step (see analogous claim 21 limitation mapping above) further comprises: processing the first relative location embedding (Fig. 1, objects 104, location of an object relative to the vehicle 102) and a first input sequence describing object detection data corresponding with the first time step (process of identifying relevant objects) by the machine-learned interaction transformer model (machine-learning technique) to generate the first motion forecast data describing movement of the plurality of actors at the first time step (movement/trajectories 108(1), 108(2), and 108(3) associated with objects 104(1), 104(2), and 104(3), respectively).
(11) col 2, ll. 39-68, The vehicle computing system may be configured to determine an initial position of each detected object. In various examples, the vehicle computing system may determine one or more predicted trajectories associated with each detected object, such as from an initial position associated therewith. In some examples, the one or more predicted trajectories may be determined based on the sensor data. Each predicted trajectory may represent a potential path that the detected object may travel through the environment. The one or more predicted trajectories may be based on passive prediction (e.g., independent of an action the vehicle and/or another object takes in the environment, substantially no reaction to the action of the vehicle and/or other objects, etc.), active prediction (e.g., based on a reaction to an action of the vehicle and/or another object in the environment), or a combination thereof. … In some examples, the one or more predicted trajectories may be determined utilizing machine learning techniques.
(31) col 8, ll. 40-65, In various examples, the vehicle computing system may receive the sensor data and may determine a type of object 104 (e.g., classify the type of object), such as, for example, whether the object 104 is a car, such as objects 104(1) and 104(2), a truck, motorcycle, moped, bicyclist, pedestrian, such as object 104(3), or the like. In various examples, the vehicle computing system may determine one or more predicted object trajectories 108 (trajectories 108) based on the sensor data and/or the type of object 104, such as trajectories 108(1), 108(2), and 108(3) associated with objects 104(1), 104(2), and 104(3), respectively. In some examples, the trajectories 108 may include any number of potential paths in which the object 104 may travel from a current position (e.g., at the time of perception) and/or based on a direction of travel. In some examples, a potential path for an object 104 may include remaining stationary, such as object 104(2) stopped at an intersection 110 in the environment 100. In such an example, the corresponding trajectory 108, such as 108(2), may represent little to no motion. In some examples, the number of trajectories may vary depending on a variety of factors, such as the classification of the object (e.g., type of object), other stationary and/or dynamic objects, drivable surfaces, etc.
(37) col 10, ll. 5-30, The initial relevance may be based on geometric data (e.g., distance, angle, direction of travel, velocity, acceleration, trajectory, etc.), semantic data (e.g., classification of the object), or a combination thereof. In various examples, the vehicle computing system may utilize machine learning techniques to determine an initial relevance of a detected object. In such examples, the machine learning algorithms may be trained to determine, based on geometric and/or semantic data, that an object 104 is or is irrelevant to the vehicle 102. In some examples, the vehicle computing system may utilize one or more rules to determine an initial relevance of a detected object 104. In some examples, the rule(s) applied to a particular object 104 may be based on a semantic classification of the object 104. For example, a first set of rule(s) may be applied to a pedestrian and a second set of rule(s) may be applied to a bus to determine whether the pedestrian and/or the bus are relevant to the vehicle 104. In various examples, the rule(s) may be determined using machine learning techniques. In such examples, machine learning algorithms may be trained to generate rule(s) (e.g., threshold distances, relative locations, etc.) for determining relevance of an object 104.
(43) col 11, ll. 40-55, Additionally, as an illustrative example, the vehicle computing system may determine that objects 104(1) and 104(2) may occupy the same space as the vehicle 102 (e.g., determination of relevance to the vehicle 102) during the time period. Based on an initial relevance determination (at time T=0), the vehicle computing device may store data associated with the relevant objects 104(1) and 104(2). The data may include a classification, a location, and/or trajectory data associated with the relevant objects 104(1) and 104(2). Though the illustration in FIG. 1 includes two relevant objects, the vehicle computing system may be configured to identify a greater or fewer number of relevant objects. The vehicle computing system may consider the data associated with the relevant objects 104(1) and 104(2) in control planning considerations.
(46) col 12, ll. 25-45, In various examples, for each applicable action and sub-action, the vehicle computing system may generate a set of estimated states of the vehicle and a relevant object 104, such as objects 104(1) and/or 104(2), into the future.
(50) col 13, ll. 20-30, In various examples, the vehicle computing system may utilize machine learning techniques to determine the movement of the relevant object 104 in the environment. In such examples, machine learning algorithms may be trained to determine movement of a relevant object 104 in an environment based on various input factors, such as environmental factors, weather factors, timing factors, known reactions to particular classes of objects 104 or vehicles in proximity, or the like.
See also (86) col 23, ll. 1-30, , (117) col 30, ll. 34-65, , (127) col 32, ll. 10-40,
Regarding claim 23, Kiiski discloses the limitations of claim 21, and further discloses: wherein processing the second relative location embedding by a machine-learned interaction transformer model to generate second motion forecast data describing movement of the plurality of actors at the second time step (see analogous claim 21 limitation mapping above) further comprises: processing the second relative location embedding (determining an object occupies the same space as the vehicle over a pre-determined period of time, future environment, 114(2)… 114(5), 116(2)…116(4)) and a second input sequence describing object detection data corresponding with the second time step (process of verifying relevant objects) by the machine-learned interaction transformer model to generate the second motion forecast data describing movement of the plurality of actors at the second time step (movement/trajectories 108(1), 108(2), and 108(3) associated with objects 104(1), 104(2), and 104(3), respectively).
(50) col 13, ll. 20-30, In various examples, the vehicle computing system may utilize machine learning techniques to determine the movement of the relevant object 104 in the environment. In such examples, machine learning algorithms may be trained to determine movement of a relevant object 104 in an environment based on various input factors, such as environmental factors, weather factors, timing factors, known reactions to particular classes of objects 104 or vehicles in proximity, or the like.
(53) col 13, l. 55- col. 14, l. 15, In various examples, the vehicle computing system may perform one or more relevance verifications during a set of estimated states. The relevance verification(s) may include a determination that an object previously determined to be relevant to the vehicle 102 remains relevant to the vehicle (e.g., may occupy the same space as the vehicle 102 for the remainder of the set of estimated states). In various examples, a relevance verification may determine whether an action and/or sub-action is independent of an object 104 (e.g., the object 104 has no bearing on the vehicle 102 performing the action) previously determined to be relevant. In some examples, the relevance verification(s) may be performed substantially concurrently with one or more estimated states of the set of estimated states, such as utilizing data associated with a respective estimated state. The relevance verification(s) may be based on geometric data (e.g., distance, angle, direction of travel, velocity, acceleration, trajectory, etc.), semantic data (e.g., classification of the object), or a combination thereof. In various examples, the vehicle computing system may compare an estimated position of the vehicle 114 in an estimated state to an estimated position of the object 116 in the estimated state. For example, for a fourth estimated state, the vehicle computing system may compare the estimated position of the vehicle 114(4) to the estimated position of the relevant object 116(4) to determine if the relevant object remains relevant to the vehicle.
(54) col 14, ll. 15-40, In various examples, the vehicle computing system may utilize machine learning techniques to verify relevance of a relevant object 104. In such examples, the machine learning algorithms may be trained to determine, based on geometric and/or semantic data, that an object 104 is or is irrelevant to the vehicle 102 at a particular time during the set of estimated states (e.g., time period). In some examples, the vehicle computing system may apply one or more rules (e.g., set of rules) based on the geometric data and/or the semantic data to determine if the relevant object 104 remains relevant to the vehicle 102. In some examples, the rule(s) may be generated utilizing machine learning techniques. In such examples, machine learning algorithms may be trained to generate one or more rules for determining relevance of an object. As discussed above with regard to the initial relevance determination, the rule(s) may be based on a classification associated with the relevant object 104. In some examples, the rule(s) may be based on predictability of the classification associated with the relevant object 104. The rule(s) may include relative distances (e.g., threshold distance from vehicle 102), relative locations (e.g., in front, behind, abeam the vehicle 102, etc.), comparison of trajectory 108 to vehicle trajectory 112 (e.g., speeds, and/or directions of travel between the vehicle 102 and the relevant object 104).
See also (92) col 24, ll. 35-65, , (117) col 30, ll. 34-65, , (127) col 32, ll. 10-40,
Regarding claim 24, Kiiski discloses the limitations of claim 21, and further discloses: wherein the second input sequence is based at least in part on the first motion forecast data.
(53) col 13, l. 55- col. 14, l. 15, In various examples, the vehicle computing system may perform one or more relevance verifications during a set of estimated states. The relevance verification(s) may include a determination that an object previously determined to be relevant to the vehicle 102 remains relevant to the vehicle (e.g., may occupy the same space as the vehicle 102 for the remainder of the set of estimated states). In various examples, a relevance verification may determine whether an action and/or sub-action is independent of an object 104 (e.g., the object 104 has no bearing on the vehicle 102 performing the action) previously determined to be relevant. In some examples, the relevance verification(s) may be performed substantially concurrently with one or more estimated states of the set of estimated states, such as utilizing data associated with a respective estimated state. The relevance verification(s) may be based on geometric data (e.g., distance, angle, direction of travel, velocity, acceleration, trajectory, etc.), semantic data (e.g., classification of the object), or a combination thereof. In various examples, the vehicle computing system may compare an estimated position of the vehicle 114 in an estimated state to an estimated position of the object 116 in the estimated state. For example, for a fourth estimated state, the vehicle computing system may compare the estimated position of the vehicle 114(4) to the estimated position of the relevant object 116(4) to determine if the relevant object remains relevant to the vehicle.
(54) col 14, ll. 15-40, In various examples, the vehicle computing system may utilize machine learning techniques to verify relevance of a relevant object 104. In such examples, the machine learning algorithms may be trained to determine, based on geometric and/or semantic data, that an object 104 is or is irrelevant to the vehicle 102 at a particular time during the set of estimated states (e.g., time period). In some examples, the vehicle computing system may apply one or more rules (e.g., set of rules) based on the geometric data and/or the semantic data to determine if the relevant object 104 remains relevant to the vehicle 102. In some examples, the rule(s) may be generated utilizing machine learning techniques. In such examples, machine learning algorithms may be trained to generate one or more rules for determining relevance of an object. As discussed above with regard to the initial relevance determination, the rule(s) may be based on a classification associated with the relevant object 104. In some examples, the rule(s) may be based on predictability of the classification associated with the relevant object 104. The rule(s) may include relative distances (e.g., threshold distance from vehicle 102), relative locations (e.g., in front, behind, abeam the vehicle 102, etc.), comparison of trajectory 108 to vehicle trajectory 112 (e.g., speeds, and/or directions of travel between the vehicle 102 and the relevant object 104).
Regarding claim 25, Kiiski discloses the limitations of claim 23, and further discloses: wherein the second input sequence further describes one or more features for the plurality of actors respectively at the first time step or the second time step, the one or more features comprising at least one of: a derivative of the respective relative location of a respective actor of the plurality of actors relative to the autonomous vehicle; a size of the respective actor; an orientation of the respective actor relative to the autonomous vehicle; or a center location of the respective actor.
(78) col 21, ll. 1-35, In some instances, the perception component 322 may include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception component 322 may provide processed sensor data that indicates a presence of an object (e.g., entity) that is proximate to the vehicle 302 and/or a classification of the object as an object type (e.g., car, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In some examples, the perception component 322 may provide processed sensor data that indicates a presence of a stationary entity that is proximate to the vehicle 302 and/or a classification of the stationary entity as a type (e.g., building, tree, road surface, curb, sidewalk, unknown, etc.). In additional or alternative examples, the perception component 322 may provide processed sensor data that indicates one or more characteristics associated with a detected object (e.g., a tracked object) and/or the environment in which the object is positioned. In some examples, characteristics associated with an object may include, but are not limited to, an x-position (global and/or local position), a y-position (global and/or local position), a z-position (global and/or local position), an orientation (e.g., a roll, pitch, yaw), an object type (e.g., a classification), a velocity of the object, an acceleration of the object, an extent of the object (size), etc. Characteristics associated with the environment may include, but are not limited to, a presence of another object in the environment, a state of another object in the environment, a time of day, a day of a week, a season, a weather condition, an indication of darkness/light, etc.
(84) col 22, ll. 40-55, As illustrated in FIG. 3, the vehicle computing device 304 may include a relevance determination component 330. The relevance determination component 330 may be configured to determine relevance of one or more objects, such as objects 104 of FIG. 1, to the vehicle 302. In various examples, the relevance determination component 330 may receive one or more characteristics associated with the detected object(s) from the perception component 322 and/or from the sensor system(s) 306.
See also (97) col 26, ll. 5-15, , (108) col 28, ll. 55 - col. 29, l. 5
Regarding claim 26, Kiiski discloses the limitations of claim 21, and further discloses: wherein one or more of the first relative location embedding and the second relative location embedding respectively encodes the respective relative locations of the plurality of actors as a multi-channel positional embedding.
(31) col 8, ll. 40-65, In various examples, the vehicle computing system may receive the sensor data and may determine a type of object 104 (e.g., classify the type of object), such as, for example, whether the object 104 is a car, such as objects 104(1) and 104(2), a truck, motorcycle, moped, bicyclist, pedestrian, such as object 104(3), or the like. In various examples, the vehicle computing system may determine one or more predicted object trajectories 108 (trajectories 108) based on the sensor data and/or the type of object 104, such as trajectories 108(1), 108(2), and 108(3) associated with objects 104(1), 104(2), and 104(3), respectively. In some examples, the trajectories 108 may include any number of potential paths in which the object 104 may travel from a current position (e.g., at the time of perception) and/or based on a direction of travel. In some examples, a potential path for an object 104 may include remaining stationary, such as object 104(2) stopped at an intersection 110 in the environment 100. In such an example, the corresponding trajectory 108, such as 108(2), may represent little to no motion. In some examples, the number of trajectories may vary depending on a variety of factors, such as the classification of the object (e.g., type of object), other stationary and/or dynamic objects, drivable surfaces, etc.
(43) col 11, ll. 40-55, Additionally, as an illustrative example, the vehicle computing system may determine that objects 104(1) and 104(2) may occupy the same space as the vehicle 102 (e.g., determination of relevance to the vehicle 102) during the time period. Based on an initial relevance determination (at time T=0), the vehicle computing device may store data associated with the relevant objects 104(1) and 104(2). The data may include a classification, a location, and/or trajectory data associated with the relevant objects 104(1) and 104(2). Though the illustration in FIG. 1 includes two relevant objects, the vehicle computing system may be configured to identify a greater or fewer number of relevant objects. The vehicle computing system may consider the data associated with the relevant objects 104(1) and 104(2) in control planning considerations.
(46) col 12, ll. 25-45, In various examples, for each applicable action and sub-action, the vehicle computing system may generate a set of estimated states of the vehicle and a relevant object 104, such as objects 104(1) and/or 104(2), into the future.
(96) col 25, ll. 53 - col. 26, l. 7, Based on a determination that an object is relevant to the vehicle 302 at a time associated with a relevance verification estimated state, the relevance determination component 330 may continue to include data associated with the object in subsequent estimated states of the set of estimated states.
See Fig. 1
Regarding claim 27, Kiiski discloses the limitations of claim 21, and further discloses: further comprising: processing the second motion forecast data with a machine-learned prediction model to generate respective trajectories of the plurality of actors for the first time step and respective projected trajectories of the plurality of actors for the second time step; and
(abstract) The techniques may include determining locations and trajectories associated with a detected object and generating simulations of movement (e.g., estimated states)
(11) col 2, ll. 39-68, The vehicle computing system may be configured to determine an initial position of each detected object. In various examples, the vehicle computing system may determine one or more predicted trajectories associated with each detected object, such as from an initial position associated therewith. In some examples, the one or more predicted trajectories may be determined based on the sensor data. Each predicted trajectory may represent a potential path that the detected object may travel through the environment. The one or more predicted trajectories may be based on passive prediction (e.g., independent of an action the vehicle and/or another object takes in the environment, substantially no reaction to the action of the vehicle and/or other objects, etc.), active prediction (e.g., based on a reaction to an action of the vehicle and/or another object in the environment), or a combination thereof. … In some examples, the one or more predicted trajectories may be determined utilizing machine learning techniques.
(53) col 13, l. 55- col. 14, l. 15, In various examples, the vehicle computing system may perform one or more relevance verifications during a set of estimated states. The relevance verification(s) may include a determination that an object previously determined to be relevant to the vehicle 102 remains relevant to the vehicle (e.g., may occupy the same space as the vehicle 102 for the remainder of the set of estimated states). In various examples, a relevance verification may determine whether an action and/or sub-action is independent of an object 104 (e.g., the object 104 has no bearing on the vehicle 102 performing the action) previously determined to be relevant. In some examples, the relevance verification(s) may be performed substantially concurrently with one or more estimated states of the set of estimated states, such as utilizing data associated with a respective estimated state. The relevance verification(s) may be based on geometric data (e.g., distance, angle, direction of travel, velocity, acceleration, trajectory, etc.), semantic data (e.g., classification of the object), or a combination thereof. In various examples, the vehicle computing system may compare an estimated position of the vehicle 114 in an estimated state to an estimated position of the object 116 in the estimated state. For example, for a fourth estimated state, the vehicle computing system may compare the estimated position of the vehicle 114(4) to the estimated position of the relevant object 116(4) to determine if the relevant object remains relevant to the vehicle.
(56) col 14, ll. 55 - col. 15, l. 15, For example, the vehicle computing system may perform a first relevance verification substantially concurrently with a second estimated state and a second relevance verification substantially concurrently with a fourth estimated state
(91) col 24, ll. 10-40, In some examples, the relevance verification(s) may be performed substantially concurrently with one or more estimated states of the set of estimated states. In some examples, the relevance verification(s) may be performed using data (e.g., estimated positions, etc.) associated with an estimated state.
See also (100) col 26, ll. 30-65,
controlling the motion of the autonomous vehicle further based on the respective trajectories and the respective projected trajectories generated by the machine-learned prediction model.
(44) col 11, ll. 55 - col. 12, l. 5, In various examples, the vehicle computing system may be configured to determine actions to take while operating (e.g., control planning) based on the relevant objects 104(1) and 104(2). The actions may include reference actions (e.g., one of a group of maneuvers the vehicle is configured to perform in reaction to a dynamic operating environment) such as a right lane change, a left lane change, staying in a lane, going around an obstacle (e.g., double-parked vehicle, traffic cones, etc.), or the like. The actions may additionally include sub-actions, such as speed variations (e.g., maintain velocity, accelerate, decelerate, etc.), positional variations (e.g., changing a position in a lane), or the like. For example, an action may include staying in a lane (action) and adjusting a position of the vehicle in the lane from a centered position to operating on a left side of the lane (sub action).
(118) col 30, ll. 65 - col. 31, l. 15, At operation 404, the process may include determining an action that the vehicle may take while operating in the environment. In some examples, the action may be based on the detected object and/or the object trajectory associated therewith. The action may include a reference action such as a right lane change, a left lane change, staying in a lane, going around an obstacle (e.g., double-parked vehicle, traffic cones, etc.), or the like. The action may additionally include one or more sub actions, such as speed control (e.g., maintain velocity, accelerate, decelerate, etc.), positional variations (e.g., changing a position in a lane), or the like. For example, an action may include staying in a lane (action) and adjusting a position of the vehicle in the lane from a centered position to operating on a left side of the lane (sub action).
See also: (35) col 9, ll. 40-55, In various examples, the trajectories 108 may be determined utilizing one or more machine learning algorithms. In such examples, the vehicle computing system, such as via a prediction system, may receive the sensor data associated with the object 104 and predict a behavior of the object 104 based on the sensor data.
Regarding claim 28, Kiiski discloses the limitations of claim 27, and further discloses: wherein the machine- learned prediction model comprises one or more multi-layer perceptrons.
(99) col 26, ll. 20-35, As described herein, an exemplary neural network is a biologically inspired technique which passes input data through a series of connected layers to produce an output. Each layer in a neural network may also comprise another neural network, or may comprise any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of such techniques in which an output is generated based on learned parameters.
(100) col 26, ll. 30-65, Although discussed in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning techniques may include, but are not limited to, regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based techniques (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decisions tree techniques (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), Chi-squared automatic interaction detection (CHAID), decision stump, conditional decision trees), Bayesian techniques (e.g., naïve Bayes, Gaussian naïve Bayes, multinomial naïve Bayes, average one-dependence estimators (AODE), Bayesian belief network (BNN), Bayesian networks), clustering techniques (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning techniques (e.g., perceptron, back-propagation, hopfield network, Radial Basis Function Network (RBFN)), deep learning techniques (e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Network (CNN), Stacked Auto-Encoders), Dimensionality Reduction Techniques (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), Ensemble Techniques (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, PointNet, and the like.
Regarding claim 29, Kiiski discloses the limitations of claim 27, and further discloses: generating respective trajectory sequences for the plurality of actors, the respective trajectory sequences comprising the respective trajectories of the plurality of actors for the first time step and the respective projected trajectories of the plurality of actors for the second time step.
(21) col 6, ll. 1-25, In various examples, the vehicle computing system may perform the relevance verification(s) randomly throughout the period of time (e.g., at 2.5 seconds, at 7.3 seconds, with the 52.sup.nd estimated state of a set of estimated states, etc.). In some examples, the vehicle computing system may perform the relevance verification(s) periodically throughout the period of time (e.g., every 2 seconds, 4 seconds, 5 seconds, every 10 estimated states, 20 estimated states, etc.)
(51) col 13, ll. 25-45, The estimated state(s) may each represent an estimated position of the vehicle 114 and an estimated position of one or more relevant objects 116. The estimated position of the vehicle 114 and the estimated position of relevant objects 116 may be based on the movement of the vehicle 102 (e.g., vehicle trajectory 112 based on an action, etc.) and the movement of the relevant object 104 (e.g., trajectory 108), respectively. The illustrative example depicted in FIG. 1 depicts a plurality of estimated positions corresponding to a plurality of estimated states. For example, estimated positions of the vehicle 114(1), 114(2), 114(3), 114(4), and 114(5) correspond to estimated states 1 through 5, and estimated positions of the relevant object 116(1), 116(2), 116(3), and 116(4) correspond to estimated states 1 through 4.
Regarding claim 30, Kiiski discloses the limitations of claim 21, and further discloses: wherein the machine- learned interaction transformer model comprises: a machine-learned interaction model configured to receive the first or second relative location embedding that describes the respective relative locations of the plurality of actors with respect to the autonomous vehicle, and in response to receipt of the relative location embedding, generate an attention embedding with respect to the plurality of actors;
(36) col 9, l. 50 - col. 10, l. 5, In various examples, the vehicle computing system may determine an initial relevance of each detected object 104 in the environment 100 to the vehicle 102. The initial relevance may be based on a determination that a detected object 104 may occupy the same space as the vehicle 102 at an initial time (e.g., current time) and/or over a time period (e.g., 4 seconds, 8 seconds, 10 seconds, 1 minute, etc.). n some example, the time period may include a predetermined amount of time. In at least one example, the time period may be eight (8) seconds. In some examples, the time period may be based on one or more operating factors associated with the vehicle 102, such as speed, environmental factors (e.g., congested area (e.g., a number of detected objects 104), size of a road (e.g., number of lanes, width of lanes, etc.), proximity to intersection, proximity to a playground, school zone, speed limit associated with the road, etc.), weather considerations (e.g., rain, sleet, snow; ice, wind, etc.), timing factors (e.g., a time of day, day of the year, etc.), or the like.
(51) col 13, ll. 25-45, The estimated state(s) may each represent an estimated position of the vehicle 114 and an estimated position of one or more relevant objects 116. The estimated position of the vehicle 114 and the estimated position of relevant objects 116 may be based on the movement of the vehicle 102 (e.g., vehicle trajectory 112 based on an action, etc.) and the movement of the relevant object 104 (e.g., trajectory 108), respectively. The illustrative example depicted in FIG. 1 depicts a plurality of estimated positions corresponding to a plurality of estimated states. For example, estimated positions of the vehicle 114(1), 114(2), 114(3), 114(4), and 114(5) correspond to estimated states 1 through 5, and estimated positions of the relevant object 116(1), 116(2), 116(3), and 116(4) correspond to estimated states 1 through 4.
(53) col 13, l. 55- col. 14, l. 15, In various examples, the vehicle computing system may perform one or more relevance verifications during a set of estimated states. The relevance verification(s) may include a determination that an object previously determined to be relevant to the vehicle 102 remains relevant to the vehicle (e.g., may occupy the same space as the vehicle 102 for the remainder of the set of estimated states). In various examples, …
(54) col 14, ll. 15-40, In various examples, the vehicle computing system may utilize machine learning techniques to verify relevance of a relevant object 104.
a machine-learned recurrent model configured to receive the attention embedding, and in response to receipt of the attention embedding, generate the first or second motion forecast data with respect to the plurality of actors.
(96) col 25, ll. 53 - col. 26, l. 7, Based on a determination that an object is relevant to the vehicle 302 at a time associated with a relevance verification estimated state, the relevance determination component 330 may continue to include data associated with the object in subsequent estimated states of the set of estimated states. Based on a determination that an object is no longer relevant to the vehicle at a time associated with a relevance verification estimated state, the relevance determination component 330 may remove data associated with the object from subsequent estimated states of the set of estimated states and/or other vehicle control planning operations. By removing the data associated with irrelevant objects from future estimated states, the relevance determination component 330 may increase an amount of memory and processing power available to the vehicle computing device 304 for other calculations, programs, applications, etc. In some examples, the removal of data associated with the irrelevant object may increase processing speed related to relevance determinations corresponding to other objects in the environment. Accordingly, the techniques described herein may improve the functioning of the vehicle computing system.
See also: (99) col 26, ll. 20-35, (100) col 26, ll. 30-65
Regarding claim 31, Kiiski discloses the limitations of claim 21, and further discloses: wherein the machine- learned interaction transformer model comprises one or more neural networks.
(98) col 26, ll. 15-25, In some instances, aspects of some or all of the components discussed herein may include any models, techniques, and/or machine learning techniques. For example, in some instances, the components in the memory 318 (and the memory 334, discussed below) may be implemented as a neural network.
(99) col 26, ll. 20-35, As described herein, an exemplary neural network is a biologically inspired technique which passes input data through a series of connected layers to produce an output. Each layer in a neural network may also comprise another neural network, or may comprise any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, a neural network may utilize machine learning, which may refer to a broad class of such techniques in which an output is generated based on learned parameters.
Regarding claim 32, see the mapping of claim 21 above, which recites analogous limitations. In addition, Kiiski further discloses: A computing system for controlling motion of an autonomous vehicle, the computing system comprising: one or more processors; and one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the one or more processors to perform operations comprising [see claim 21]
(74) col. 20, ll. 15-20 FIG. 3 is a block diagram of an example system 300 for implementing the techniques described herein. In at least one example, the system 300 may include a vehicle 302, such as vehicle 102.
(75) col. 20, ll. 20-25, The vehicle 302 may include a vehicle computing device 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312, and one or more drive modules 314.
(76) col. 20, ll. 25-50, The vehicle computing device 304 may include one or more processors 316 and memory 318 communicatively coupled with the one or more processors 316.
Regarding claim 33, Kiiski discloses the limitations of claim 32. In addition, see the mapping of claim 22 above, which recites analogous limitations.
Regarding claim 34, Kiiski discloses the limitations of claim 32. In addition, see the mapping of claim 23 above, which recites analogous limitations.
Regarding claim 35, Kiiski discloses the limitations of claim 34. In addition, see the mapping of claim 24 above, which recites analogous limitations.
Regarding claim 36, Kiiski discloses the limitations of claim 34. In addition, see the mapping of claim 25 above, which recites analogous limitations.
Regarding claim 37, Kiiski discloses the limitations of claim 32. In addition, see the mapping of claim 26 above, which recites analogous limitations.
Regarding claim 38, Kiiski discloses the limitations of claim 32. In addition, see the mapping of claim 27 above, which recites analogous limitations.
Regarding claim 39, see the mapping of claim 21 above, which recites analogous limitations. In addition, Kiiski further discloses: An autonomous vehicle, comprising:
one or more processors; and one or more non-transitory computer-readable media that store instructions for execution by the one or more processors to cause the one or more processors to perform operations comprising: [see claim 21]
(74) col. 20, ll. 15-20 FIG. 3 is a block diagram of an example system 300 for implementing the techniques described herein. In at least one example, the system 300 may include a vehicle 302, such as vehicle 102.
(75) col. 20, ll. 20-25, The vehicle 302 may include a vehicle computing device 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312, and one or more drive modules 314.
(76) col. 20, ll. 25-50, The vehicle computing device 304 may include one or more processors 316 and memory 318 communicatively coupled with the one or more processors 316.
Regarding claim 40, Kiiski discloses the limitations of claim 39. In addition, see the mapping of claim 27 above, which recites analogous limitations.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Packer et al. (US 20200086855 A1), disclosing collision prediction and avoidance for vehicles.
Ogale et al. (US 20190022085 A1), disclosing neural networks for vehicle trajectory planning.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ABBY J FLYNN whose telephone number is (571)272-9855. The examiner can normally be reached Monday - Friday 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James Trammell can be reached at 571-272-6712. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ABBY J FLYNN/ Examiner, Art Unit 3663
Prosecution Insights