DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This action is responsive to the following communication: Original claims filed 04/14/23. This action is made non-final.
3. Claims 1-17 are pending in the case. Claims 1-8 and 12-17 are elected. Claims 1 and 12 are the elected independent claims.
Applicant’s election without traverse of Claims 1-8 and 12-17 in the reply filed on 3/27/26 is acknowledged.
Claims 9-11 are withdrawn from further consideration pursuant to 37 CFR 1.142(b) as being drawn to a nonelected invention, there being no allowable generic or linking claim. Election was made without traverse in the reply filed on 3/27/26.
35 USC § 112(F)
4. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
5. With regard to claim 1, claim limitations “a shared information generation module” and “a future trajectory prediction module” have been interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because they use a generic placeholder (e.g. “module”) coupled with functional language (e.g. “configured to”, etc.) without reciting sufficient structure to achieve the function. Furthermore, the generic placeholder is not preceded by a structural modifier.
Absence of the word “means” (or “step for”) in a claim creates a rebuttable presumption that the claim element is not to be treated in accordance with 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph). The presumption that 35 U.S.C. 112(f) (pre-AIA 35 U.S.C. 112, sixth paragraph) is not invoked is rebutted when the claim element recites function but fails to recite sufficiently definite structure, material or acts to perform that function.
Claim elements in this application that use the word “means” (or “step for”) are presumed to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action. Similarly, claim elements that do not use the word “means” (or “step for”) are presumed not to invoke 35 U.S.C. 112(f) except as otherwise indicated in an Office action.
Since the claim limitation(s) invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, claims 2-8 have been interpreted to cover the corresponding structure described in the specification that achieves the claimed function, and equivalents thereof.
If applicant wishes to provide further explanation or dispute the examiner’s interpretation of the corresponding structure, applicant must identify the corresponding structure with reference to the specification by page and line number, and to the drawing, if any, by reference characters in response to this Office action.
If applicant does not intend to have the claim limitation(s) treated under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112 , sixth paragraph, applicant may amend the claim(s) so that it/they will clearly not invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, or present a sufficient showing that the claim recites/recite sufficient structure, material, or acts for performing the claimed function to preclude application of 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
For more information, see MPEP § 2173 et seq. and Supplementary Examination Guidelines for Determining Compliance With 35 U.S.C. 112 and for Treatment of Related Issues in Patent Applications, 76 FR 7162, 7167 (Feb. 9, 2011)
Claim Objections
6. Claims 7-8 and 16-17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 101
7. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-8 and 12-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claim 1 is an apparatus type claim. Claim 12 is a method type claimTherefore, claims 1-8 and 12-17 are directed to either a process, machine, manufacture or composition of matter.
With respect to claim 1:
2A Prong 1:
a shared information generation module configured to: collect location information of one or more objects around an autonomous vehicle for a predetermined time, generate past movement trajectories for the one or more objects based on the location information (mental process – a user collect information and generate past movements);
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
generate a driving environment feature map for the autonomous vehicle based on road information around the autonomous vehicle and the past movement trajectories; and a future trajectory prediction module configured to generate future trajectories for the one or more objects based on the past movement trajectories and the driving environment feature map (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
generate a driving environment feature map for the autonomous vehicle based on road information around the autonomous vehicle and the past movement trajectories; and a future trajectory prediction module configured to generate future trajectories for the one or more objects based on the past movement trajectories and the driving environment feature map (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 2:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
wherein the shared information generation module is configured to collect type information of the one or more objects, and wherein the apparatus for predicting future trajectories of various types of objects comprises a plurality of future trajectory prediction modules corresponding to respective types that the type information can have. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the shared information generation module is configured to collect type information of the one or more objects, and wherein the apparatus for predicting future trajectories of various types of objects comprises a plurality of future trajectory prediction modules corresponding to respective types that the type information can have (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 3:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
collect location information of the one or more objects, and generate past movement trajectories for the one or more objects based on the location information; a driving environment context information generator configured to generate a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories; and a driving environment feature map generator configured to generate the driving environment feature map by inputting the driving environment context information image to a first convolutional neural network. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
collect location information of the one or more objects, and generate past movement trajectories for the one or more objects based on the location information; a driving environment context information generator configured to generate a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories; and a driving environment feature map generator configured to generate the driving environment feature map by inputting the driving environment context information image to a first convolutional neural network. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 4:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
an object past trajectory information extractor configured to generate a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories; an object-centered context information extractor configured to generate an object environment feature vector by using a second convolutional neural network based on the driving environment feature map; and a future trajectory generator configured to generate the future trajectories by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
an object past trajectory information extractor configured to generate a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories; an object-centered context information extractor configured to generate an object environment feature vector by using a second convolutional neural network based on the driving environment feature map; and a future trajectory generator configured to generate the future trajectories by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 5 & 14:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
extract the road information including a lane centerline from an HD map, and generate the driving environment context information image in a method for displaying the road information and the past movement trajectories on a 2D image (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
extract the road information including a lane centerline from an HD map, and generate the driving environment context information image in a method for displaying the road information and the past movement trajectories on a 2D image (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 6 & 15:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
extract the road information including a lane centerline from an HD map, generate a road image based on the road information, generate a past movement trajectory image based on the past movement trajectories, and generate the driving environment context information image by combining the road image and the past movement trajectory image with each other in a channel direction (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
extract the road information including a lane centerline from an HD map, generate a road image based on the road information, generate a past movement trajectory image based on the past movement trajectories, and generate the driving environment context information image by combining the road image and the past movement trajectory image with each other in a channel direction (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 7 & 16:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
generate a lattice template in which a plurality of location points are arranged in a lattice shape, move all the location points included in the lattice template to a coordinate system being centered around a location and a heading direction of a specific object, generate an agent feature map by extracting a feature vector at a location in the driving environment feature map corresponding to all the moved location points, and generate the object environment feature vector by inputting the agent feature map to a second convolutional neural network. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
generate a lattice template in which a plurality of location points are arranged in a lattice shape, move all the location points included in the lattice template to a coordinate system being centered around a location and a heading direction of a specific object, generate an agent feature map by extracting a feature vector at a location in the driving environment feature map corresponding to all the moved location points, and generate the object environment feature vector by inputting the agent feature map to a second convolutional neural network. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 8 & 17:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
wherein the object-centered context information extractor is configured to set at least one of a horizontal spacing and a vertical spacing between the location points included in the lattice template based on the type of the specific object. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the object-centered context information extractor is configured to set at least one of a horizontal spacing and a vertical spacing between the location points included in the lattice template based on the type of the specific object. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 12:
2A Prong 1:
a step of collecting location information of one or more objects around an autonomous vehicle for a predetermined time, and generating past movement trajectories for the one or more objects based on the location information (mental process – a user collect information and generate past movements);
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
a step of generating a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating a driving environment feature map by inputting the driving environment context information image to a first convolutional neural network; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating an object environment feature vector by using a second convolutional neural network based on the driving environment feature map (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating future trajectories for the one or more objects by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
a step of generating a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating a driving environment feature map by inputting the driving environment context information image to a first convolutional neural network; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories; (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating an object environment feature vector by using a second convolutional neural network based on the driving environment feature map (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
a step of generating future trajectories for the one or more objects by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
With respect to claim 13:
2A Prong 2: This judicial exception is not integrated into a practical application.
Additional elements:
a step of transforming the past movement trajectories into an object-centered coordinate system, wherein the step of generating the motion feature vector generates the motion feature vector by using the LSTM based on the past movement trajectories having been transformed into the object-centered coordinate system. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
a step of transforming the past movement trajectories into an object-centered coordinate system, wherein the step of generating the motion feature vector generates the motion feature vector by using the LSTM based on the past movement trajectories having been transformed into the object-centered coordinate system. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP 2106.05(f)).
Claim Rejections - 35 USC § 102
8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
9. Claims 1-6 and 12-15 are rejected under 35 U.S.C. 102(a)(1) as being rejected by anticipated by Sun (US 20210174668).
Regarding claim 1, Sun discloses an apparatus for predicting future trajectories of various types of objects comprising:
a shared information generation module configured to:
collect location information of one or more objects around an autonomous vehicle for a predetermined time, generate past movement trajectories for the one or more objects based on the location information (the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045) and generate a driving environment feature map for the autonomous vehicle based on road information around the autonomous vehicle and the past movement trajectories (the perception module 108 is configured to identify one or more objects based on sensor data from the one or more sensors 102, and map data about the surrounding environment. The map data may include local map data of the lanes that are within a threshold vicinity of the autonomous vehicle. The perception module 108 may use the sensor data and map data to identify a current state of objects in the vicinity of the autonomous vehicle. The current state information may include, for example, trajectory of moving objects near the autonomous vehicle, as well as classification of the objects as vehicles, pedestrians, and the like, see paragraph 0032); and
a future trajectory prediction module configured to generate future trajectories for the one or more objects based on the past movement trajectories and the driving environment feature map (The prediction module 110 may include one or more neural networks, such as, for example, one or more convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory (LSTM) recurrent neural networks, gated recurrent units (GRUs) and/or the like. The neural network that is employed may include different number of layers and different number of nodes within each layer of the neural network. The one or more neural networks may be trained, among other things, to extract relevant features for the inputs to the networks, generate predictions of one or more lanes that the autonomous vehicle may like enter, and/or predict one or more future trajectories for the autonomous vehicle. The output of the prediction module 110 may be, for example, the predicted future trajectories of the autonomous vehicle, see paragraph 0033).
Regarding claim 2, Sun discloses wherein the shared information generation module is configured to collect type information of the one or more objects, and wherein the apparatus for predicting future trajectories of various types of objects comprises a plurality of future trajectory prediction modules corresponding to respective types that the type information can have (the predicted future trajectories from the prediction module 110 are provided to the motion planning module 111. In one embodiment, the motion planning module 111 may be configured to determine a motion plan for the autonomous vehicle 100, based on the predicted future trajectories and associated probability values. In this regard, the motion planning module 111 may generate a motion plan that follows one of the predicted future trajectories in a safe manner, see paragraph 0034).
Regarding claim 3, Sun discloses wherein the shared information generation module comprises:
a location data receiver for each object configured to: collect location information of the one or more objects (the trajectory information includes position histories and velocity histories of the target vehicle, see paragraph 0004), and generate past movement trajectories for the one or more objects based on the location information (the determining of the trajectory feature includes: encoding the position histories into a first vector; encoding velocity histories into a second vector; and concatenating the first vector and the second vector, see paragraph 0005);
a driving environment context information generator configured to generate a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories (the perception module 108 is configured to identify one or more objects based on sensor data from the one or more sensors 102, and map data about the surrounding environment. The map data may include local map data of the lanes that are within a threshold vicinity of the autonomous vehicle. The perception module 108 may use the sensor data and map data to identify a current state of objects in the vicinity of the autonomous vehicle. The current state information may include, for example, trajectory of moving objects near the autonomous vehicle, as well as classification of the objects as vehicles, pedestrians, and the like, see paragraph 0032); and
a driving environment feature map generator configured to generate the driving environment feature map by inputting the driving environment context information image to a first convolutional neural network (The trajectory encoder 208 may include one or more be neural networks (e.g. LSTM, GRU, or the like) trained to extract and encode features of the trajectories for the target vehicle 100 and nearby vehicles 200. In one embodiment, the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045).
Regarding claim 4, Sun discloses wherein the future trajectory prediction module comprises:
an object past trajectory information extractor configured to generate a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories (The trajectory encoder 208 may include one or more be neural networks (e.g. LSTM, GRU, or the like) trained to extract and encode features of the trajectories for the target vehicle 100 and nearby vehicles 200. In one embodiment, the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045);
an object-centered context information extractor configured to generate an object environment feature vector by using a second convolutional neural network based on the driving environment feature map (The lane encoder 210 may include one or more neural networks trained to extract and encode features of lane segments within the threshold vicinity 206 of the target vehicle 100. In one embodiment, the lane encoder 210 includes a convolutional neural network with one or more convolutional layers (e.g. two 1-dimensional convolutional layers) and a multiple layer perceptron (MLP), for extracting lane features based on the set of ordered lane-points 204 of the lane segments. The final output of the convolutional neural network may be pooled and stored in a fixed size feature vector for each lane. As opposed to using rasterized maps to extract feature representations for an entire scene surrounding the target vehicle 100, the use of local maps for extracting features of only the lanes in the vicinity of the target vehicle is lightweight and more memory efficient, see paragraph 0046); and
a future trajectory generator configured to generate the future trajectories by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector (The lane feature vectors for the lanes generated by the lane encoder 210, and the trajectory feature vector for the target vehicle 100 that is generated by the trajectory encoder 208, are used by the lane attention module 212 for identifying one or more target lanes that are predicted to be the goal of the target vehicle in the upcoming future (e.g. in the next 3 seconds). It is not uncommon for drivers to have their attention on one or a few of the lanes based on their intention. The drivers tend to follow the direction of one of those lanes that is subject of their intention, determining the trajectory of the target vehicle, see paragraph 0046).
Regarding claim 5, Sun discloses wherein the driving environment context information generator is configured to: extract the road information including a lane centerline from an HD map, and generate the driving environment context information image in a method for displaying the road information and the past movement trajectories on a 2D image (FIG. 3 is a conceptual layout diagram of a local map 300 of lanes 302a-302d (collectively referenced as 302) within a vicinity of a target vehicle 100 according to one embodiment. The local map 300 may update as the position of the target vehicle 100 changes. In one embodiment, a segment of a center of the lane 302 is identified by a start point 304 and an end point 306. In one embodiment, the local map 300 provides a series of geographic coordinates of the lane segment to the lane encoder 210 for extracting the features of the lane, see paragraph 0053).
Regarding claim 6, Sun discloses wherein the driving environment context information generator is configured to: extract the road information including a lane centerline from an HD map, generate a road image based on the road information, generate a past movement trajectory image based on the past movement trajectories, and generate the driving environment context information image by combining the road image and the past movement trajectory image with each other in a channel direction (FIG. 4 is a conceptual layout diagram of the lane attention module 212 according to one embodiment. It should be appreciated that the lane attention module 212 according to the various embodiments is flexible in that it may deal with a varying number of lanes in the dynamic driving environment. In this regard, the lane attention module 212 may receive as input, a varying number of lane feature vectors 400a-400c (collectively referenced as 400) produced by the lane encoder module 210. The lane attention module may also receive as input, the trajectory feature vector 402 for the target vehicle 100. In one embodiment, a dot product computation is performed between each lane feature vector 400 and the trajectory feature vector 402 for producing an attention score for each lane. The generated scores may run through a normalization layer 404 such as, for example, a softmax layer, see paragraph 0054).
Regarding claim 12, Sun discloses a method for predicting future trajectories of various types of objects, the method comprising:
a step of collecting location information of one or more objects around an autonomous vehicle for a predetermined time, and generating past movement trajectories for the one or more objects based on the location information (the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045) and
a step of generating a driving environment context information image based on road information around the autonomous vehicle and the past movement trajectories (the perception module 108 is configured to identify one or more objects based on sensor data from the one or more sensors 102, and map data about the surrounding environment. The map data may include local map data of the lanes that are within a threshold vicinity of the autonomous vehicle. The perception module 108 may use the sensor data and map data to identify a current state of objects in the vicinity of the autonomous vehicle. The current state information may include, for example, trajectory of moving objects near the autonomous vehicle, as well as classification of the objects as vehicles, pedestrians, and the like, see paragraph 0032);
a step of generating a driving environment feature map by inputting the driving environment context information image to a first convolutional neural network (The prediction module 110 may include one or more neural networks, such as, for example, one or more convolutional neural networks (CNN), recurrent neural networks (RNN), long short-term memory (LSTM) recurrent neural networks, gated recurrent units (GRUs) and/or the like. The neural network that is employed may include different number of layers and different number of nodes within each layer of the neural network. The one or more neural networks may be trained, among other things, to extract relevant features for the inputs to the networks, generate predictions of one or more lanes that the autonomous vehicle may like enter, and/or predict one or more future trajectories for the autonomous vehicle. The output of the prediction module 110 may be, for example, the predicted future trajectories of the autonomous vehicle, see paragraph 0033);
a step of generating a motion feature vector by using a long short-term memory (LSTM) based on the past movement trajectories (e.g. LSTM, GRU, or the like) trained to extract and encode features of the trajectories for the target vehicle 100 and nearby vehicles 200. In one embodiment, the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045);
a step of generating an object environment feature vector by using a second convolutional neural network based on the driving environment feature map (The trajectory encoder 208 may include one or more be neural networks (e.g. LSTM, GRU, or the like) trained to extract and encode features of the trajectories for the target vehicle 100 and nearby vehicles 200. In one embodiment, the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045); and
a step of generating future trajectories for the one or more objects by using a variational auto-encoder (VAE) and an MLP based on the motion feature vector and the object environment feature vector (The lane feature vectors for the lanes generated by the lane encoder 210, and the trajectory feature vector for the target vehicle 100 that is generated by the trajectory encoder 208, are used by the lane attention module 212 for identifying one or more target lanes that are predicted to be the goal of the target vehicle in the upcoming future (e.g. in the next 3 seconds). It is not uncommon for drivers to have their attention on one or a few of the lanes based on their intention. The drivers tend to follow the direction of one of those lanes that is subject of their intention, determining the trajectory of the target vehicle, see paragraph 0046).
Regarding claim 13, Sun discloses further comprising a step of transforming the past movement trajectories into an object-centered coordinate system (FIG. 3 is a conceptual layout diagram of a local map 300 of lanes 302a-302d (collectively referenced as 302) within a vicinity of a target vehicle 100 according to one embodiment. The local map 300 may update as the position of the target vehicle 100 changes. In one embodiment, a segment of a center of the lane 302 is identified by a start point 304 and an end point 306. In one embodiment, the local map 300 provides a series of geographic coordinates of the lane segment to the lane encoder 210 for extracting the features of the lane, paragraph 0053),
wherein the step of generating the motion feature vector generates the motion feature vector by using the LSTM based on the past movement trajectories having been transformed into the object-centered coordinate system (The trajectory encoder 208 may include one or more be neural networks (e.g. LSTM, GRU, or the like) trained to extract and encode features of the trajectories for the target vehicle 100 and nearby vehicles 200. In one embodiment, the trajectory encoder 208 includes a first neural network for encoding position histories of an agent into a first vector, and a second neural network for encoding velocity histories of the agent in a second vector, see paragraph 0045).
Regarding claim 14, the subject matter of the claim is substantially similar to claim 5 and a such the same rationale of rejection applies.
Regarding claim 15, the subject matter of the claim is substantially similar to claim 6 and a such the same rationale of rejection applies.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID E CHOI whose telephone number is (571)270-3780. The examiner can normally be reached on M-F: 7-2, 7-10 (PST). If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Bechtold, Michelle T. can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID E CHOI/Primary Examiner, Art Unit 2148
Prosecution Insights