DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 04/27/2026 have been fully considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 8 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wei Wu et al. “SMART: Scalable Multi-agent Real-time Simulation via Next-token Prediction, 2024” (henceforth Wu). The examiner has provided a copy of this paper.
Regarding claim 1,
Wu discloses:
A trajectory prediction method, adapted for an electronic device, wherein the trajectory prediction method comprises:
during a training phase, obtaining movement information of a plurality of agents at a plurality of time points, (See at least Page 2, Section 1, “We propose a novel framework for motion generation, incorporating a tokenization scheme for both vectorized road and agent trajectories and utilizing a decoder-only transformer for training on the next token prediction task” and Page 4, Section 3.1, “the agent motion token is represented as A ∈ R^(NA×NT×FA), where NA denotes the total number of agents, and NT represents the number of time steps”. Movement information of a plurality of agents at a plurality of time points is obtained during a training phase.)
and dividing the time points into a plurality of training patches, wherein each of the training patches comprises at least two of the time points; (See at least Page 4, Section 3.1, “we segment the continuous trajectories of all agents in the dataset into trajectory sets by fixed time intervals t = 0.5s.” Fig. 1(a) in Page 4 illustrates the patches at t= 0s, 0.5s, 1.0s, and 1.5s with each token covering the multi-step interval between successive markers.)
selecting one of the training patches as a current training patch, obtaining at least one previous training patch preceding the current training patch among the training patches, and predicting the current training patch according to the at least one previous training patch to update a plurality of parameters of a machine learning model; (See at least Page 6, Section 3.3, “motion next token prediction”, wherein at each position in the token sequence, the token at time t+1 is the “current training patch” being predicted. With regards to Equation 3, e_1:t is the historical tokenized agent motion embeddings, and a_t+1 is the next predicted agent motion token. Furthermore, Section 3.3 discloses “SMART is trained to minimize the cross entropy between the distribution of the ground truth token label and the predicted distribution”, wherein the minimization is encompassed in Equation 3. Therefore, the parameters of a machine learning model (i.e. theta in Equation 3) are updated based on predicting the current training patch according to the at least one previous training patch.)
changing the current training patch to another one of the training patches, and repeatedly performing the training phase to update the parameters of the machine learning model; (See at least Page 6, Section 3.3, Equation 3, wherein equation 3 includes an outer summation wherein at each value of t, the current training patch rotates to the next position (i.e. changing the current training patch to another one of the training patches) to update the parameters (i.e. theta in Equation 3) of the machine learning model.
Additionally, across training steps, SMART iterates over the dataset in batches (See Page 14, A. 1, training details includes a batch size of 4). Furthermore, Table 9 and Page 16 discloses “Validation is conducted every 50,000 train steps. The model is considered to have converged if there is no significant loss reduction or metric improvement after five consecutive validations”, which discloses repeatedly performing the training phase to update the parameters of the machine learning model.)
during an inference phase, obtaining a plurality of inference patches, and inputting the inference patches to the machine learning model to predict movement information of a first patch; and inputting the inference patches and the first patch to the machine learning model to predict movement information of a second patch. (See at least Section 4.1, Page 7, “Smart only needs to compute the next token for the upcoming frame at the current moment during inference, without the need to re-encode historical motion tokens. By reusing the token embeddings computed in previous observation time horizons..” The historical motion tokens (i.e. the plurality of inference patches) are input into the model and the model emits the next token (i.e. the first predicted batch), wherein each token contains movement information (i.e. “coordinates, heading, and shapes”, Section 3.1, Page 4). Additionally, see Section 4.1, Page 7, “By reusing the token embeddings computed in previous observation time horizons, the complexity of the agent motion decoder is reduced to O(NA NT)+O(NA NR)+O(N2 A).” The first predicted patch is then a previous observation time horizon when predicting the second patch.
Additionally, see Page 14, Section A. 1, Inference for WOSAC, “each scene requires running the model inference 32 x T times to generate the 32 simulations for a group of agents. During model inference, each simulation step produces the classified distribution of next tokens.”)
Regarding claim 8,
Wu discloses the same limitations as recited in claim 1 above, and is therefore rejected under the same rational.
Wu further discloses:
An electronic device, comprising: a memory, configured to store a plurality of instructions; and a processor, communicatively connected to the memory for performing the instructions to complete a plurality of steps:
(See at least Page 16, wherein the training is done on GPUs, which is an electronic device that comprises at least a memory and a processor.)
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-6 and 9-13 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Zikang Zhou et al. “Query-Centric Trajectory Prediction, 2023” (henceforth Zhou). The examiner has provided a copy of this paper.
Regarding claim 2,
Wu discloses the limitations as recited in claim 1 above.
Wu further discloses:
Movement information of the inference patches comprises a position, (See at least Page 4, Section 3.1, “agent motion token”, wherein F_A contains “coordinates, heading, and shapes”. The coordinates are the agent’s position. Additionally, see Fig. 1(a), which comprises the 0.5s inference patches that comprises a position.)
Wu does not specifically state wherein movement information comprises a position, a velocity, and a yaw angle, and the trajectory prediction method further comprises: calculating a position difference, an angle vector difference, a yaw angle difference, and a time difference between two time points; and inputting the position difference, the angle vector difference, the yaw angle difference, and the time difference to a first neural network to obtain a relative feature vector.
However, Zhou discloses:
wherein movement information comprises a position, a velocity, and a yaw angle, (See at least Page 3, Section 3.1, “the i-th agent’s state at time step t comprises the spatial position pt I =(pt i, x, pt i,y), the angular position θti (i.e., the yaw angle), the temporal position t (i.e., the time step), and the velocity vt”. The movement information comprises a position, a velocity, and a yaw angle.)
and the trajectory prediction method further comprises: calculating a position difference, an angle vector difference, a yaw angle difference, and a time difference between two time points; (See at least Page 4, Section 3.2, under “Relative Spatial-Temporal Positional Embedding”, the method further comprises calculating a position difference (i.e. relative reference), and angle vector difference (i.e. relative direction), a yaw angle difference (i.e. relative orientation) and a time difference between two time points (i.e. time gap s – t ). )
and inputting the position difference, the angle vector difference, the yaw angle difference, and the time difference to a first neural network to obtain a relative feature vector. (See at least Page 4, Section 3.2, under “Relative Spatial-Temporal Positional Embedding” following the position, angle, yaw, and time difference passage, “Then, we transform the 4D descriptor into Fourier features and pass them through an MLP to produce the relative positional embedding”. Therefore, these are input into a first neural network to obtain a relative feature vector (i.e. positional embedding).)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Wu to incorporate the teachings of Zhou to include “wherein movement information comprises a position, a velocity, and a yaw angle, and the trajectory prediction method further comprises: calculating a position difference, an angle vector difference, a yaw angle difference, and a time difference between two time points; and inputting the position difference, the angle vector difference, the yaw angle difference, and the time difference to a first neural network to obtain a relative feature vector” since Wu discloses in Page 5 Section 3.2, “Akin to query-centric methodologies [52], we utilize relative positional embeddings to differentiate between agents’ local coordinate frames, enabling symmetric encoding”. Wu uses the query-centric methodologies of Zhou to differentiate between agents’ local coordinate frames, which enables symmetric encoding. This would create a more robust system for trajectory prediction for autonomous driving. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Wu and Zhou. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 3,
Wu further discloses:
further comprising: generating a first query according to movement information of a latest time point of a last one of the inference patches; generating a first key and a first value according to the relative feature vector between other time points and the latest time point of the last one of the inference patches; and performing a multi-head self-attention algorithm to obtain a patch feature vector according to the first query, the first key, and the first value.
(See at least Page 5, Section 3.2 “we leverage a factorized Transformer architecture with multi-head cross-attention (MHCA) to decode complex road-agent and agent-agent relationships along the time series….Denoted as Eq. 2a, given the query derived from the agent motion token’s embedding e_it, we employ temporal attention by computing the key and value based on which are the i_-th agent’s token embeddings from time step t − τ to time step t − 1 and the corresponding relative positional embeddings.” SMART uses MHSA (multi-head self-attention) in Eq. 2a , wherein the output is the latest patch’s feature vector. Equations 2a-2c in Page 5 includes q(), k(), and v() which are the query, key, and values respectively, which are used within the MHSA algorithm to obtain a patch feature vector.)
Regarding claim 4,
Wu further discloses:
further comprising: using the patch feature vector as a second query; generating a second key and a second value according to the relative feature vector between a plurality of time points of other inference patches of the inference patches and the latest time point; and performing the multi-head self-attention algorithm to obtain a time feature vector according to the second query, the second key, and the second value. (Similarly to claim 3, they keys and values are built from token embeddings (Page 5, Section 3.2) from time step t − τ to time step t − 1 and the corresponding relative positional embeddings. Therefore, the keys and values are built from a plurality of other patches (i.e. plurality of time points of other inference patches of the inference patches and the latest time point). In Page 6, Section 3.2, “Likewise, in Eq.2b and Eq.2c, the key and value for agent-map and agent-agent attention are derived from road token rj, j ∈ Ni and agents’ motion token etj, j ∈ Ni in the neighborhood respectively, where the neighbor set Ni is determined by a distance threshold of 50 meters. We stack the temporal, the agent-agent, and the agent-map attention sequentially as one fusion block and repeat such blocks K times.” Since the temporal, agent-agent, and the agent-map attention are stacked sequentially and repeated K times, then the second time it is repeated includes using the patch feature vector as a second query, such that the multi-head self-attention algorithm is performed to obtain a time feature vector according to the second query, the second key, and the second value. Since the operation “captures the temporal dynamics of an agent’s movements” (Page 5, Section 3.2), then this includes a time feature vector that is output.)
Regarding claim 5,
Wu further discloses:
further comprising: using the time feature vector as a third query; generating a third key and a third value according to a correlation between the movement information of the latest time point and a plurality of adjacent points on a map; and performing a multi-head cross-attention algorithm to obtain an agent-map feature vector according to the third query, the third key, and the third value.
(Similarly to claim 4, the temporal, the agent-agent, and the agent-map attention are stacked sequentially as one fusion block and repeated a K number of times. See Equation 2b in Page 5, which uses a multi-head cross-attention (MHCA) algorithm and Page 5, Section 3.2, “we leverage a factorized Transformer architecture with multi-head cross-attention (MHCA) to decode complex road-agent and agent-agent relationships along the time series”. Further see Page 6, Section 3.2, “Likewise, in Eq.2b and Eq.2c, the key and value for agent-map and agent-agent attention are derived from road token rj, j ∈ Ni and agents’ motion token etj, j ∈ Ni in the neighborhood respectively, where the neighbor set Ni is determined by a distance threshold of 50 meters. We stack the temporal, the agent-agent, and the agent-map attention sequentially as one fusion block and repeat such blocks K times.” The “neighbor set Ni” are the plurality of adjacent points on a map, and the multi-head cross-attention (MHCA) algorithm is performed to obtain an agent-map feature vector according to the third query, the third key, and the third value. Since the “temporal, the agent-agent, and the agent-map attention are stacked sequentially as one fusion block and repeated a K number of times”, then that includes using the time feature vector as a third query.)
Regarding claim 6,
Wu further discloses:
wherein the agent-map feature vector belongs to a first agent, (See at least Page 5, Section 3.2, “Take the i_th agent at time step t as an example” and in equation 2b, the output agent-map feature vector belongs to a first agent (since “i” is included in the equation).)
and the trajectory prediction method further comprises: using the agent-map feature vector as a fourth query; generating a fourth key and a fourth value according to an agent-map feature vector of a second agent and the relative feature vector between a plurality of time points of the last one of the inference patches of the second agent and the latest time point; and performing the multi-head self-attention algorithm to obtain an agent to agent feature vector according to the fourth query, the fourth key, and the fourth value.
(Similarly to claim 5, the temporal, the agent-agent, and the agent-map attention are stacked sequentially as one fusion block and repeated a K number of times. This includes using the agent-map feature vector (from claim 5) as a fourth query, wherein equation 2c in Page 5 includes performing the multi-head self-attention (MHSA) algorithm to obtain an agent to agent feature vector according to the fourth query, the fourth key, and the fourth value.)
Regarding claim 9,
Wu and Zhou discloses the same limitations as recited in claim 2 above, and is therefore rejected under the same rejection and obviousness rational.
Regarding claim 10,
Wu and Zhou discloses the same limitations as recited in claim 3 above, and is therefore rejected under the same rejection and obviousness rational.
Regarding claim 11,
Wu and Zhou discloses the same limitations as recited in claim 4 above, and is therefore rejected under the same rejection and obviousness rational.
Regarding claim 12,
Wu and Zhou discloses the same limitations as recited in claim 5 above, and is therefore rejected under the same rejection and obviousness rational.
Regarding claim 13,
Wu and Zhou discloses the same limitations as recited in claim 6 above, and is therefore rejected under the same rejection and obviousness rational.
Claims 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Wu and Zhou further in view of Deo et al. “Convolutional Social Pooling for Vehicle Trajectory Prediction, 2018” (henceforth Deo). The examiner has provided a copy of this paper.
Regarding claim 7,
Wu and Zhou discloses the limitations as recited in claims 1-6 above.
Wu does not specifically state inputting the agent to agent feature vector to a second neural network to obtain probability values under a plurality of modes and inputting the agent to agent feature vector to a recurrent neural network to predict movement information of a future patch.
However, Deo teaches:
inputting the agent to agent feature vector to a second neural network to obtain probability values under a plurality of modes, (See at least Section 4.2, “Additionally, the LSTM state of the predicted vehicle is passed through a fully connected layer to obtain the vehicle dynamics encoding. The two encodings are concatenated to form the complete trajectory encoding, which is then passed to the decoder” and Section 4.3, “predicting the distribution for each of the six maneuver classes described in section 3.4 along with the probability for each maneuver class. The decoder has two SoftMax layers that output the lateral and longitudinal maneuver probabilities.” The LSTM’s trajectory encoding is the agent to agent feature vector that is input to the two SoftMax layers of the decoder (i.e. a second neural network) to obtain probability values under a plurality of modes.)
and inputting the agent to agent feature vector to a recurrent neural network to predict movement information of a future patch.
(See at least Section 4.3, “Additionally, an LSTM is used to generate the parameters of a bivariate Gaussian distribution over tf frames to give the predictive distribution for vehicle motion.” The LSTM is a recurrent neural network (See Page 2, Section 2, “recurrent neural network” and “LSTM”) to predict movement information of a future patch. Additionally, see Section 3.2, wherein the output of the model is future coordinates of the vehicle being predicted.)
It would have been obvious to one of the ordinary skill in the art before the effective filing date of the claimed invention to have modified Wu to incorporate the teachings of Deo to include “inputting the agent to agent feature vector to a second neural network to obtain probability values under a plurality of modes and inputting the agent to agent feature vector to a recurrent neural network to predict movement information of a future patch” since “Our LSTM decoder generates the probability distribution over future motion for six maneuver classes and assigns a probability to each maneuver class. This accounts for the multi-modal nature of vehicle motion.” (Page 2, Section 1, Deo). This would create a more robust system for predicting movement information of neighboring vehicles. Additionally, a person having ordinary skill in the art would have a reasonable expectation of success in combining the teachings of Wu and Deo. The claimed invention is merely a combination of known elements and in combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the results of the combination would have been predictable.
Regarding claim 14,
Wu, Zhou, and Deo discloses the same limitations as recited in claim 7 above, and is therefore rejected under the same rejection and obviousness rational.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Li et al. US20210287531A1 discloses The encoder also extracts interaction patterns from observed trajectories from the sensor data and context information. The encoder further generates a static latent interaction graph for a first time step based on the interaction patterns. The recurrent generates a distribution of time dependent static latent interaction graphs iteratively from the first time step for a series of time steps based on the static latent interaction graph. The series of time steps are separated by a re-encoding gap. The decoder generates multi-modal distribution of future states based on the distribution of time dependent static latent interaction graphs. (See abstract)
Djuric et al. US20190049970A1 discloses a computer implemented method that includes obtaining state data indicative of at least a current or a past state of an object that is within a surrounding environment of an autonomous vehicle. The method includes obtaining data associated with a geographic area in which the object is located. The method includes generating a combined data set associated with the object based at least in part on a fusion of the state data and the data associated with the geographic area in which the object is located. The method includes obtaining data indicative of a machine-learned model. The method includes inputting the combined data set into the machine-learned model. The method includes receiving an output from the machine-learned model. The output can be indicative of a plurality of predicted trajectories of the object. (See abstract)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL J LAMBERT whose telephone number is (571)272-4334. The examiner can normally be reached M-F 10:00 am- 6:00 pm MDT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Erin Piateski can be reached at (571) 270-7429. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Erin M Piateski/Supervisory Patent Examiner, Art Unit 3669
/G.J.L./
Examiner
Art Unit 3669
Prosecution Insights