DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 – 2, 7 – 10, and 13 – 14 are rejected under 35 U.S.C. 103 as being unpatentable over Gao et al., "Vectornet: Encoding hd maps and agent dynamics from vectorized representation" in view of Wayne et al., Pub. No.: US20210034969A1.
Regarding claim 1, Gao teaches: A method for training a neural network system to predict a behavior of a set of interacting agents,
((Gao, page: 11525), “with the scene context, provided as prior knowledge often in the form of High Definition (HD) maps. Our goal is to build a system which learns to predict the intent of vehicles [for training a neural network system to predict a behavior of a set of interacting agents], which are parameterized as trajectories.”)
the neural network system including an encoder configured to convert input data regarding each agent into a one-dimensional agent representation with values representing agent features,
((Gao, page: 11527), “Our vectorization process is a one-to-one mapping between continuous trajectories, map annotations and the vector set, although the latter is unordered. This allows us to form a graph representation on top of the vector sets, which can be encoded by graph neural networks [the neural network system including an encoder configured to convert input data regarding each agent into a one-dimensional agent representation with values representing agent features] (i.e.: each trajectory/polyline is encoded into feature vectors (embeddings) before graph processing)”.)
a graph neural network (GNN) configured to predict, based on a complete graph of the agent representations, a complete graph of modified agent representations, and
((Gao, page: 11526), “We use graph neural networks (GNNs) to incorporate these sets of vectors [a graph neural network (GNN) configured to predict, based on a complete graph of the agent representations, a complete graph of modified agent representations]. We treat each vector as a node in the graph, and set the node features to be the start location and end location of each vector, along with other attributes such as polyline group id and semantic labels.”)
a decoder configured to convert the modified agent representations into predicted behavior data regarding each agent, the method comprising the following steps:
((Gao, page: 11526), “The context information from HD maps, along with the trajectories of other moving agents are propagated to the target agent node through the GNN. We can then take the output node feature corresponding to the target agent to decode its future trajectories [a decoder configured to convert the modified agent representations into predicted behavior data regarding each agent].”)
providing training records of input data regarding each agent;
((Gao, page: 11526), “More specifically, we randomly mask out the input node features belonging to either scene context or agent trajectories [providing training records of input data regarding each agent], and ask the model to reconstruct the masked features.”)
generating, from each training record, by the encoder, agent representations;
((Gao, page: 11526), “Figure 2. An overview of our proposed VectorNet. Observed agent trajectories and map features are represented as sequence of vectors, and passed to a local graph network to obtain polyline-level features [generating, from each training record, by the encoder, agent representations]. Such features are then passed to a fully-connected graph to model the higher-order interactions. We compute two types of losses: predicting future trajectories from the node features corresponding to the moving agents and predicting the node features when their features are masked out.”)
processing, by the to-be-trained GNN and the decoder, the agent representations into predicted behavior data regarding each agent;
((Gao, page: 11526), “The context information from HD maps, along with the trajectories of other moving agents are propagated to the target agent node through the GNN. We can then take the output node feature corresponding to the target agent to decode its future trajectories [processing, by the to-be-trained GNN and the decoder, the agent representations into predicted behavior data regarding each agent].”)
determining, from the agent representations, masked agent representations by modifying, in the agent representations for at least two chosen ones of the agents, only respective strict subsets of values of each agent representation; processing, by the to-be-trained GNN, the masked agent representations into interaction representations;
((Gao, page: 11526), “More specifically, we randomly mask out the input node features belonging to either scene context or agent trajectories, and ask the model to reconstruct the masked features [determining, from the agent representations, masked agent representations by modifying, in the agent representations for at least two chosen ones of the agents, only respective strict subsets of values of each agent representation]. The intuition is to encourage the graph networks [processing, by the to-be-trained GNN, the masked agent representations into interaction representations] to better capture the interactions between agent dynamics and scene context.”)
optimizing parameters that characterize the behavior of the GNN and
((Gao, page: 11526), “More specifically, we randomly mask out the input node features belonging to either scene context or agent trajectories, and ask the model to reconstruct the masked features. The intuition is to encourage the graph networks to better capture the interactions between agent dynamics and scene context [optimizing parameters that characterize the behavior of the GNN].”)
Gao does not teach:
determining, by a to-be-trained helper network, from the interaction representations, reconstructions of the agent representations;
rating, using a predetermined loss function, the predicted behavior data, and a deviation of the reconstructions from the agent representations;
parameters that characterize the behavior of the helper network towards a goal of improving, when processing further training records, the rating by the loss function.
Wayne teaches:
determining, by a to-be-trained helper network, from the interaction representations, reconstructions of the agent representations;
(Wayne, “[0017] In implementations a decoding system e.g. a one or more decoder neural networks is provided to generate, from the latent (state) representation, the predicted return and optionally to provide reconstructed input data i.e. a (predicted) reconstruction of one or more components of the input observation [determining, by a to-be-trained helper network, from the interaction representations, reconstructions of the agent representations]. This enables the system parameters i.e. the memory-based prediction parameters and in particular the representation system parameters, to be trained using a reconstruction loss.”)
rating, using a predetermined loss function, the predicted behavior data, and a deviation of the reconstructions from the agent representations; and
(Wayne, “[0017] In implementations a decoding system e.g. a one or more decoder neural networks is provided to generate, from the latent (state) representation, the predicted return and optionally to provide reconstructed input data i.e. a (predicted) reconstruction of one or more components of the input observation. This enables the system parameters i.e. the memory-based prediction parameters and in particular the representation system parameters, to be trained using a reconstruction loss [rating, using a predetermined loss function, the predicted behavior data, and a deviation of the reconstructions from the agent representations]. Thus generating the predicted return may comprise processing, using a decoding system and in accordance with current values of decoding system parameters,”)
parameters that characterize the behavior of the helper network towards a goal of improving, when processing further training records, the rating by the loss function.
(Wayne, “[0018] … This approach allows the values of the memory-based prediction parameters, including values of the encoding system and decoding system parameters, to be adjusted i.e. trained using a divergence loss between the prior and posterior probability distributions [parameters that characterize the behavior of the helper network towards a goal of improving, when processing further training records, the rating by the loss function].”)
Wayne and Gao are related to the same field of endeavor (i.e.: reinforcement learning). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Wayne with teachings of Gao to maintain and update an internal memory representation of the environment over time to better capture sequential information during agent interactions (Wayne, Abstract).
Regarding claim 2, Gao in view of Wayne teach the method of claim 1.
Gao further teaches: wherein the strict subsets of modified values of the agent representations are chosen at most so large that the masked agent representations for the at least two chosen agents are not identical.
((Gao, page: 11526), “Figure 2. An overview of our proposed VectorNet. Observed agent trajectories and map features are represented as sequence of vectors, and passed to a local graph network to obtain polyline-level features. Such features are then passed to a fully-connected graph to model the higher-order interactions. We compute two types of losses: predicting future trajectories from the node features corresponding to the moving agents [the strict subsets of modified values of the agent representations are chosen at most so large that the masked agent representations for the at least two chosen agents are not identical] (i.e.: masking the feature representations of selected moving agent nodes and reconstructing these node features using graph interactions) and predicting the node features when their features are masked out.”)
Regarding claim 7, Gao in view of Wayne teach the method of claim 1.
Gao further teaches: wherein a multilayer perceptron (MLP) is the helper network.
((Gao, page: 11528), “In practice, genc(·) is a multi-layer perceptron (MLP) [wherein a multilayer perceptron (MLP) is the helper network] whose weights are shared over all nodes; specifically, the MLP contains a single fully connected layer followed by layer normalization [3].”)
Regarding claim 8, Gao in view of Wayne teach the method of claim 1.
Gao further teaches: wherein the input data include time series data of a position of the agents and/or a trajectory of the agents and/or a behavior of the agents.
((Gao, page: 11526), “Figure 2. An overview of our proposed VectorNet. Observed agent trajectories and map features are represented as sequence of vectors, and passed to a local graph network to obtain polyline-level features wherein the input data include time series data of a position of the agents and/or a trajectory of the agents and/or a behavior of the agents]. Such features are then passed to a fully-connected graph to model the higher-order interactions. We compute two types of losses: predicting future trajectories from the node features corresponding to the moving agents and predicting the node features when their features are masked out.”)
Regarding claim 9, Gao in view of Wayne teach the method of claim 8.
Gao further teaches: wherein the time series data of the position of the agents and/or the trajectory of the agents and/or the behavior of the agents is split into an earlier part that forms training record, and
((Gao, page: 11528), “To encourage our global interaction graph to better capture interactions among different trajectories [wherein the time series data of the position of the agents and/or the trajectory of the agents and/or the behavior of the agents] and map poly lines, we introduce an auxiliary graph completion task. During training time, we randomly mask out the features for a subset of polyline nodes [is split into an earlier part that forms training record,], e.g. pi.”)
a later part that serves as ground truth for a prediction of the position and/or the trajectory and/or the behavior by the neural network system based on the training records.
((Gao, page: 11528), “where L traj is the negative Gaussian log-likelihood for the groundtruth future trajectories, L node is the Huber loss between predicted node features and groundtruth masked node features, and α = 1.0 is a scalar that balances the two loss terms [a later part that serves as ground truth for a prediction of the position and/or the trajectory and/or the behavior by the neural network system based on the training records].”)
Regarding claim 10, Gao in view of Wayne teach the method of claim 1.
Gao further teaches: wherein the agents are traffic participants interacting in a traffic situation.
((Gao, page: 11525), “This paper focuses on behavior prediction in complex multi-agent systems, such as self-driving vehicles [wherein the agents are traffic participants interacting in a traffic situation]. The core interest is to find a unified representation which integrates the agent dynamics,”)
Regarding claim 13, Wayne teaches: A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for training a neural network system
(Wayne, “[0078] To write data to the memory 116, the memory interface system 206 stores the latent representation 114 in a row of the memory 116 [A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for training a neural network system].”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Wayne with teachings of Gao for the same reasons disclosed for claim 1.
The rest of the limitations are analogous to claim 1, so are rejected under similar rationale.
Claim 14, recites limitations analogous to claim 13, so is rejected under the same rationale.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Gao in view of Wayne and in further view of Banino et al., Pub. No.: US20240095495A1.
Regarding claim 3, Gao in view of Wayne teach the method of claim 1.
Gao in view of Wayne do not teach:
wherein the strict subsets of modified values of the agent representations are chosen at least so large that original values are not derivable from the respective masked agent representation alone.
Banino teaches:
wherein the strict subsets of modified values of the agent representations are chosen at least so large that original values are not derivable from the respective masked agent representation alone.
(Banino, “[0092] As described above, the encoded representation of the current observation can be in the form of an input vector having a respective input value at each of a plurality of input positions in an input order. By contrast, the masked encoded representation masks the respective input value at each of one or more of the plurality of input positions in the input order, i.e., includes a fixed value (e.g., negative infinity, positive infinity, or another predetermined mask value) in place of the original input value at each of the one or more input positions [wherein the strict subsets of modified values of the agent representations are chosen at least so large that original values are not derivable from the respective masked agent representation alone].”)
Banino, Gao and Wayne are related to the same field of endeavor (i.e.: reinforcement learning). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Banino with teachings of Wayne and Gao for selecting actions based on learned representations of observations and sequential environmental information. (Banino, Abstract).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Gao in view of Wayne and in further view of Osindero et al., Pub. No.: US20190340509A1.
Regarding claim 4, Gao in view of Wayne teach the method of claim 1.
Gao in view of Wayne do not teach:
wherein the modifying of values of agent representations includes overwriting the values with a predetermined value
Osindero teaches:
wherein the modifying of values of agent representations includes overwriting the values with a predetermined value.
(Osindero, “[0006] … receive an observation characterizing the current state of the environment at the time step, generate the intermediate representation from the observation, provide the intermediate representation as input to the manager neural network to generate the final goal vector for the time step, provide the intermediate representation and the final goal vector as input to the worker neural network to generate the action scores, and select an action from the predetermined set of actions to be performed by the agent in response to the observation using the action scores [wherein the modifying of values of agent representations includes overwriting the values with a predetermined value].”)
Osindero, Gao and Wayne are related to the same field of endeavor (i.e.: reinforcement learning). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Osindero with teachings of Gao and Wayne to add a hierarchical reinforcement learning framework that decomposes decision making into high level goal generation and low level action selection to enable more structured control in dynamic environments. (Osindero, Abstract).
Claim(s) 5 – 6 are rejected under 35 U.S.C. 103 as being unpatentable over Gao in view of Wayne and in further view of Dai et al., Pub. No.: US20240119279A1.
Regarding claim 5, Gao in view of Wayne teach the method of claim 1.
Gao in view of Wayne do not teach:
wherein the agent representations that are modified are randomly drawn such that each agent representation is modified with a predetermined probability
Dai teaches:
wherein the agent representations that are modified are randomly drawn such that each agent representation is modified
(Dai, “[0014] … randomly select a subtree (t) in the tree representation (x.sub.tree) having a height less than the maximum subtree height (H), select a replacement subtree (r) from the subtree replacement lookup dictionary (L) [wherein the agent representations that are modified are randomly drawn such that each agent representation is modified], and replace the subtree (t) from the tree representation (x.sub.tree) with the replacement subtree (r) to generate an updated tree representation (x.sub.tree (updated)).”)
… with a predetermined probability
(Dai, “[0009] … applying the contrastive learning algorithm to the set of similar trees to obtain a deep neural network; and mapping, using the deep neural network, each tree in the list of similar trees to a corresponding vector representation having a predetermined length [… with a predetermined probability]. In some embodiments, generating the list of similar trees is performed by: receiving an anchor sequence (x), a number of replacements (1, 2, . . . , N), a maximum subtree height (H) and a subtree replacement lookup dictionary (L);”)
Dai, Gao and Wayne are related to the same field of endeavor (i.e.: reinforcement learning). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Dai with teachings of Gao and Wayne to add contrastive learning techniques for learning more robust representations by preserving structural relationships between related samples to improve the quality of learned embedding. (Dai, Abstract).
Regarding claim 6, Gao in view of Wayne teach the method of claim 1.
Gao in view of Wayne do not teach:
wherein, in each to-be-modified agent representation, the values that are modified are randomly drawn such that each value is modified with a predetermined probability
Dai teaches:
wherein, in each to-be-modified agent representation, the values that are modified are randomly drawn such that each value is modified
(Dai, [0014] “… a random choice selector operable to randomly select a random choice (n) from the number of replacements (1, 2, . . . , N); and a tree representation generator 110 configured to: from 1 to the random choice (n): randomly select a subtree (t) [wherein, in each to-be-modified agent representation, the values that are modified are randomly drawn such that each value] in the tree representation (xtree) having a height less than the maximum subtree height (H), select a replacement subtree (r) from the subtree replacement lookup dictionary (L), and replace the subtree (t) from the tree representation (xtree) with the replacement subtree (r) to generate an updated tree representation (xtree (updated)).”)
… with a predetermined probability.
(Dai, “[0009] … applying the contrastive learning algorithm to the set of similar trees to obtain a deep neural network; and mapping, using the deep neural network, each tree in the list of similar trees to a corresponding vector representation having a predetermined length [… with a predetermined probability]. In some embodiments, generating the list of similar trees is performed by: receiving an anchor sequence (x), a number of replacements (1, 2, . . . , N), a maximum subtree height (H) and a subtree replacement lookup dictionary (L);”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Dai with teachings of Gao and Wayne for the same reasons disclosed for claim 5.
Claim(s) 11 – 12 are rejected under 35 U.S.C. 103 as being unpatentable over Gao in view of Wayne and in further view of Ivanovic et al., Pub. No.: US20230074293A1.
Regarding claim 11, Gao in view of Wayne teach the method of claim 1.
Gao in view of Wayne do not teach:
further comprising the following steps: acquiring, by at least one sensor, measurement data that relates to a plurality of agents; and providing the measurement data as input data to the trained neural network system, such that the neural network system outputs predicted behavior data regarding each agent of the plurality of agents.
Ivanovic teaches:
further comprising the following steps: acquiring, by at least one sensor, measurement data that relates to a plurality of agents; and
(Ivanovic, “[0013] … For example, a computing device in a traffic infrastructure can be programmed to acquire one or more images from one or more sensors included in the traffic infrastructure system and detect objects in the images using a DNN [acquiring, by at least one sensor, measurement data that relates to a plurality of agents]. The images can be acquired from a still or video camera and can include range data acquired from a range sensor including a lidar sensor.”)
providing the measurement data as input data to the trained neural network system, such that the neural network system outputs predicted behavior data regarding each agent of the plurality of agents.
(Ivanovic, “[0014] … Determining a vehicle path upon which to operate a vehicle can be improved by predicting a plurality of future locations, i.e., trajectories, of agents in a traffic scene based on an environment around the vehicle [providing the measurement data as input data to the trained neural network system, such that the neural network system outputs predicted behavior data regarding each agent of the plurality of agents]. A trajectory is a vector that describes a direction and a speed at which an agent is moving.”)
Ivanovic, Gao and Wayne are related to the same field of endeavor (i.e.: reinforcement learning). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Ivanovic with teachings of Gao and Wayne to add uncertainty aware trajectory prediction techniques that incorporate confidence estimates into the prediction process to improve robustness. (Ivanovic, Abstract).
Regarding claim 12, Gao in view of Wayne and Ivanovic teach the method of claim 11.
Ivanovic further teaches: further comprising the following steps: determining, based on the predicted behavior data regarding each agent of the plurality of agents, an actuation signal; and actuating, using the actuation signal, a vehicle and/or a robot and/or a driving assistance system.
(Ivanovic, “[0025] … A controller may be an electronic control unit (ECU) such as is known, possibly including additional programming as described herein. The controllers may communicatively be connected to and receive instructions from the computing device 115 to actuate the subsystem according to the instructions [based on the predicted behavior data regarding each agent of the plurality of agents, an actuation signal]. For example, the brake controller 113 may receive instructions from the computing device 115 to operate the brakes of the vehicle 110 [and actuating, using the actuation signal, a vehicle and/or a robot and/or a driving assistance system].”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Ivanovic with teachings of Gao and Wayne for the same reasons disclosed for claim 11.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Iba et al., Pub. No.: US20090326710A1.
Iba teaches a behavior control system is capable of causing an agent carry out a task by smooth motions. The behavior control system makes it possible to reproduce a typical shape characteristic of a reference trajectory, i.e., the characteristic of a motion of an instructor carrying out a task, by using a first model defined on the basis of a plurality of reference trajectories representing the position of a first state variable in a time-series manner.
Hou, et al. "Graphmae2: A decoding-enhanced masked self-supervised graph learner."
Hou teaches a strategies of multi-view random re-mask decoding and latent representation prediction to regularize the feature reconstruction. The multi-view random re-mask decoding is to introduce randomness into reconstruction in the feature space, while the latent representation prediction is to enforce the reconstruction in the embedding space.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to MATIYAS T MARU whose telephone number is (571)270-0902 or via email: matiyas.maru@uspto.gov. The examiner can normally be reached Monday 8:00am - Friday 4:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to
use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization were this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA)
or 571-272-1000.
/M.T.M./ Examiner, Art Unit 2148
/MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights