Detailed Action
This action is in response to the amendment filed on 03/06/2026 for application 18/194,844, in which:
Claims 1, 11, and 15 are the independent claims.
Claims 1-11, and 15 are currently amended.
Claims 1-20 are currently pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Regarding the Duplicate Claims, Warning:
Applicant’s arguments regarding the warning have been fully considered and are persuasive. Thus, the warning has been withdrawn.
Response to Arguments
Applicant's arguments filed 03/06/2026 have been fully considered but they are not persuasive.
Regarding the 35 U.S.C. § 102 Rejections:
Applicant's arguments regarding the 35 U.S.C. § 102 rejections of the previous office action have been fully considered, but are unpersuasive.
Applicant asserts (Pages 7-9), that amended independent claim 1 recites that feeding the LSTM forward output to a duration encoder to generate an edge duration output is based on a last segment of a past trajectory of one of the two or more agents. Applicant contends that Graber does not teach or suggest all features of claim 1, as amended. Specifically, Graber fails to disclose feeding the LSTM forward output to a duration encoder to generate an edge duration output based on a last segment of a past trajectory of one of the two or more agents. Applicant contends that Graber does not disclose feeding the LSTM forward output to a duration encoder to generate an edge duration output based on a last segment of a past trajectory of one of the two or more agents. It is alleged that the LSTM forward output (LSTMprior) is fed to fenc (which is interpreted by the Examiner as the duration encoder ... (where the duration encoder is described by Equation 15 of Graber)). However, the fenc described by Graber is different from the duration encoder of claim 1. There are differences between the DIDER model and Dynamic Neural Relational Inference (DNRI) models, such as the one described by Graber. The duration encoder is not a part of the DNRI model, but is a part of the DIDER model. Graber discusses the prior and the encoder but does not mention a duration encoder that generates an edge duration output based on a last segment of a past trajectory of an agent. The differences are supported by [0062] and [0051] within the specification. Equation 15 of Graber and Equation 13 of the specification identify these differences. Specifically, the superscripts for z and h are different. Therefore, Graber is silent with regard to duration encoder of claim 1 and fails to teach or suggest all features of independent claim 1.
Examiner respectfully disagrees. As noted in the updated rejection, based on a last segment of a past trajectory is interpreted as based on a period of time of a past trajectory; where Graber teaches dNRI model’s edge embeddings are based on time t where the edge duration outputs take into consideration the past predictions/historical data. Even though the equations vary, the claim only recites based on a last segment of a past trajectory where in the broadest reasonable interpretation allows for the dNRI model to be based on past trajectory timesteps (which are previous periods of time and interpreted as last segments) via previous distributions to create a final entire trajectory. This is noted within Graber, Page 4386, Column 2, Paragraph 3 , “To train the parameters … and … the encoder/prior and decoder, we proceed as follows: the input trajectories x are passed through the GNN model to produce relation embeddings … for every time t and every entity pair (i, j). These representations are input into the forward/backward LSTMs, and the prior … and approximate posterior … are computed ... At test time, we are tasked with predicting future states of the system. This means that we cannot utilize the encoder to predict edges, as we do not have the proper information about the future. Therefore, given previous predictions x1:t, we compute the prior distribution over relations … This process continues until the entire trajectory is predicted”. Although the Claims are interpreted in light of the specification, limitations from the specification are not read into the Claims. Thus, all limitations within the amended claims are explicitly taught under the broadest reasonable interpretation. The rejection below has been updated for the amended limitations.
Applicant asserts (Page 9), Claim 11 recites feeding the LSTM forward output to a duration encoder to generate an edge duration output based on a last segment of a past trajectory of one of the two or more agents. As discussed above, Graber fails to disclose this feature. As such, claim 11 should be found to define over the cited references.
Examiner respectfully disagrees. Applicant’s arguments regarding the other independent and dependent claims rely upon the same assertions as with respect to Claim 1, and are thus likewise unpersuasive. Therefore, for the reasons given above and in the rejections below, the rejection to all Claims (including Claim 1, similar independent claims, and all dependent Claims) are maintained. More specific details are discussed below within the 35 USC § 102 Rejections.
Applicant asserts (Page 9), Claim 15 recites feeding the LSTM forward output to a duration encoder to generate an edge duration output based on a last segment of a past trajectory of one of the two or more agents. As discussed above, Graber fails to disclose this feature. As such, claim 15 should be found to define over the cited references.
Examiner respectfully disagrees. Applicant’s arguments regarding the other independent and dependent claims rely upon the same assertions as with respect to Claim 1, and are thus likewise unpersuasive. Therefore, for the reasons given above and in the rejections below, the rejection to all Claims (including Claim 1, similar independent claims, and all dependent Claims) are maintained. More specific details are discussed below within the 35 USC § 102 Rejections.
Applicant asserts (Pages 9-10), For the above reasons, Applicant contends that Graber does not teach or suggest all features of claim 1. Claims 2-10, 12-14, and 16-20 depend from independent claims 1, 11, and 15, respectively, which are believed to be allowable over Graber for at least the forgoing reasons. Accordingly, the above-discussed shortcomings of Graber with respect to claim 1 are applicable to claims 2-10, 12-14, and 16-20. Therefore, withdrawal of the 35 U.S.C. §102(a)(1) rejection of claims 1-20 based on Graber is respectfully requested.
Examiner respectfully disagrees. Applicant’s arguments regarding the other independent and dependent claims rely upon the same assertions as with respect to Claim 1, and are thus likewise unpersuasive. Therefore, for the reasons given above and in the rejections below, the rejection to all Claims (including Claim 1, similar independent claims, and all dependent Claims) are maintained. More specific details are discussed below within the 35 USC § 102 Rejections.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Graber et al., “Dynamic Neural Relational Inference for Forecasting Trajectories”.
Regarding Claim 1:
Graber teaches:
A system for training a discovering interpretable dynamically evolving relations (DIDER) model, comprising: a memory storing one or more instructions; and a processor executing one or more of the instructions stored on the memory to
(Graber, Page 4386, Figure 3; Page 4387, Column 2, Paragraph 2, “Code used to implement these models and run these experiments can be found at https://github.com/cgraber/cvpr_dNRI”; Page 4386, Column 1, Paragraph 2, “.. this LSTM models the evolution of the relations between entities across time”. The method within Graber uses a dNRI model architecture which requires tracking the evolution of the relations between entities across time (interpreted by the examiner as evolving relations), depicted in Figure 3, which can be implemented with the source code from GitHub. The implementation of the code implies a processor, memory, and non-transitory computer program as they are inherent within an system for training (as taught within Section 3.5 within Graber) the dNRI model which utilizes source code to implement prediction tasks utilizing a neural network model).
perform learning a DIDER model for multi-agent interactions (Graber, Pages 4385-4386, Figures 2&3; Pages 4386, Column 2, “3.5 Training/Inference: To train the parameters φ and θ of the encoder/prior and decoder, we proceed as follows ... we always provide ground-truth states to the decoder as input during training ... process continues until the entire trajectory is predicted”. The dNRI model stands for (Dynamic Neural Relational Inference). The DIDER model is for discovering interpretable dynamically evolving relations and the dNRI is an LSTM model which discovers evolving relations via inference/prediction; thus, interpreted by the examiner as the DIDER model. Figures 2 and 3 both show the multi-agent interactions (the examiner interprets the figures displaying/denoting multi-agent interactions as the nodes (x) within both Figures show the entities/agents having multiple interactions) used to perform the learning see Section 3.5: Training/Inference within Graber) represented by a set of edge embeddings indicative of trajectory interactions between two or more agents for one or more time steps by: (Graber, Page 4386, Figures 3: “The three model components of dNRI. The inputs are fed through a fully-connected GNN to produce an embedding for every pair of entities at every time step”; Page 4384, Column 1, Paragraph 3, “To uncover interactions between entities of a system, it is common to study a surrogate task: predicting their trajectories across time … interactions between entities take the form of a latent variable zi,j”. Figures 3 shows the set of edge embeddings which indicate the correlations/interactions between at least two agents (where agents are interpreted as nodes/entities by the examiner) as shown in the Figure 2 & 3 (within the Encoder/Prior); thus, interpreted by the examiner as edge embeddings indicative of trajectory interactions between two or more agents).
feeding the set of edge embeddings to a long short-term memory network (LSTM) forward to generate an LSTM forward output; feeding the set of edge embeddings to a long short-term memory network (LSTM) reverse to generate an LSTM reverse output;
(Graber, Page 4386, Figures 3: “… The inputs are fed through a fully-connected GNN to produce an embedding for every pair of entities at every time step. These are aggregated using a forward LSTM to encode the past history of entity relations and a backwards LSTM to encode the future history of entity relations”. Figure 3 depicts the feed forward and feed backward process of feeding the aggregated x nodes (agents) based embeddings using a forward and backward LSTM to generate an LSTM forward and reverse output (LSTMprior & LSTMenc, respectively) as noted within the caption).
feeding the LSTM forward output to a duration encoder to generate an edge duration output
(Graber, Page 4386, Figures 3; Equation 15. LSTM forward output (LSTMprior) is fed to fenc (which is interpreted by the examiner as the duration encoder as it contains the history (forward prior data and backward encoder data) to generate qϕ (where the duration encoder is described by Equation 15); where qϕ is described as the encoder’s inference on the interaction types (edge types: between i and j entities/agents nodes) at each timestep; thus, interpreted by the examiner as an edge duration output as qϕ describes the encoder and prior models shared parameters (ϕ) for inferencing given observed trajectories/variables (x) over latent edge variables (z)).
based on a last segment of a past trajectory of one of the two or more agents; and
(Graber, Page 4386, Figures 3 and Equation 15; Page 4386, Column 2, Paragraph 3 , “To train the parameters … and … the encoder/prior and decoder, we proceed as follows: the input trajectories x are passed through the GNN model to produce relation embeddings … for every time t and every entity pair (i, j). These representations are input into the forward/backward LSTMs, and the prior … and approximate posterior … are computed ... At test time, we are tasked with predicting future states of the system. This means that we cannot utilize the encoder to predict edges, as we do not have the proper information about the future. Therefore, given previous predictions x1:t, we compute the prior distribution over relations … This process continues until the entire trajectory is predicted”. The dNRI model’s edge embeddings are based on time t where the edge duration outputs take into consideration the past predictions/historical data to predict the final entire trajectory by each timestep; thus, interpreted by the examiner as based on a last segment (timestep) of the nodes/entities (agents) of a past trajectory to predict the entire trajectory).
training the DIDER model based on a probability distribution for one or more different edge types obtained by feeding the LSTM forward output or the LSTM reverse output to an edge prior and an edge encoder.
(Graber, Page 4386, Figures 3, Column 2, Paragraph 3, “To train the parameters ϕ and θ of the encoder/prior and decoder, we proceed as follows: the input trajectories x are passed through the GNN model to produce relation embeddings ht(i,j),emb for every time t and every entity pair (i, j). These representations are input into the forward/backward LSTMs, and the prior pϕ(z|x) and approximate posterior qϕ(z|x) are computed …” . Different edge types are shown in Graber between i and j entities/agents nodes for each time site. Figure 3 shows the architecture for the training of the model based on the edge prior/historical distribution (pϕ); thus, corresponds to the probability distribution obtained by feeding the LSTM forward output or the LSTM reverse output to an edge prior and an edge encoder which is shown in Figure 3 in more detail for the edge types within the Encoder & Prior section).
Regarding Claim 2:
Graber teaches the system of Claim 1 and further teaches:
wherein the set of edge embeddings indicative of trajectory interactions between two or more agents is derived from a graph neural network (GNN) wherein nodes of the GNN represent the two or more agents and edges of the GNN represent relationships between two connected nodes.
(Graber, Page 4385, Column 2, Paragraph 5, “… The prior architecture which we use is as follows: the input for each time step is passed through the following GNN architecture to produce an embedding per edge per time step … This architecture implements a form of neural message passing in a graph, where vertices v represent entities i and edges e represent relations between entity pairs (i, j) …” ; Page 4386, Figures 3. Figure 3 shows the edge embeddings being derived from the GNNs (graph neural networks) where the vertices/nodes are the entities/agents and the edges are the relationships between at least two connected nodes).
Regarding Claim 3:
Graber teaches the system of Claim 1 and further teaches:
wherein the edge encoder is conditioned on full trajectories for all time steps.
(Graber, Page 4386, Column 2, Paragraph 1, “The role of the encoder is to approximate the distribution of relations at each time step as a function of the entire input, as opposed to just the past input history”’, Figure 3, Equation 15. The ‘Encoder’ within the Encoder & Prior section shown in Figure 3 depicts the edge encoder which is based on the approximate distribution of relations at each time step of the entire input unlike just input-only history; which is interpreted by the examiner as conditioned on full trajectories for all time steps (also represented within Equation 15)).
Regarding Claim 4:
Graber teaches the system of Claim 1 and further teaches:
wherein the edge prior is conditioned on an observation and a relation prediction from a previous time step.
(Graber, Page 4386, Figure 3: “… The prior is computed as a function of only the past history, while the approximate posterior is computed as a function of both the past and future. … ”; Column 1, Paragraph 2, “Fig. 3 provides an illustration of the prior model. Note that, in lieu of passing the previous relation predictions to the prior as input, we encode the dependence of the prior on the relations for previous time steps in the hidden state h(i,j),prior”; Equations 12 & 13. The ‘Prior’ within the Encoder & Prior section shown in Figure 3 depicts the edge prior which is conditioned on an observation and a relation prediction from a previous time step as this is encoded within the hidden state (Equations 12 & 13)).
Regarding Claim 5:
Graber teaches the system of Claim 1 and further teaches:
wherein the processor feeds an output of the edge encoder to a decoder to predict future states of two or more of the agents.
(Graber, Page 4386, Figure 3. Figure 3 depicts the output of the edge encoder to a decoder to predict future states of two or more of the agents which is explicitly shown between the dotted-mid-line).
Regarding Claim 6:
Graber teaches the system of Claim 5 and further teaches:
wherein the decoder includes a multi-layer perceptron (MLP).
(Graber, Page 4386, Column 1, Paragraph 1, “… Every model f is a multilayer perceptron (MLP) …”; Figure 3. Figure 3 shows the ‘Decoder’ section which comprises multiple fdec which are MLPs).
Regarding Claim 7:
Graber teaches the system of Claim 1 and further teaches:
wherein the processor trains the DIDER model based on maximizing an evidence lower bound (ELBO).
(Graber, Page 4387, Column 1, Paragraph 1, “… we calculate the ELBO: the reconstruction error is computed following Eq. (4) …”; Page 4384, Column 2, Paragraph 4, “… Following a classical VAE, the following evidence lower bound (ELBO) is maximized … (1) …”, Column 1, Paragraph 4, “… The ELBO described in Eq. (1) contains two terms: … (4) … ”. Within the training section, Graber teaches training the dNRI model based on the ELBO being maximizing (like Equation (1)) and utilizing the reconstruction loss mentioned in Equation (4)).
Regarding Claim 8:
Graber teaches the system of Claim 1 and further teaches:
wherein the LSTM reverse output is indicative of future states of the set of edge embeddings.
(Graber, Page 4386, Figures 3: “… The inputs are fed through a fully-connected GNN to produce an embedding for every pair of entities at every time step. These are aggregated using a … a backwards LSTM to encode the future history of entity relations”. Figure 3 depicts the backward LSTM to generate an LSTM reverse output LSTMenc; where LSTMenc is indicative of future states of the set of edge embeddings as it used to encode the future history of entity/agent relations).
Regarding Claim 9:
Graber teaches the system of Claim 1 and further teaches:
wherein the processor feeds a concatenation of the LSTM forward output and the LSTM reverse output to the edge encoder.
(Graber, Page 4386, Column 2, Paragraph 1, “… The final approximate posterior is then obtained by concatenating this reverse state and the forward state provided by the prior and passing the result into a MLP … (15)”, Figure 3. The LSTM forward and reverse outputs are concatenated and highlighted within Figure 3 and Equation (15)).
Regarding Claim 10:
Graber teaches the system of Claim 1 and further teaches:
wherein the edge prior or the edge encoder are implemented via a softmax function.
(Graber, Page 4386, Equations 13 and 15. Equations 13 and 15 show both the edge prior and edge encoder implementing a softmax function).
Regarding Claims 11-14:
Claims 11-14 incorporate substantively all the limitations of Claims 1-4 in a computer-implemented method and further recites no new limitations; thus, Claims 11-14 are rejected for reasons set forth in the rejections of Claims 1-4, respectively.
Regarding Claims 15-20:
Claims 15-20 incorporate substantively all the limitations of Claims 1-6 in a system and further recites no new limitations; thus, Claims 15-20 are rejected for reasons set forth in the rejections of Claims 1-6, respectively.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM RAHMAN whose telephone number is (703)756-1646. The examiner can normally be reached M-F 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/I.R./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights