DETAILED ACTION
This action is in response the communications filed on 07/02/2026 in which claims 1-2, 4-6, 8, 10, 12 and 18-20 are amended, claims 3 and 16-17 has been canceled, and claims 1-2, 4-15 and 18-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 07/06/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Objections
Claims 1 and 18-19 are objected to because of the following informalities:
In claims 1 and 18-19, “training the synthetic return neural network using an objective function” should be “training the synthetic return neural network system using an objective function”
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-2, 4-15 and 18-20 are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 1 and 18-19 recites the limitation “training the synthetic return neural network system, wherein the synthetic return neural network system comprises a contribution neural network, the training comprising: processing a plurality of previous latent representations of previous states of the environment stored in the episodic memory using a contribution neural network….” It is unclear whether the second element, 'a contribution neural network,' is actually the one that was introduced first. For examination purposes examiner has interpreted the second element “a contribution neural network” to be “the contribution neural network.”
Claims 18-19 recite the limitation “processing the observation using the computer-implemented neural network system.” There is insufficient antecedent basis for “the computer-implemented neural network system” in the claim. For examination purposes examiner has interpreted “the computer-implemented neural network system” to be “a computer-implemented neural network system.”
Claims 2, 4-15 and 20 and are also rejected due to their dependency on a rejected claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4-9, 11-15 and 18-19 rejected under 35 U.S.C. 103 as being unpatentable over Hung ("Optimizing agent behavior over long time scales by transporting value" 20191119) in view of Liu ("Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learning" 20190531)
In regard to claims 1 and 18-19, Hung teaches: A method of training a computer-implemented neural network system used to control an agent interacting with an environment to perform a task, (Hung, p. 11, Implementation and optimization "The gradient computed by each worker [training] was sent to a 'parameter server' that asynchronously ran an optimization step with each incoming gradient.")... The pseudocode for each RMA worker is presented in Supplementary Methods 2. For all experiments, we used the open source package Sonnet [a computer-implemented system] available..."; p. 2, Episodic RL "The agent’s behavior is governed [control an agent interacting with an environment to perform a task] by parameters θ, and it operates in the environment by receiving at each discrete time step t sensory observations o_t..."; p. 10, Policy "The policy module receives z_t, h_t, and m_t as inputs. The inputs are passed through a single hidden-layer MLP with 200 tanh units."; p. 10, Agent model "At a high level, the RMA consists of four modules: an encoder for processing observations at each time step; a memory augmented RNN, which contains a deep LSTM controller network... a policy that takes the state variable and the memory’s recurrent states as input to generate an action distribution;..."; Agent model or RMA worker is implemented using open source Sonnet, i.e. agent model and its hardware implementation make up a computer-implemented system. Further, agent model or RMA worker comprises policy module (a MLP) and LSTM, i.e. agent model is a neural network system.)
wherein the computer-implemented neural network system includes a synthetic return neural network system and episodic memory, the method comprising, for each of a plurality of time steps: (Hung, p. 10, Agent model "At a high level, the RMA consists of four modules: an encoder for processing observations at each time step; [for each of a plurality of time steps] a memory augmented RNN, which contains a deep LSTM controller network and an external memory [episodic memory] that stores a history of the past;... a policy that takes the state variable and the memory’s recurrent states as input to generate an action distribution;..."; p. 2, Episodic RL "We consider the setting of episodic RL, with time divided into separate trials (episodes) terminating after T time steps... The objective is to maximize the sum of rewards that the agent receives until the final time step. Let R_t ≡ r_t+ r_t+1 + ... + r_T, where r_t is the reward at time step t and R_t is called the return. [a synthetic return]"; p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, [the synthetic return] and it calculates the policy gradient based on it too."; Agent model or RMA worker operates in the Episodic RL framework, i.e. Agent model is a synthetic return system. Further, agent model or RMA worker comprises policy module (a MLP) and LSTM, i.e. agent model is a neural network system)
obtaining an observation characterizing a current state of the environment; (Hung, p. 2, Episodic RL "The agent’s behavior is governed by parameters θ, and it operates in the environment by receiving at each discrete time step t sensory observations o_t, [obtaining an observation o_t characterizing a current state of the environment]"; p. 4, The Reconstructive Memory Agent (RMA) "The agent compresses sensory observations into a vector, zt... In this model, an image I_t , the previous reward r_t-1, and the previous action a_t-1 constitute the observation o_t at time step t. These inputs are processed by encoder networks and merged into an embedding vector e_t... [obtaining an observation (e_t based on o_t) characterizing a current state of the environment]")
processing the observation using the computer-implemented neural network system to generate an action selection output to select an action to be performed by the agent at the time step; (Hung, p. 2, Episodic RL "processing those observations into an internal representation ht = h(o_0,...,o_t; θ) [processing the observation h_t or o_t] and emitting actions a_t [generate an action selection output to select an action a_t to be performed by the agent at the time step] using a policy probability distribution π(at|ht, yt; θ)"; p. 4, The Reconstructive Memory Agent (RMA) "... these outputs are combined with the embedding vector by a feedforward network into a state representation z_t = f(e_t, h_t-1, m_t-1) [processing the observation e_t] ... Finally, the state representation z_t and RNN outputs (h_t,t) are provided as input to the policy network to construct the policy distribution π(a_t|z_t, h_t,t) , which is a multinomial distribution over the discrete actions here. At each time step, an action [generate an action selection output to select an action a_t to be performed by the agent at the time step] is sampled and emitted.")
storing, in the episodic memory, a latent representation dependent on the current state of the environment; (Hung, p. 4, The Reconstructive Memory Agent (RMA) "The agent compresses sensory observations into a vector, z_t, that is both propagated to the policy to make decisions and inserted into a memory system for later retrieval [storing z_t]... the state representation z_t has the same dimension as a memory read vector. Indeed, once produced it will be inserted into the tth row of memory [storing z_t in the episodic memory] at the next time step: M_t[t, ·] ← z_t."; p. 10, Agent model "At each time step... Write the state variable z_t to slot t in the external memory.")
causing the agent to perform the selected action and, in response, receiving an actual reward characterizing a progress made in the environment as a result of the agent performing the selected action; (Hung, p. 2, Episodic RL "emitting actions a_t [causing the agent to perform the selected action] using a policy probability distribution π(at|ht, yt; θ)"; p. 4, The Reconstructive Memory Agent (RMA) "At each time step, an action [causing the agent to perform the selected action] is sampled and emitted."; p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t [an actual reward r_t characterizing a progress made in the environment as a result of the agent performing the selected action] + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T")
training the synthetic return neural network system, (Hung, p. 2, Episodic RL "We consider the setting of episodic RL... To improve performance, it is common to evaluate the episodic policy gradient, [training Episodic RL, training the synthetic return neural network system] which under fairly general conditions can be shown to have the form: ∇θV0 = ... (2) where ∇θ is the gradient with respect to θ"; p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, and it calculates the policy gradient based on it too. [training the synthetic return neural network system]")
wherein the synthetic return neural network system comprises a contribution neural network, the training comprising: (Hung, p. 10, Agent model "At a high level, the RMA consists of four modules:... a memory augmented RNN, which contains a deep LSTM controller network [RNN/LSTM, a contribution neural network] and an external memory that stores a history of the past;..."; p. 2, Episodic RL "We consider the setting of episodic RL, with time divided into separate trials (episodes) terminating after T time steps... Let R_t ≡ r_t+ r_t+1 + ... + r_T, where r_t is the reward at time step t and R_t is called the return. [the synthetic return]"; Agent model or RMA worker operates in the Episodic RL framework, and comprises policy module (a MLP), i.e. agent model is the synthetic return neural network system. Further, agent model comprises RNN/LSTM [a contribution neural network])
… a plurality of previous latent representations of previous states of the environment stored in the episodic memory… (Hung, p. 4, The Reconstructive Memory Agent (RMA) "The agent compresses sensory observations into a vector, z_t, that is both propagated to the policy to make decisions and inserted into a memory system for later retrieval [latent representation stored]... the state representation z_t has the same dimension as a memory read vector. Indeed, once produced it will be inserted into the tth row of memory [stored in the episodic memory] at the next time step: M_t[t, ·] ← z_t.")
… training the synthetic return neural network using an objective function that measures a discrepancy between: (i) the actual reward at the time step, and (ii) predicted reward for the time step; (Hung, p. 10-11, Loss functions "We combine a policy gradient loss with reconstruction objectives for decoding observations. We also have a specific loss that regularizes the use of memory for TVT. The reconstruction loss is the negative conditional log-likelihood of the observations and return, i.e., -log p(o_t, R_t|z_t)... mean-squared error... losses for the reward and the value function... Thus we have a negative conditional log-likelihood loss contribution at each time step of -log p(o_t, R_t|z_t) = α_image L_image + α_value L_value + α_rew L_ rew+ α_act L_ act [training using an objective function] where... L_value = 1/2 [||R_t - V^π(zt...)||]2... [a discrepancy between: (i) the actual reward at the time step, and (ii) predicted reward for the time step]")
processing the latent representation dependent on the current state of the environment using the synthetic return neural network system to generate a synthetic return for the current state of the environment; and (Hung, p. 10, Agent model "The value function predictor is a multi-layer perceptron (MLP) that takes in the concatenation of the state variable [processing the latent representation dependent on the current state z_t]... This function is a state-value function V^π_t ≡ V^π(z_t, [the latent representation dependent on the current state] StopGradient(logπ_t))"; p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T… we use 'truncation windows'… the return within the window is R_t = R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^k-t+1V^π(z_k+1, StopGradient(logπ_k+1)… [to generate a synthetic return for the current state]"; also see the following limitation, generating the bootstrapped return R [generate a synthetic return], which includes V. The reward is generated according to the value function in the episodic RL setting; Agent model or RMA worker operates in the Episodic RL framework, i.e. agent model is the synthetic return neural network system. )
PNG
media_image1.png
414
642
media_image1.png
Greyscale
training, through reinforcement learning, the computer-implemented neural network system using at least the synthetic return for the current state of the environment. (Hung, p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, and it calculates the policy gradient based on it too. [training, through reinforcement learning (the policy module)] The bootstrapped return can trivially be regrouped, R~t_1 := (r_t1 + αV^_t3) + γV^t1+1 [using the synthetic return], which facilitates the interpretation of the transported value as fictitious reward introduced to time t1."; Agent model or RMA worker is implemented using open source Sonnet, i.e. [a computer-implemented system]. Further, agent model or RMA worker comprises a policy module (a MLP) and LSTM, i.e. [a neural network system])
Hung does not teach, but Liu teaches: processing a plurality of previous latent representations of previous states of the environment... using a contribution neural network to generate, for each previous latent representation, a reward contribution for each of the plurality of previous latent representations; (Liu, p. 6, 2.4 Learning Temporal Credit Assignment using Language Models "To gain the dependency between the current time step t and other time steps before t, [previous] we use a encoder layer to process the state-action pairs (s_t, a_t): h_t... We represent all the T representation vectors h_t as a T by d matrix H = (h_0, h_1,…, h_T-1). [previous latent representation] To smmarize the information in H, we apply a self-attention mchanism: z = ... Some other networks we consider include the feed-forward neural network and long short-term memory network (LSTM). Please see Figure A2 to see their differences"; p. 12, Figure A2 "Network structures for the reward predictor: [generate reward] (a)Feed-forwardnetwork, (b)LSTM network... [using a contribution neural network to generate, for each previous latent representation h0, h1, h2... , a reward contribution rφ_0, rφ_1, rφ_2... for each of the plurality of previous latent representations]"; see Fig. A2 (b))
PNG
media_image2.png
324
828
media_image2.png
Greyscale
generating a predicted reward for the time step by combining the reward contributions from the plurality of previous latent representations; and (Liu, p. 6, 2.4 Learning Temporal Credit Assignment using Language Models "Vector z has size T, and each entry z_t ranges from 0 to 1, quantifying the importance of the state-action pair (s_t, a_t) in predicting the reward r(s_αt, a_αt) for interval α_t = {0, 1, ... t}. We combine the hidden representations in H using z to obtain a summarized representation h*t = z_t h_t. To predict the reward ^r(sαt, a_αt), we add one regression layer parameterized by wr and br and output the predicted reward as ^r(sαt, a_αt) = srht* + br [combining the reward contributions from the plurality of previous latent representations] Some other networks we consider include the feed-forward neural network and long short-term memory network (LSTM)...")
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Hung to incorporate the teachings of Liu by including by including the episodic return back to each time-step. Doing so would improve the learning efficiency. (Liu, p. 1, Abstract "In this work, we introduce a new algorithm for temporal credit assignment, which learns to decompose the episodic return back to each time-step in the trajectory using deep neural networks. With this learned reward signal, the learning efficiency can be substantially improved for episodic reinforcement learning.")
Claims 18-19 recite substantially the same limitation as claim 1, therefore the rejection applied to claim 1 also apply to claims 18-19. In addition, Hung teaches: (claim 18) A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations (claim 19) One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations (Hung, p. 11, Implementation and optimization "The pseudocode for each RMA worker is presented in Supplementary Methods 2. For all experiments, we used the open source package Sonnet [a computer-implemented system] available..."; the open source and its implementation inherently teach all the computer components)
In regard to claim 4, Hung teaches: wherein processing the latent representation dependent on the current state of the environment using the synthetic return neural network system to generate a synthetic return for the current state of the environment comprises: (Hung, p. 10, Agent model "The value function predictor is a multi-layer perceptron (MLP) that takes in the concatenation of the state variable [processing the latent representation dependent on the current state z_t]... This function is a state-value function V^π_t ≡ V^π(z_t, [the latent representation dependent on the current state] StopGradient(logπ_t))"; p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T… we use 'truncation windows'… the return within the window is R_t = R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^k-t+1V^π(z_k+1, StopGradient(logπ_k+1)… [to generate a synthetic return for the current state]"; also see the following limitation, generating the bootstrapped return R [generate a synthetic return], which includes V. The reward is generated according to the value function in the episodic RL setting; Agent model or RMA worker operates in the Episodic RL framework, i.e. agent model is the synthetic return neural network system. )
processing the latent representation dependent on the current state of the environment using the contribution neural network to generate a reward contribution for the current state of the environment; and (Hung, p. 4, The Reconstructive Memory Agent (RMA) "The state representation zt is provided to the RNN, alongside the previous time step’s memory read vectors t-1 to produce the next ht"; p. 10, Agent model "The value function predictor is a multi-layer perceptron (MLP) that takes in the concatenation of the state variable... This function is a state-value function V^π_t [a reward contribution for the current state] ≡ V^π(z_t, StopGradient(logπ_t))... At each time step... (1)... (2) Update the deep LSTM state with ht = LSTM (zt...) [processing the latent representation zt using the contribution neural network LSTM] (3) Construct the read key and read from the external memory; (4) Write the state variable z_t to slot t in the external memory.")
using the reward contribution for the current state of the environment for the synthetic return. (Hung, p. 5, Temporal Value Transport "The bootstrapped return can trivially be regrouped, R~t_1 [the synthetic return R] := (r_t1 + αV^_t3) + γV^t1+1 [the reward contribution V] , which facilitates the interpretation of the transported value as fictitious reward introduced to time t1. [current state]"; p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T… we use 'truncation windows'… the return within the window is R_t = R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^k-t+1V^π(z_k+1, StopGradient(logπ_k+1)…"; using value function [the reward contribution] for generating R [the synthetic return])
In regard to claim 5, Hung teaches: wherein the reward contribution for the current state of the environment presents an expected future return of future states of the environment after the current state as a result of the agent performing the selected action. (Hung, p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T [an expected future return of future states, estimate future value]…"; p. 5, Temporal Value Transport "We denote this bootstrapped return as R_t = r_t + γV^t1+1. The agent with TVT (and the other agent models) likewise bootstraps from the next time step and uses a discount factor to reduce variance further. However, it additionally bootstraps from the distant future. This initiates a splice event in which the bootstrapped return calculation for is revaluated to R_t:= r_t1 + γV^t1+1 αV^_t3, where α is a form of discount factor that diminishes the impact of future value [an expected future return of future states] over multiple stages of TVT.")
In regard to claim 6, Hung teaches: wherein the synthetic return neural network system further comprises a baseline contribution neural network, and the method further comprises: processing the latent representation dependent on the current state of the environment using the baseline contribution neural network to generate a baseline reward contribution; and (Hung, p. 10, Agent model "At a high level, the RMA consists of four modules... a decoder, [a baseline contribution neural network] which takes in the state variable and predicts the value function as well as all current observations... The decoder is composed of four sub-networks. Three of these sub-networks are matched to the encoder sub-networks of image, previous action, and previous reward. The fourth sub-network decodes the value function... The value function predictor is a multi-layer perceptron (MLP) that takes in the concatenation of the state variable... This function is a state-value function V^π_t [a baseline reward contribution] ≡ V^π(z_t, [the latent representation dependent on the current state] StopGradient(logπ_t))"
wherein summing the reward contributions includes summing the baseline reward contribution. (Hung, p. 10-11, Loss functions "On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T… we use 'truncation windows'… the return within the window is R_t = R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^k-t+1V^π(z_k+1, StopGradient(logπ_k+1)… [summing the baseline reward contribution]")
In regard to claim 7, Hung teaches: wherein the computer-implemented neural network system comprises an action selection neural network system to generate the action selection output and wherein the latent representation comprises a representation of a state of the action selection neural network system. (Hung, p. 10, Agent model [the computer-implemented neural network system, an action selection neural network system] "At a high level, the RMA consists of four modules:... a policy that takes the state variable and the memory’s recurrent states as input to generate an action distribution;... [the action selection output]"; p. 10, Policy "The policy module receives z_t [the latent representation, a state of the action selection neural network system], h_t, and m_t as inputs. The inputs are passed through a single hidden-layer MLP with 200 tanh units... The action a_t [the action selection output] is sampled and executed in the environment."; Agent model or RMA worker is implemented using open source Sonnet, i.e. agent model and its hardware implementation make up a computer-implemented system. Further, agent model or RMA worker comprises policy module that generates an action and LSTM, i.e. agent model is a neural network system and an action selection neural network system)
In regard to claim 8, Hung teaches: wherein the action selection neural network system comprises a recurrent neural network, and (Hung, p. 10, Agent model [the action selection neural network system] "At a high level, the RMA consists of four modules: an encoder for processing observations at each time step; a memory augmented RNN, which contains a deep LSTM controller network... [a recurrent neural network] a policy that takes the state variable and the memory’s recurrent states as input to generate an action distribution;...")
further comprising determining the latent representation of the current state of the environment from a hidden state of the recurrent neural network. (Hung, p. 4, The Reconstructive Memory Agent (RMA) " This RNN consists of a recurrent LSTM network h... these outputs are combined with the embedding vector by a feedforward network into a state representation z_t = f(e_t, h_t-1, m_t-1) [determining the latent representation of the current state z_t from a hidden state of the recurrent neural network h_t-1]"; p. 10, Deep LSTMs "To produce a complete output h_t, we concatenate the output vectors from each layer: h_t =[ht1 ht2]. These are passed out for downstream processing.")
In regard to claim 9, Hung teaches: wherein training, through reinforcement learning, the computer-implemented neural network system using at least the synthetic return for the current state of the environment comprises: forming a combined reward comprising a weighted combination of the actual reward and the synthetic return; and training the computer-implemented neural network system using the combined reward. (Hung, p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, and it calculates the policy gradient based on it too. [training, through reinforcement learning (the policy module)] The bootstrapped return can trivially be regrouped, R~t_1 [a combined reward, a weighted combination] := (r_t1 [the actual reward] + αV^_t3) + γV^t1+1 [the synthetic return]..., which facilitates the interpretation of the transported value as fictitious reward introduced to time t1."; Agent model or RMA worker is implemented using open source Sonnet, i.e. [a computer-implemented system]. Further, agent model or RMA worker comprises a policy module (a MLP) and LSTM, i.e. [a neural network system])
In regard to claim 11, Hung teaches: comprising training the computer-implemented neural network system using a temporal difference reinforcement learning technique. (Hung, p. 10-11, Loss functions "We combine a policy gradient loss with reconstruction objectives for decoding observations. We also have a specific loss that regularizes the use of memory for TVT... Thus we have a negative conditional log-likelihood loss contribution at each time step of -log p(o_t, R_t|z_t) = α_image L_image + α_value L_value + α_rew L_ rew+ α_act L_ act [training using an objective function] where... L_value = 1/2 [||R_t - V^π(zt...)||]2 [a temporal difference reinforcement learning technique]... On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T")
In regard to claim 12, Hung teaches: wherein training the computer-implemented neural network system using a temporal difference reinforcement learning technique comprises, for each of the time steps, training the computer-implemented neural network system using a temporal difference loss that comprises a difference between a state or state-action value output of the computer-implemented neural network system and an estimated return dependent upon at least the synthetic return. (Hung, p. 10-11, Loss functions "We combine a policy gradient loss with reconstruction objectives for decoding observations. We also have a specific loss that regularizes the use of memory for TVT... Thus we have a negative conditional log-likelihood loss contribution at each time step of -log p(o_t, R_t|z_t) = α_image L_image + α_value L_value + α_rew L_ rew+ α_act L_ act [training using an objective function] where... L_value = 1/2 [||R_t - V^π(zt...)||]2 [ a temporal difference loss that comprises a difference between a state or state-action value output and an estimated return]... On all but the standard RL control experiment tasks, we constructed the target return value as R_t = r_t + γr_t+1 + γ^2r_t+2 + ... + γ^T-tr_T"; p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, [dependent upon the synthetic return] and it calculates the policy gradient based on it too.")
In regard to claim 13, Hung teaches: comprising jointly training the computer-implemented neural network system and the synthetic return neural network system. (Hung, p. 11, Implementation and optimization "The gradient computed by each worker [training] was sent to a 'parameter server' that asynchronously ran an optimization step with each incoming gradient.")... The pseudocode for each RMA worker is presented in Supplementary Methods 2. For all experiments, we used the open source package Sonnet [the computer-implemented system] available..."; p. 2, Episodic RL "We consider the setting of episodic RL, [the synthetic return system] with time divided into separate trials (episodes) terminating after T time steps... The objective is to maximize the sum of rewards that the agent receives until the final time step. Let R_t ≡ r_t+ r_t+1 + ... + r_T, where r_t is the reward at time step t and R_t is called the return. [the synthetic return]"; p. 5, Temporal Value Transport "the agent tries to estimate the value function prediction based on this revaluated bootstrapped return, [the synthetic return] and it calculates the policy gradient based on it too."; Agent model or RMA worker is implemented using open source Sonnet, i.e. [the computer-implemented system]. Further, agent model or RMA worker comprises policy module (a MLP) and LSTM, i.e. [the neural network system], and is trained using the TVT bootstrapped return in episodic RL setting [jointly trained with the synthetic return neural network system])
In regard to claim 14, Hung does not teach, but Liu teaches: further comprising using the trained computer-implemented neural network system to select actions to be performed by the agent in the environment to perform the task. (Liu, p. 14, E Interpretability of the Learned Reward Function "We visualize the learned agent [the trained system] and reward function to demonstrate that the reward predictor could attain knowledge from interacting with environment ... In the Hopper environment, the goal is to make the 2D one-legged robot move forward as fast as possible. A successfully trained agent essentially learns to hop forward periodically."; p. 2, Reinforcement learning "The agent’s action a is selected by a conditional probability distribution πθ(a|s) parameterized by θ ∈Θ.")
The rationale for combining the teachings of Hung and Liu is the same as set forth in the rejection of claim 1.
In regard to claim 15, Hung does not teach, but Liu teaches: wherein the agent is a mechanical agent, the environment is a real-world environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. (Liu, p. 14, E Interpretability of the Learned Reward Function "We visualize the learned agent and reward function to demonstrate that the reward predictor could attain knowledge from interacting with environment [a real-world environment]... In the Hopper environment, the goal is to make the 2D one-legged robot move forward [a mechanical agent to perform a task] as fast as possible. A successfully trained agent essentially learns to hop forward periodically.")
The rationale for combining the teachings of Hung and Liu is the same as set forth in the rejection of claim 1.
Claim 10 rejected under 35 U.S.C. 103 as being unpatentable over Hung and Liu as applied to claim 1, and in further view of Blundell ("Model-Free Episodic Control" 20160614)
In regard to claim 10, Hung does not teach, but Liu teaches: wherein the latent representation of a state of the environment includes a latent representation of the action performed in the state of the environment, (Liu, p. 6, 2.4 Learning Temporal Credit Assignment using Language Models "To gain the dependency between the current time step t and other time steps before t, we use a encoder layer to process the state-action pairs (s_t, a_t): h_t... We represent all the T representation vectors h_t [a latent representation of the action a_t performed in the state] as a T by d matrix H = (h_0, h_1,…, h_T-1).")
The rationale for combining the teachings of Hung and Liu is the same as set forth in the rejection of claim 1.
Hung and Liu do not teach, but Blundell teaches: wherein the synthetic return neural network system is configured to generate a synthetic return for actions of a set of possible actions in the current state of the environment, and wherein generating the synthetic return for the current state of the environment comprises generating the synthetic return for each of the possible actions in the current state of the environment for the reinforcement learning. (Blundell, p. 3, Algorithm 1 Model-Free Episodic Control. [the synthetic return neural network system] "Receive observation ot from environment. Let st = φ(ot). [the current state of the environment] Estimate return for each action a via (2) [generate a synthetic return for actions of a set of possible actions, the synthetic return for each of the possible actions] Let at = argmaxa QEC(st,a) Take action at, receive reward rt+1")
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified Hung and Liu to incorporate the teachings of Blundell by including a model hippocampal episodic control. Doing so would attain a highly rewarding strategy and achieve a higher overall reward. (Blundell, p. 1, Abstract "Here we investigate whether a simple model of hippocampal episodic control can learn to solve difficult sequential decision making tasks. We demonstrate that it not only attains a highly rewarding strategy significantly faster than state-of-the-art deep reinforcement learning algorithms, but also achieves a higher overall reward on some of the more challenging domains.")
Response to Arguments
Applicant's amendments with respect to 35 U.S.C. 112(b) rejection have been fully considered, but the 112(b) rejection is maintained with respect to new issues.
Applicant's arguments with respect to the rejection of the claims under 35 U.S.C. 103 have been fully considered but they are moot:
Argument: (p. 10) Applicant respectfully submits that the cited portion of Ritter does not disclose or suggest "processing a plurality of previous latent representations of previous states of the environment stored in the episodic memory using a contribution neural network to generate, for each previous latent representation, a reward contribution for each of the plurality of previous latent representations," "generating a predicted reward for the time step by combining the reward contributions from the plurality of previous latent representations ," or "training the synthetic return neural network using an objective function that measures a discrepancy between: (i) the actual reward at the time step, and (ii) predicted reward for the time step," as recited by amended claim 1.
Response: the arguments do not apply to the references (Hung and Liu) being used in the current rejection.
Allowable Subject Matter
Claims 2 and 20 would be allowable if rewritten to overcome the rejection(s) under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), 2nd paragraph, set forth in this Office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Wayne (US 20210034969 A1) teaches training an unsupervised memory-based prediction system to learn compressed representations of an environment.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SU-TING CHUANG whose telephone number is (408)918-7519. The examiner can normally be reached Monday - Thursday 8-5 PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.C./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights