DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is made non-final.
This Office action is in response to the amendments filed March 30, 2026.
Claims 1, 30, and 31 have been amended in the case.
Response to Amendment
The amendment filed March 30, 2026 has been entered. Claims 1-31 are pending in the case and have been examined, they are rejected.
Response to Arguments
Regarding the 101 Arguments
Applicant’s arguments, see pages 10-12, filed March 30, 2026, with respect to introducing a technical improvement (Step 2A Prong 2) have been fully considered and are persuasive. The rejection of December 11, 2025 has been withdrawn.
Regarding the 103 arguments
Applicant’s arguments with respect to claim(s) 1, 3-16,18-20, 22-26, 30 and 31 have been considered but are moot because the amendments necessitated the new ground of rejection. Applicant’s arguments are directed to the rejection set forth in the prior Office Action, whereas the present rejection has been modified to account for the amened claim language and further relies on the new ground of rejection for the amended language.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1, 3-16, 18-20, 22-26, 30 and 31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rakelly et al. (“Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables”, referred to as Rakelly), in view of Reed et al. (US 20200104680 A1, referred to as Reed), in view of Cabi et al. (US 11712799 B2, referred to as Cabi), in view of Cachet et al. (US 20220395975 A1, referred to as Cachet).
Regarding claim 1, Rakelly teaches, a method performed by one or more computers, the method comprising:
performing a meta reinforcement learning phase using (i)training data collected as a result of using one or more robots to perform a plurality of different robotic control tasks (Page 2, Section 2. Related Work: Discloses collecting training data in the form of trajectories of states, actions, rewards, and next states form an agent interacting with multiple tasks/environments during meta training. These trajectories correspond to training data collected as a result of an agent (robot) performing a plurality of different control tasks.; Page 4, Section 5: Outlines a meta-reinforcement learning approach to learn from multi-task buffers for training policies. ;Page 5, Algorithm 1: Describes batches of training tasks which are initialized for each training task, this corresponds with experience gathered separately for many distinct tasks from any number of separate robotic control tasks. Each training iteration computes a new actor and updates the policy parameters, using batches from the per-task replay buffers. These steps correspond to updating a control policy from the training process.)
wherein performing the meta reinforcement learning phase comprises updating a robotic control policy using the training data (Page 4-Page 5, Section 5 Off-Policy Meta-Reinforcement Learning and Algorithm 1: Describes updating policy parameters of a task-conditioned policy using off-policy training data during the meta-training phase. )
Although Rakelly teaches a method performed by one or more computers, the method comprising performing a meta reinforcement learning phase using (i)training data collected as a result of using one or more robots to perform a plurality of different robotic control tasks wherein performing the meta reinforcement learning phase comprises updating a robotic control policy using the training data, it does not teach demonstration data generated by one or more demonstrators for each of the plurality of different robotic control tasks and training an encoder network using the demonstration data.
Reed teaches, (ii) demonstration data generated by one or more demonstrators for each of the plurality of different robotic control tasks ([0044], and FIG. 4: Describes that expert trajectories, sequences of expert observations that serve as expert demonstrations of how a task can be accomplished Figure 4 402 and 406. These expert trajectories constitute demonstration data generated by one or more demonstrations of the task.), and training an encoder network using the demonstration data ([0030]: Describes training a network using expert demonstrations “the encoder neural network is trained, using the method of the first aspect, on training data including a set of expert trajectories”. ; [0041]: Describes that the system trains an encoder neural network based on observations.) …
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Rakelly meta-reinforcement learning framework with Reed’s demonstration trajectories. Doing so would have enabled the system to exploit expert demonstrations to produce more informative task context representations, improving the encoders’ ability to capture task specific features, enhancing the policy training.
Rakelly in view of Reed teaches, wherein the robotic control policy is conditioned on task-specific context information generated by the encoder network (Rakelly page 3, Section 4.1: Describes teaching a policy conditioned on latent context variable z produced by an encoder qφ(z | c) from task data. The PEARL setup is the policy being conditioned on a task embedding/context variable. ; Reed [0018], [0021-0022] and [0062-0063]: Describes a context latent representation as a task specific context information which is generated from encoder outputs used downstream in the training pipeline.)based on the demonstration data (Reed, [002-0027]: Describes that the encoder neural network is trained on training data which comprises a plurality of expert trajectories which processes the expert observations form those trajectories to generate latent representations) while the robotic control policy is being updated using the training data (Rakelly, Page 4-5, Section 5 and Algorithm 1: Describes Meta-RL algorithm updates the policy parameters using training data (off-policy RL transitions) while the policy depends on the context. Where the training data updates the policy, which uses context z from encoder during training.; Reed describes training the policy network using RL/imitation learning with the demonstration-based embedding as an input.)
Although Rakelly in view of Reed teaches, a method performed by one or more computers, the method comprising performing a meta reinforcement learning phase using (i)training data collected as a result of using one or more robots to perform a plurality of different robotic control tasks and (ii) demonstration data generated by one or more demonstrators for each of the plurality of different robotic control tasks, wherein performing the meta reinforcement learning phase comprises updating a robotic control policy according to using the training data and training an encoder network using the demonstration data ,wherein the robotic control policy is conditioned on task-specific context information generated by the encoder network based on the demonstration data while the robotic control policy is being updated using the training data, including iteratively updating the encoder network including iteratively updating the encoder network by processing each demonstration of the plurality of demonstrations, thereby training the encoder network to learn environmental features of successful task runs it does not teach performing an adaptation phase using a plurality of demonstrations for the particular task.
Cabi teaches performing an adaptation phase using a plurality of demonstrations for the particular task (Col. 7, lines 19-30: Describes gathering multiple demonstration trajectories that are specific to the task the robot is about to learn, corresponding to a plurality of demonstrations for the task. This phase is describes taking place after a pretraining phase, corresponding to a separate adaptation phase from learning.)
using the robotic control policy and the encoder network to control a robot to perform the particular task (Col. 1, lines 28-36: Describes that the system is used to design policies to control robots.; Col. 2, lines 12-24, and FIG. 1: Describes that a robot receives actions form the policy of the neural network which has been trained by the system.; Col. 6, lines36-39: Describes that a robot uses the trained policy to perform the task needed.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Rakelly’s in view of Reed’s iterative training encoder, with Cabi’s demonstration trajectories. Doing so would allow the demonstrations to be fed in the encoder, and refined one at a time learning contextual cues from successful runs.
Although Rakelly in view of Reed, in view of Cabi teaches performing an adaptation phase using a plurality of demonstrations for the particular task, and using the robotic control policy and the encoder network to control a robot to perform the particular task. They do not teach, to determine trained parameter values of the encoder network, and including iteratively updating the trained parameter values of the encoder network by processing each demonstration of the plurality of demonstrations, thereby training the encoder network to learn updated parameter values of the encoder network from the trained parameter values, wherein the encoder network is configured to generate, in accordance with the updated parameter values, new values for the task-specific context information for the particular task.
Cachet teaches, to determine trained parameter values of the encoder network ([0034-0035]: Describes a training module configured to input a set of demonstrations for a task different than training tasks And train weight parameters of encoder modules of a transformer architecture based on a multidimensional tensor generated from the input set of demonstrations, it further describes training the weight parameters of the encoder modules based on maximizing an average return of the policy network.; [0037-0040]: Describes that the transformer architecture includes an encoding module configured to generate the multidimensional tensor based on the set of demonstrations, where in each demonstration includes a time series of observations, and each observation includes at least one of a state action pair, state, position, image, or measurement.)
including iteratively updating the trained parameter values of the encoder network by processing each demonstration of the plurality of demonstrations, thereby training the encoder network to learn updated parameter values of the encoder network from the trained parameter values, wherein the encoder network is configured to generate, in accordance with the updated parameter values, new values for the task-specific context information for the particular task ([0034-0042]; Describes a demonstration condition to reinforcement learning system in which a training module inputs a set of demonstrations for a task that is different than the training tasks and trains weight parameters of encoder modules of a transformer architecture based on a multi-dimensional tensor generated from the input set of demonstrations. The weight parameters of the encoder modules are trained based on maximizing an average return of the policy network. The transformer architecture includes an encoding module configured to generate the multidimensional tensor based on the set of demonstrations, and that each demonstration includes a time series of observations, Such as a state action pair, state, position, image, or measurement. The task may include manipulating an object or navigating toward a target position, where the demonstrations include positions and/or orientations of a robot.; [0083-0089]:Describes that each iteration of the DCRL training loop includes populating a replay buffer and updating model parameters, and that the model parameters are updated using a reinforcement learning algorithm.; [0091-0097]: Describes a demonstration conditioned policy structure in which an embedding function maps a collection of demonstrations to an embedding space, and the policy maps histories and embeddings to action probabilities. The policy network includes an encoder module and a decoder module, where the encoder module maps a collection of demonstrations to an embedding, and the decoder module treats the embedding as context and determines an action and value based on the embedding and the agent’s history. Therefore, the updated/trained encoder module parameters are used to generate a demonstration derived embedding/context for the particular task, and the policy uses that embedding/context to determine actions.
Corresponding to updating trained encoder/module parameter values based on processing demonstrations for a task and using the resulting encoder generating embedding/context for determining actions for that task.)
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to incorporate Cachet’s demonstration based encoder parameter training into the meta reinforcement learning system of Rakelly, in view of Reed, in view of Cabi. Doing so would have enabled the system to select the appropriate robot control actions for the new task using fewer demonstrations and less task-specific training.
Regarding claim 3, Rakelly further teaches, wherein the meta reinforcement learning phase comprises performing offline reinforcement learning (Page 4, Section 5 and Algorithm 1: Describes an off-policy meat-RL phase by sampling many tasks from p(T) and jointly updating an actor, critic and task-encoder. Since the algorithm is off-policy these updates can be computed entirely from previously collected replay buffers, being done so offline without access to new data.).
Regarding claim 4, Rakelly further teaches, wherein performing the meta reinforcement learning phase comprises: maintaining, at one or more replay buffers and for each of the plurality of different robotic control tasks, a plurality of transitions that each represent a past experience of controlling the robot to perform the different robotic control task; for each of multiple training steps and for each of the plurality of different robotic control tasks: sampling one or more transitions from the plurality of transitions for the robotic control task (Page 4, Section 5 cont. Page 5, and Algorithm 1: Describes that it creates a distinct buffer per task which continuously stores every transition, the algorithm further draws batches from the tasks own buffer for each step. Then at every optimization step, draws both a context set and an RL minibatch from the same tasks replat buffer.); determining, for each of the one or more sampled transitions, a corresponding learning target that is dependent on respective values of one or more context variables determined based on using the encoder network, wherein the one or more context variables represent context information that is specific to the task (Page 4, Section 5 cont. Page 5, and Algorithm 1: Describes that the encoder infers a latent context z from each context batch, that z is then fed into the critic loss to compute the Bellman target Q-values for every sampled transition corresponding to coupling the learning target to the context variable.); and determining an update to the current values of action selection network parameters of an action selection neural network that implements the robotic control policy, wherein the update enables the action selection neural network to generate the action selection outputs that result in actions being selected that improve the estimate of the return that would be received if the robot performed the selected actions in response to the current observation, while constraining the selected actions according to past experience represented by the sampled transitions (Page 4, Section 5 cont. Page 5, and Algorithm 1: “The actor and critic are trained with batches of transitions drawn uniformly from the entire replay buffer.” The actor-loss updates the policy parameters to maximize actions with higher Q-values under a KL regulariser that keeps the new action distribution close to the behavior implied by the replay buffer. Since these are implemented with the off-policy data, the learned policy is constrained to remain near the support of the stored transitions while improving expected return.).
Regarding claim 5, Rakelly further teaches, wherein for each of the plurality of different robotic control tasks, each transition comprises: (i) a current observation characterizing a current state of the environment (Page 3, Section 3: Describes that the replay buffer contains a collecti0on of transitions, where st denoted the current state observation); (ii) a current action performed by the robot in response to the current observation (Page 3, Section 3: Describes that each stored transition records at where the action is taken by the policy given from the observation st.); (iii) a next observation characterizing a next state of the environment after the robot performs the current action(Page 3, Section 3: Describes that the tuple includes st+1 representing the subsequent state observed after executing action at.); and (iv) a current reward received in response to the robot performing the current action (Page 3, Section 3: Describes that each transition stores rt, the scalar reward returned by the environment upon taking action at in state st.).
Regarding claim 6, Rakelly further teaches, wherein sampling the one or more transitions from the plurality of transitions for the robotic control task comprises: determining a respective value for each of the one or more context variables for the robotic control task, comprising processing an encoder network input that includes a sampled transition using the encoder network having a plurality of encoder network parameters and in accordance with current values of the encoder network parameters to generate a predicted distribution over a set of possible values for each of the one or more context variables (Page 5, Algorithm 1: Describes sample contexts built from sampling transitions form the task replay buffer, to form encoder input. Which is then passed through the inference network returning parameters of a Gaussian distribution over the task-specific variable. The sample z form this predicated distribution provides the respective value of the context variable.).
Regarding claim 7, Rakelly further teaches, wherein the learning target comprises a target Q value, and wherein determining the corresponding target Q value for each of the one or more sampled transitions comprises: processing a value network input that includes (i) the next observation included in the transition and (ii) the one or more context variables having the respective determined values using a value neural network having a plurality of value network parameters and in accordance with current values of the value network parameters to generate a predicted Q value that is an estimate of a return that would be received by the robot starting from the next state characterized by the next observation included in the transition (Page 5, Section 5.1: Describes “The critic loss can then be written as…” the term (r + V(s,z)) is the target Q value for each sampled transition, the value network input to V is the next observation s together with the latent context z, this computes a Bellman target Q by forward passing (S, z) through the target critic V. Every transition sampled from the replay buffer feeds the next state observation and the task context latent z into the value network. Adding the immediate reward yields a target Q that the critic is regressed toward.).
Regarding claim 8, Rakelly further teaches, wherein the method further comprises, for each of multiple training steps and for each of the plurality of different robotic control tasks: determining an update to the current values of the value network parameters based on optimizing a value objective function that measures, for the each of the one or more sampled transitions, a difference between the learning target and a predicted Q value, wherein the predicted Q value is generated by using the value neural network and in accordance with the current values of the value network parameters to process a value network input that includes (i) the current observation included in the transition and (ii) the one or more context variables having the respective determined values (Page 5, Section 5.1 and Algorithm 1: Describes that for each meta-training step every task calculates a critic loss that is the squared difference between the target Q value (r + V(s,z)) and the critics current prediction Qϴ(s; a; z). Then updates the value-network parameters by taking a gradient step that minimizes that loss. This uses an input tuple for the current observation and the encoder derived context variable when generating Qϴ(s; a; z)).
Regarding claim 9, Rakelly further teaches, wherein determining the update to the current values of the action selection network parameters comprises: determining the update based on optimizing an action selection objective function that includes a term dependent on an advantage value estimate for the current state characterized by the current observation included in each of the one or more sampled transitions (Page 5, Section 5.1: Describes a derivation built on a soft actor critic algorithm (SAC), which the normalization exp(Q)/Z(S) subtracts the log-partition (soft value) from Q the difference being an advantage.)The policy gradient step updates the actor parameters by minimizing the L_actor, whose gradient magnitude is directly weighted by the advantage term, which corresponds to optimizing the action selection objective including terms dependent on advantage value estimates.).
Regarding claim 10, Rakelly further teaches, wherein the method further comprises, for each of multiple training steps and for each of the plurality of different robotic control tasks: determining, based on optimizing an encoder objective function that measures at least a difference between the predicted distribution generated by the encoder network and a predetermined distribution for each of the one or more context variables (Page 3, Section 4.1: Describes an equation which is an encoder objective function, comprising a KL term computing the difference (KL divergence) between the encoders predicted posterior and a predetermined distribution.), an update to the current values of the encoder network parameters that constrains mutual information between the context information represented by the one or more context variables and information contained in the one or more sampled transitions (Page 3, Section 4.1: “The KL divergence term can also be interpreted as the result of a variational approximation to an information bottleneck (Alemi et al., 2016) that constrains the mutual information between Z and c.” Describes that the same KL term serves as an information bottleneck regulariser, which limits the mutual information between the latent context Z and the transition set c. ; Page 5, Algorithm 1: Describes the gradient step computed form the very objective that includes the KL term.).
Regarding claim 11, Rakelly further teaches, wherein the action selection neural network is configured to process an action selection network input that includes (i) the current observation included in the sampled transition and (ii) the one or more context variables in accordance with current values of the action selection network parameters to generate the action selection output (Page 5, Algorithm 1: Describes πϴ(a|s, z) which shows that the action selection network takes two inputs: the current observation s from the sampled transition and the context variable z produced from the encoder. The network then outputs the action distribution.).
Regarding claim 12, Cabi further teaches, wherein the action selection network input(Described above in claim 11) also includes data specifying each action in a set of possible actions that can be performed by the robot (Col. 2, lines 60-65: Describes that every action vector is a command the robot can execute.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Rakelly’s iterative training encoder with Cabi’s demonstration trajectories. Doing so would allow robot command functions to abstract action vectors to facilitate robotic functionality in real time by enabling the encoder and policy networks to jointly reason over state and action contexts.
Regarding claim 13, Cabi further teaches, wherein the action selection output includes a respective numerical probability value for each action in the set of possible actions that can be performed by the robot (Col. 2, lines 35-38: Describes that each action in the robot’s action set is assigned a numeric probability.).
Regarding claim 14, Rakelly further teaches, wherein determining the respective value for each of the one or more context variables for the robotic control task further comprises, for each of the one or more context variables: determining a combined predicted distribution from the predicted distributions generated by using the encoder neural network from processing the encoder network inputs that each include a respective sampled transition (Page 4, Section 4.1 Equation (2): Describes an inference network which models the latent context posterior as the product of per-transition factors, so that each observed transition jointly refines the context estimates. Corresponding to computing combined predicted distribution by multiplying individual context distributions form each sampled transition.).
Regarding claim 15, Rakelly further teaches, wherein determining the combined predicted distribution comprises computing a product of the predicted distributions (Page 4, Section 4.1 Equation (2): Describes that the inference network forms the combined context posterior by taking the product of each per-transition distribution factor.).
Regarding claim 16, Rakelly further teaches, wherein determining the respective value for each of the one or more context variables for the robotic control task further comprises, for each of the one or more context variables: sampling a respective value in accordance with the combined predicted distribution (Page 5, Algorithm 1 and Algorithm 2: Describes a training and testing procedure which sample a context variable z from the inferred posterior qφ(z|c) each context variable is obtained by sampling from the combined posterior.).
Regarding claim 18, Cabi further teaches, wherein the value network input also includes data specifying a possible action that can be performed by the robot (Col. 3, lines 13-20: Describes that a critic network takes both the current observation and a candidate action as inputs, producing a Q values estimate for that state action pair.).
Regarding claim 19, Rakelly further teaches, wherein the predetermined distribution is a unit Gaussian distribution (Page 3, Section 4.1, Equation (1): Describes a predetermined distribution p(z) to be a standard normal/unit Gaussian, when applying the KL-divergence term in its encoder objective. By defining p(z) = N(0,I).).
Regarding claim 20, Rakelly further teaches, wherein the encoder objective function also measures, for the each of the one or more sampled transitions, the difference between the target Q value and the predicted Q value (Page 5, Algorithm 1: Describes a context encoder parameter φ which are updated via (line 19, Algorithm 1) where equation 3 (page 5) is a per transition (Qpred – Qtraget)2 term. This corresponds to the encoder loss including the Q-value regression component).
Regarding claim 22, Rakelly further teaches, wherein the difference between the predicted distribution and the predetermined distribution is computed as a Kullback-Leibler (KL) divergence (Page 3, Section 4.1: Describes an encoder objective measuring gaps between the inferred posteriors and the prior p(z) using the Kullback-Leibler divergence DKL(q||p) computing the differences between predicted distributions.).
Regarding claim 23, Cabi further teaches, wherein control the robot to perform the particular task comprises causing the robot to perform actions selected by using the robotic control policy (Col 6, lines 36-40: Describes that, once the policy network produces its action selection outputs, the system uses those outputs to command the robot to carry out those actions in the real environment.).
Regarding claim 24, Rakelly further teaches, wherein the encoder network and the action selection neural network are trained on different sampled transitions (Page 5, Algorithm 1: Describes two distinct batches for each training step, c for encoder updates and Bi for actor/critic updates. because Sc draws from a separate context buffer the encoder and the action selection (actor/critic) networks are trained on different sampled transitions.).
Regarding claim 25, Rakelly further teaches, obtaining a plurality of demonstration transitions generated by a demonstrator for the particular task (Page 3, Section 4.1: Describes “or train a value function, it is enough to have access to a collection of transitions” with which the process can train a model.); and using the plurality of demonstration transitions to adjust the current values of the action selection network parameters, comprising determining a respective value for each of the one or more context variables for the particular robotic control task based on using the encoder network to process an encoder network input that includes a demonstration transition in accordance with trained values of the encoder network parameters (Page 5, Algorithm 1: Describes an iterative sample z ~ qφ(z|C) showing that the encoder network process the collected transitions C to produce the context variable z.; Page 5, Algorithm 2: Shoes that the updates for the policy network πϴ(a|s,z) using those context transitions, corresponding to adjusting the current values of the action selection network parameters, based on the demonstrations and inferred context.).
Regarding claim 26, Rakelly further teaches, wherein the particular task is different from any of the plurality of different robotic control tasks (Page 5, Algorithm 2: Describes that the meta-test task T is sampled anew from the task distribution p(T), ensuring it is distinct from the training tasks.).
Regarding claim 30, which recites substantially the same limitations as claim 1. Claim 30 further recites, a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations to train a robotic control policy to perform a particular task(Cabi et al., Col. 12, lines 10-31: Describes a computer system comprising hardware such as storage devices, to execute stored instructions for operations of training.) to perform the method steps of claim 1, respectively, and is therefore rejected on the same premise.
Regarding claim 31, which recites substantially the same limitations as claim 1. Claim 20 further recites, a non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations to train a robotic control policy to perform a particular task(Cabi et al., Col. 12, lines 10-31: Describes a computer system comprising hardware such as storage devices, to execute stored instructions for operations of training.) to perform the method steps of claim 1, respectively, and is therefore rejected on the same premise.
Claim(s) 2, 17, and 27-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rakelly et al. (“Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables”, referred to as Rakelly), in view of Reed et al. (US 20200104680 A1, referred to as Reed), in view of Cabi et al. (US 11712799 B2, referred to as Cabi), in view of Cachet et al. (US 20220395975 A1, referred to as Cachet), in view of Nair et al. (“AWAC: Accelerating Online Reinforcement Learning with Offline Datasets”, referred to as Nair).
Regarding claim 2, Nair teaches, performing a fine-tuning phase for the particular task including continually updating the robotic control policy according to experience data gathered in the operating environment (Page 1, Section 1: Describes an offline initialization with a later fine-tune stage, demonstrating two sequential phases on the same task.; Page 4, Section 4, and Page 5, Algorithm 1: Describes that every iteration appends new experience and immediately performs gradient updates on the policy, corresponding to iteratively reefing the control policy with the collected data. Algorithm 1 further shows that the policy gathers new on-policy experience in the real or simulated environment corresponding to data gathered in an operating environment.).
Regarding claim 17, Nair further teaches, wherein the advantage value estimate for the current state characterized by the current observation is computed as a difference between (i) the predicted Q value for the current state that is generated by using the value neural network from processing the value network input and (ii) a predicted state value for the current state that is an estimate of a return resulting from the environment being in the current state (Page 2, Section 2: Describes an advantage function Aπ(s, a) = Qπ(s, a) − V π(s), where Qπ(s, a) is the action-value estimate and V π(s) is the state-value estimate.).
Regarding claim 27, Nair teaches, wherein constraining the selected actions according to the current actions included in the sampled transitions comprises: encouraging the selected actions to stay close to the current actions included in the sampled transitions (Page 4, Equations (8), and (9): Describes a KL-constraint that the updated policy π must remain close the behavior policy πβ, which shows past actions in the replay buffer.).
Regarding claim 28, Nair teaches, wherein the particular robotic control task is a dexterous manipulation task (Page 8, Section 6: Describes a dexterous manipulation for a robotic hand to execute particular tasks given to the robot with precision.).
Regarding claim 29, Nair teaches, wherein the dexterous manipulation task comprises one of: a valve rotation task, an object repositioning task, or a drawer opening task performed by a robotic arm (Page 8, Figure 6: Describes a dexterous manipulation for a robotic hand to execute rotating a value 180 degrees, opening a drawer and re-positioning a object to the center of a table.).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Rakelly’s meta-learning, with Reed’s demonstrators, with Cabi’s demonstration-driven policy tuning to incorporate Nair’s KL advantage weight updates, and dexterous task manipulation. Doing so would allow rapid, data-efficient adaptation while maintaining stable policy improvement and high performance on specified manipulation benchmarks.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rakelly et al. (“Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables”, referred to as Rakelly), in view of Reed et al. (US 20200104680 A1, referred to as Reed), in view of Cabi et al. (US 11712799 B2, referred to as Cabi),Cachet et al. (US 20220395975 A1, referred to as Cachet), in view of Mitchell et al. (“Offline Meta-Reinforcement Learning with Advantage Weighting”, referred to as Mitchell).
Regarding claim 21, Mitchell teaches, wherein the action selection objective function is of the form log(w)exp( A), where w is action selection output, A is the advantage value estimate, and Aa tunable hyperparameter (Page 2, Equation 1: Describes the “Advantage-Weighted Regression” which presents the policy objective (Equation 1), where RB(S,A) – Vψ(S) is the advantage A and T is a tunable temperature (λ) corresponding to log(π)(
exp
1
λ
A
).
It would have been obvious to one of ordinary skill in the art at the time of the claimed invention to combine Rakelly’s meta-learning with Reed’s demonstrators, with Cabi’s demonstration-driven policy tuning to incorporate Mitchell’s advantage weighted regression. Doing so would enable context aware advantage weighting to further boost sample efficiency and stability during policy adaptation.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. See attached PTO-892 for additional art including.
US 20210205988 A1 : task embedding network
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONALD T RODEN whose telephone number is (571)272-6441. The examiner can normally be reached Mon-Thur 8:00-5:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T.R./Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights