Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner notes the entry of the following papers:
Amended claims filed 10/16/2026.
Applicant arguments/remarks made in amendment filed 10/16/2026.
Claims 9 and 23 are cancelled. Claims 1, and 14-15 are amended. Claims 1-8, 10-22, and 24-27 are pending.
Response to Arguments
Applicant presents arguments. Each is addressed.
Applicants’ arguments that the prior art of record do not teach the amended claims are moot in view of new areas of the prior art found that teach the amended limitations.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-22, and 24-27 are rejected under 35 U.S.C. § 103 as being unpatentable over Yu, et al (Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning, herein Yu), Mishra, et al (A Simple Neural Attentive Meta-Learner, herein Mishra), Finn, et al (One-Shot Visual Imitation Learning via Meat-Learning, herein Finn), and Koskinopoulou, eta al (Learning from Demonstration Facilitates Human-Robot Collaborative Task Execution, herein Koskinopoulou).
Regarding claim 1,
Yu teaches a training system for a robot comprising: a model [having a transformer architecture] and configured to determine how to actuate at least one of arms and an end effector of the robot (Yu, Figure 1, and, page 3, paragraph 7, line 1 “Meta-reinforcement learning aims to leverage the set of training task to learn a policy π(a|s, z), that can quickly adapt to new test tasks that were not seen during training, where both training and test tasks are assumed to be drawn from the same task distribution p(T).” and, page 6, paragraph 6, line 10 “For the meta-RL evaluation, we study three algorithms: RL2 [18, 19]: an on-policy meta-RL algorithm that corresponds to training a LSTM network with hidden states maintained across episodes within a task and trained with PPO, model-agnostic meta-learning (MAML) [10, 21]: an on-policy gradient-based meta-RL algorithm that embeds policy gradient steps into the meta-optimization, and is trained with PPO, and probabilistic embeddings for actor-critic RL (PEARL) [22]: an off-policy actor-critic meta-RL algorithm, which learns to encode experience into a probabilistic embedding of the task that is fed to the actor and the critic.”
PNG
media_image1.png
638
1051
media_image1.png
Greyscale
In other words, Meta-reinforcement learning is a training system for a robot, and training tasks from Figure 1, are determine how to actuate at least one of arms and effector of the robot.) ;
a training dataset including sets of demonstrations for the robot to perform training tasks (Yu, Figure 1, In other words, Figure 1 shows a training dataset including sets of demonstrations for the robot.), respectively,
[each of the demonstrations including a temporal sequence {observation; action} tuples of successful behavior of one of the tasks],
[each observation including coordinates of joints of the robot, a pose of the end effector]; and
a training module including one or more processors and memory including code executed by the one or more processors to: (a) meta-train a policy of the model using first ones of the sets of demonstrations for first ones of the training tasks (Yu, Figure 1, and, abstract, line 10 “In this paper, we propose an open-source simulated benchmark for meta-reinforcement learning and multitask learning consisting of 50 distinct robotic manipulation tasks. Our aim is to make it possible to develop algorithms that generalize to accelerate the acquisition of entirely new, held-out tasks. We evaluate 6 state-of-the-art meta-reinforcement
learning and multi-task learning algorithms on these tasks.” Examiner notes, that it is implicit that a meta-reinforcement learning algorithm requires one or more processors and memory including code executed by the one or more processors in order to execute. In other words, evaluate 6 state-of-the-art meta-reinforcement learning and multi-task algorithms is one or more processors and memory including code executed by the one or more processors, and meta-reinforcement learning is meta-train a policy for sets of the demonstrations.), respectively; and
(b), after (a) optimize the policy of the model using second ones of the sets of demonstrations for second ones of the training tasks (Yu, page 6, paragraph 6, line 10 “For the meta-RL evaluation, we study three algorithms: RL2 [18, 19]: an on-policy meta-RL algorithm that corresponds to training a LSTM network with hidden states maintained across episodes within a task and trained with PPO, model-agnostic meta-learning (MAML) [10, 21]: an on-policy gradient-based meta-RL algorithm that embeds policy gradient steps into the meta-optimization, and is trained with PPO, and probabilistic embeddings for actor-critic RL (PEARL) [22]: an off-policy actor-critic meta-RL algorithm, which learns to encode experience into a probabilistic embedding of the task that is fed to the actor and the critic.” In other words, policy is policy, and after the policy is produced, meta-optimization is optimize the policy.), respectively; and
(c) after (b), fine-tune the model using third ones of the training tasks to adapt the model to perform a target task different from the training tasks (Yu, Fig. 3, and, page 3, paragraph 7, line 1 “Meta-reinforcement learning aims to leverage the set of training task to learn a policy π(a|s,z) that can quickly adapt to new test tasks that were not seen during training, where both training and test tasks are assumed to be drawn from the same task distribution p(T ). Typically, the training tasks are referred to as the meta-training set, to distinguish from the adaptation (training) phase performed on the (meta-) test tasks. During meta-training, the learning algorithm has access to M tasks {Ti}Mi=1 that are drawn from the task distribution p(T). At meta-test time, a new task
PNG
media_image2.png
26
100
media_image2.png
Greyscale
is sampled that was not seen during meta-training, and the meta-trained policy must quickly adapt to this task to achieve the highest return with a small number of samples.”
PNG
media_image3.png
444
928
media_image3.png
Greyscale
Examiner notes the specification of the instant application recites “Optimization based meta-learning produces an initialization of a policy to be efficiently fine-tuned on a test task from a limited amount of demonstrations. In this approach, the training module 200 trains the model 124 using an available collection of demonstrations associated with a set of training tasks (in the training dataset 204). In this case, the policy determines an action with respect to the current observation. At test time, the policy is fine-tuned using the available demonstrations of the target task.” (Specification, paragraph [0058], line 2.) The specification also recites “Another approach is to learn the initialization of a network, which is fine tuned at test time on the new task.” (Specification, paragraph [0068], line 1.) Based on this, Examiner is interpreting that “fine tuned at test time” means adapting the model during the testing phase to new tasks that weren’t included in the training data for the training phase. Further, there is no mention of “third ones” in the specification. There are sets of demonstrations in the training dataset called “first ones” and “second ones” but no “third ones”. Examiner is interpreting that “third ones” are new tasks used for fine-tuning during testing. In other words, meta-test time tasks is third ones of the dataset, new task is target task, and, from Figure 3, adapt to new tasks is fine-tune to adapt the model to perform a target task different from the training tasks.) , wherein
[the sets of demonstrations for the training tasks each include more than one demonstration and less than a first predetermined number of demonstrations].
Thus far Yu does not explicitly teach a model having a transformer architecture.
Mishra teaches a model having a transformer architecture (Mishra, page 17, paragraph 5, line 1 “Next, we considered a SNAIL agent without TC layers (only attention). Due to the sequential nature of RL tasks, we employed the positional encoding proposed by Vaswani et al. (2017b). This model, which is equivalent to their Transformer architecture…” In other words, transformer architecture is a model having a transformer architecture.)
Both Yu and Mishra are directed to meta-RL learning for robots, among other things. Yu teaches a training system for a robot, comprising: a model configured to determine how to actuate at least one of arms and an end effector of the robot and a training dataset including sets of demonstrations for the robot to perform training tasks, respectively, but does not explicitly teach the model having a transformer architecture. Mishra teaches a model having a transformer architecture.
In view of the teaching of Yu, it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Mishra into Yu. This would result in a training system for a robot, comprising: a model having a transformer architecture configured to determine how to actuate at least one of arms and an end effector of the robot and a training dataset including sets of demonstrations for the robot to perform training tasks.
One of ordinary skill in the art would be motivated to do this because neural networks have had difficulties adapting quickly to changes in tasks (Mishra, abstract, line 1 “Deep neural networks excel in regimes with large amounts of data, but tend to struggle when data is scarce or when they need to adapt quickly to changes in the task. In response, recent work in meta-learning proposes training a meta-learner on a distribution of similar tasks, in the hopes of generalization to novel but related tasks by learning a high-level strategy that captures the essence of the problem it is asked to solve. However, many recent meta-learning approaches are extensively hand-designed, either using architectures specialized to a particular application, or hard-coding algorithmic components that constrain how the meta-learner solves the task. We propose a class of simple and generic meta-learner architectures that use a novel combination of temporal convolutions and soft attention; the former to aggregate information from past experience and the latter to pinpoint specific pieces of information.”).
Thus far, the combination of Yu and Mishra does not explicitly teach each of the demonstrations including a temporal sequence {observation; action} tuples of successful behavior of one of the tasks, or, the sets of demonstrations for the training tasks each include more than one demonstration and less than a first predetermined number of demonstrations.
Finn teaches each of the demonstrations including a temporal sequence {observation; action} tuples of successful behavior of one of the tasks (Finn, page 2, paragraph 6, line 1 “We consider a policy π that maps observations O to predicted actions
PNG
media_image4.png
17
16
media_image4.png
Greyscale
. During meta-learning, the
policy is trained to adapt to a large number of tasks. Formally, each imitation task Ti = { τ =
PNG
media_image5.png
28
434
media_image5.png
Greyscale
consists of demonstration data τ generated by an expert policy π*i and a loss function L used for imitation.” In other words, each imitation task Ti is mapped to a sequence of o1,a1 pairs, is each of the demonstrations including a temporal sequence {observation; action} of tuples.)
Finn teaches the sets of demonstrations for the training tasks each include more than one demonstration and less than a first predetermined number of demonstrations (Finn, Algorithm 1, and page 1, paragraph 3, line 9 “For the first time, we demonstrate that vision based policies can be fine-tuned end-to-end from one demonstration, using meta-learning as a pre-training procedure that uses demonstrations on a diverse range of other environments.”
PNG
media_image6.png
450
1090
media_image6.png
Greyscale
In other words, from Algorithm 1, demonstration t from a batch of tasks is more than one demonstration, and the number of elements i in T is the first predetermined number of demonstrations.)
Both Finn and the combination of Yu and Mishra are directed to methods of training robots, among other things. The combination of Yu and Mishra teaches a training system for a robot, comprising: a model having a transformer architecture configured to determine how to actuate at least one of arms and an end effector of the robot, a training dataset including sets of demonstrations for the robot to perform training tasks, but does not explicitly teach demonstrations including a temporal sequence {observation; action} of tuples of successful behavior of one of the tasks or the sets of demonstrations for the training tasks each include more than one demonstration and less than a predetermined number of demonstrations. Finn teaches demonstrations including a temporal sequence {observation; action} of tuples of successful behavior of one of the tasks, and the sets of demonstrations for the training tasks each include more than one demonstration and less than a predetermined number of demonstrations.
In view of the teaching of the combination of Yu and Mishra it would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Finn into the combination of Yu and Mishra. This would result in a training system for a robot, comprising: a model having a transformer architecture configured to determine how to actuate at least one of arms and an end effector of the robot, a training dataset including sets of demonstrations for the robot to perform training tasks, where the sets of demonstrations for the training tasks each include a temporal sequence {observation; action} of tuples of successful behavior of one of the tasks and the sets of demonstrations for the training tasks each include more than one demonstration and less than a predetermined number of demonstrations.
One of ordinary skill in the art would be motivated to do this because it’s important for a robot to be able to generalize in order to avoid the computational expense of having to learn every single skill individually. (Finn, Abstract, line 1 “In order for a robot to be a generalist that can perform a wide range of jobs, it must be able to acquire a wide variety of skills quickly and efficiently in complex unstructured environments. High-capacity models such as deep neural
networks can enable a robot to represent complex skills, but learning each skill from scratch then becomes infeasible.”)
Thus far, the combination of Yu, Mishra and Finn does not explicitly teach each observation including coordinates of joints of the robot, a pose of the end effector.
Koskinopoulou teaches each observation including coordinates of joints of the robot, a pose of the end effector (Koskinopoulou, Fig. 2, and page 61, column 1, paragraph 1, line 3 “Given an observation data set X = {x1, x2, ..., xn}, xn ∈ RD, we search for an
L-dimensional representation, where usually L << D.” and, page 62, column 2, paragraph 3, line 3 “In the case of the human actor the three arm joints were used, namely shoulder, elbow and wrist, and for each one of them the 3D coordinates (x, y, z) were derived.” And, page 65, column 1, paragraph 1, line 9 “
PNG
media_image7.png
28
41
media_image7.png
Greyscale
stand for the end-effector’s 3D trajectories (3D coordinates) accomplished by actor and robot, respectively” And, page 63, column 1, paragraph 2, line 1 “Regarding the robot’s physical space, we obtained the angles of the arm’s six joints during a performed act via the available encoders.”
PNG
media_image8.png
255
567
media_image8.png
Greyscale
In other words, observation data set is observation, and three arm joints were used, namely shoulder, elbows, and wrist, each one of them the 3D coordinates (x, y, z) were derived is including coordinates of joints of the robot, end effector is end effector, and angle is pose.)
Both Koskinopoulou and the combination of Yu, Mishra and Finn are directed to training robots, among other things. The combination of Yu, Mishra and Finn teaches a training system for a robot comprising: a model having a transformer architecture and configured to determine how to actuate at least one of arms and an end effector of the robot, a training dataset including sets of demonstrations for the robot to perform training tasks, respectively, each of the demonstrations including a temporal sequence of {observation; action} tuples of successful behavior of one of the tasks; but does not explicitly teach each observation including coordinates of joints of the robot, a pose of the end effector. Koskinopoulou teaches each observation including coordinates of joints of the robot, a pose of the end effector.
In view of the teaching of the combination of Yu, Mishra and Finn, it would be obvious to one of ordinary skill in the art to combine the teaching of Koskinopoulou into the combination of Yu, Mishra, and Finn. This would result in a training system for a robot comprising: a model having a transformer architecture and configured to determine how to actuate at least one of arms and an end effector of the robot, a training dataset including sets of demonstrations for the robot to perform training tasks, respectively, each of the demonstrations including a temporal sequence of {observation; action} tuples of successful behavior of one of the tasks, and each observation including coordinates of joints of the robot, and a pose of the end effector.
One of ordinary skill in the art would be motivated to do this learning new behaviors for robots enables robots to be more useful in interacting with other agents and environments. (Koskinopoulou, page 59, column 1, paragraph 2, line 1 “Learning new behaviors enables robots to expand their repertoire of actions, learn new tasks and effectively interact with other agents and their environment. In the current work, we introduce a novel methodology to accomplish Learning from Demonstration (LfD) in robotic systems. Accordingly, action behaviors are systematically learned by observing a human actor, and are subsequently utilized in a Human-Robot Collaborative (HRC) task execution scenario.”)
Regarding claim 2,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein the code executed by the one or more processors is to
meta-train the policy using reinforcement learning (Yu, page 3, paragraph 7, line 1 “Meta-reinforcement learning aims to leverage the set of training task to learn a policy π(a|s) that can quickly adapt to new test tasks that were not seen during training, where both training and test tasks are assumed to be drawn from the same task distribution p(T ). Typically, the training tasks are referred to as the meta-training set, to distinguish from the adaptation (training) phase performed on the (meta-) test tasks.” In other words, π(a|s) is policy and meta-train is meta-train.).
Regarding claim 3,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein the code executed by the one or more processors is to
meta-train the policy using one of the Reptile algorithm and the model-agnostic meta-learning (MAML) algorithm (Yu, page 6, paragraph 6, line 10 “For the meta-RL evaluation, we study three algorithms: RL2 [18, 19]: an on-policy meta-RL algorithm that corresponds to training a LSTM network with hidden states maintained across episodes within a task and trained with PPO, model-agnostic meta-learning (MAML) [10, 21]: an on-policy gradient-based meta-RL algorithm that embeds policy gradient steps into the meta-optimization, and is trained with PPO, and probabilistic embeddings for actor-critic RL (PEARL) [22]: an off-policy actor-critic meta-RL algorithm, which learns to encode experience into a probabilistic embedding of the task that is fed to the actor and the critic.” In other words, MAML is one of Reptile algorithm and the model-agnostic meta-learning (MAML) algorithm.) .
Regarding claim 4,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein the code executed by the one or mor processors is to
meta-train the policy of the model before optimizing the policy (Yu, page 1, paragraph 3, line 2 “Multi-task RL methods aim to learn a single policy that can solve multiple tasks more efficiently than learning the tasks individually, while meta-learning methods train on many tasks, and optimize for fast adaptation to a new task.” In other words, meta-learning train on many tasks, and optimize to a new task is meta-train the policy before optimization.)
Regarding claim 5,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein the model is configured to
determine how to actuate at the least one of the arms and the end effector of the robot to advance toward or to completion of a task (Yu, Figure 1, and, page 4, paragraph 6, line 1 “To provide shared structure, the 50 environments require the same robotic arm to interact with different objects, with different shapes, joints, and connectivity. The tasks themselves require the robot to execute a combination of reaching, pushing, and grasping, depending on the task. In other words, manipulation tasks.” In other words, train the robot to execute a combination of reaching, pushing, and grasping is determine how to actuate one arm and effector to complete a task.).
Regarding claim 6,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 5 wherein
the task is different than the training tasks (Yu, Figure 1. In other words, test tasks are different than training tasks is the task is different than the training tasks.).
Regarding claim 7,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 5 wherein, after the meta-training and the optimization,
the model is configured to perform the task using less than or equal to a second predetermined number of user input demonstrations for performing the task, wherein the second predetermined number is an integer greater than zero (Finn, Algorithm 1, and, page 2, paragraph 5, line 1 “Our goal is to learn a policy that can quickly adapt to new tasks from a single demonstration of that task.” In other words, a single demonstration is a second predetermined number that is an integer greater than zero.).
Regarding claim 8,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 7 wherein
the second predetermined number is 5 (Finn, Algorithm 1, and, page 3, paragraph 5, line 1 “In this section, we describe how we can extend the model-agnostic meta-learning algorithm (MAML) to the imitation learning setting. The model’s input, ot, is the agent’s observation at time t, e.g. an image, whereas the output at is the action taken at time t, e.g. torques applied to the robot’s joints. We will denote a demonstration trajectory as t := {o1, a1, …oT, aT} and use a mean squared error loss as a function of policy parameters Φ as follows:
PNG
media_image9.png
67
787
media_image9.png
Greyscale
We will primarily consider the one-shot case, where only a single demonstration T(j) is used for the gradient update. However, we can also use multiple demonstrations to resolve ambiguity.”
In other words, setting T of the demonstration trajectory to 5 is the second predetermined number is 5.)
Regarding claim 10,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein
the present pose of the end effector includes a position of the end effector and an orientation of the end effector (Yu, page 5, paragraph 3, line 2 “All of our tasks are performed by a simulated Sawyer robot, with the action space corresponding to 3D end-effector positions. For all tasks, the robot must either manipulate one object with a variable goal position, or manipulate two objects with a fixed goal position. The observation space is represented as a 3-tuple of either the 3D Cartesian positions of the end-effector, the object, and the goal, or the 3D Cartesian positions of the end effector, the first object, and the second object, and is always 9 dimensional.” In other words, 3D Cartesian positions of the end-effector is includes a position of the end effector and an orientation of the end effector of the present pose.)
Regarding claim 11,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein
the user input demonstrations also include a position of an object to be interacted with by the robot during performance of the task (Yu, See above mapping. In other words, 3D Cartesian positions of the end-effector, the object, and the goal is also include a position of an object to be interacted with.).
18. Regarding claim 12,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 11 wherein
the user input demonstrations also include a position of a second object in an environment of the robot (Yu, page 5, paragraph 3, line 5 “The observation space is represented as a 3-tuple of either the 3D Cartesian positions of the end-effector, the object, and the goal, or the 3D Cartesian positions of the end effector, the first object, and the second object, and is always 9 dimensional.” In other words, the 3D Cartesian positions of the end effector, the first object, and the second object is the input demonstrations also include a position of a second object in the environment of the robot.)
Regarding claim 13,
The combination of Yu, Mishra, Finn, and Koskinopoulou teaches the training system of claim 1 wherein
the first predetermined number is an integer less than or equal to ten (Finn, Algorithm 1, and, page 3, paragraph 5, line 1. See mapping of claim 8. In other words, setting T of the demonstration trajectory to 10 or less is the first predetermined number is an integer less than or equal to 10.)
Claim 14 is a training system claim corresponding to training system for a robot claim 1. Otherwise, they are not patentably distinct. Therefore, claim 14 is rejected for the same reasons as claim 1.
Claims 15-22, and 24-27 are training method claims corresponding to training system for a robot claims 1-8, and 10-13, respectively. Claim 15 contains a subset of limitations of claim 1. Otherwise, they are not patentably distinct. One of ordinary skill in the art would understand the primary reference as teaching a training method for a robot. “[I]n considering the disclosure of a reference, it is proper to take into account not only specific teachings of the reference but also the inferences which one skilled in the art would reasonably be expected to draw therefrom.” MPEP § 2144.01. Therefore, claims 15-22, and 24-27 are rejected for the same reasons as claims 1-8, and 10-13, respectively.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BART RYLANDER whose telephone number is (571)272-8359. The examiner can normally be reached Monday - Thursday 8:00 to 5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at 571-270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Bart I Rylander/Examiner, Art Unit 2124