DETAILED ACTION
This Office Action is in response to Applicant's Communication received on 08/03/2023 for application number 18/275,722.
Claims 1-13 and 18-24 are presented for examination. Claims 1, 18, and 19 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/23/2024 has been considered by the Examiner.
Claim Objections
Claims 5 and 23 are objected to because of the following informalities: Claims 5 and 23 recite “the second imitator model”, which has no antecedent basis. Also, in Claims 5 and 23, “in plurality of update steps” should be “in a plurality of update steps”. Appropriate correction is required.
Claim 21 is objected to because of the following informalities: In Claim 21, “The non-transitory computer storage media of claim 10” should be “The non-transitory computer storage media of claim 19”. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10, 12-13, and 18-24 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Claims 1-10 and 12-13 are directed to a method, Claim 8 is directed to a system, and Claims 19-24 are directed to a medium. Thus, the claims fall within one of the statutory categories (process, machine, articles of manufacture) and are eligible under Step 1.
Step 2A Prong 1
Independent Claims
Claim 1 recites:
a method of training a policy model to generate action data for controlling an agent to perform a task in an environment, the method comprising: using the demonstrator trajectories to generate a demonstrator model, the demonstrator model being operative to generate, for any said demonstrator trajectory, a value indicative of the probability of the demonstrator trajectory occurring; and jointly training an imitator model and a policy model by: generating a plurality of imitation trajectories, each imitation trajectory being generated by repeatedly receiving state data indicating a state of the environment, using the policy model to generate action data indicative of an action; training the imitator model using the imitation trajectories, the imitator model being operative to generate, for any said imitation trajectory, a value indicative of the probability of the imitation trajectory occurring; and training the policy model using a reward function which is a measure of the similarity of the demonstrator model and the imitator model - these limitations encompass mental processes and mathematical concepts. Generating a model using trajectories and generating trajectories encompass mental process that can be practically performed in the human mind or by a human using a pen and paper; computing trajectory probabilities and a similarity-based reward, and adjusting model parameter therefrom are mathematical calculations and relationships.
Claims 18 and 19 recite:
Operations for training a policy model to generate action data for controlling an agent to perform a task in an environment, the operations comprising: using the demonstrator trajectories to generate a demonstrator model, the demonstrator model being operative to generate, for any said demonstrator trajectory, a value indicative of the probability of the demonstrator trajectory occurring; and jointly training an imitator model and a policy model by: generating a plurality of imitation trajectories, each imitation trajectory being generated by repeatedly receiving state data indicating a state of the environment, using the policy model to generate action data indicative of an action; training the imitator model using the imitation trajectories, the imitator model being operative to generate, for any said imitation trajectory, a value indicative of the probability of the imitation trajectory occurring; and training the policy model using a reward function which is a measure of the similarity of the demonstrator model and the imitator model - these limitations encompass mental processes and mathematical concepts. Generating a model using trajectories and generating trajectories encompass mental process that can be practically performed in the human mind or by a human using a pen and paper; computing trajectory probabilities and a similarity-based reward, and adjusting model parameter therefrom are mathematical calculations and relationships.
Accordingly, these claims recite an abstract idea that falls under “mental process” and “mathematical concepts” grouping.
Step 2A Prong 2
Independent Claims
Additional elements
Claims 1, 18, and 19 recite:
obtaining, for each of a plurality of performances of the task, a corresponding demonstrator trajectory comprising a plurality of sets of state data characterizing the environment at each of a plurality of corresponding successive time steps during the performance of the task - these limitations amount to insignificant extra-solution activity of mere data gathering and outputting (see MPEP § 2106.05(g)).
causing the action to be performed by the agent - these limitations are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the abstract idea with a generic agent(see MPEP § 2106.05(f)).
Claim 18 recites:
A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations -these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
Claim 19 recites:
One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations -these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
Accordingly, these additional elements do not integrate the judicial exception into a practical application because they do not impose any meaningful limits on practicing the abstract idea. These claims are directed to the abstract idea.
Step 2B
Independent Claims
Additional elements
Claims 1, 18, and 19 recite:
obtaining, for each of a plurality of performances of the task, a corresponding demonstrator trajectory comprising a plurality of sets of state data characterizing the environment at each of a plurality of corresponding successive time steps during the performance of the task - these limitations amount to insignificant extra-solution activity of mere data gathering , which is well-understood, routine, and conventional activity (see MPEP § 2106.05(d), “receiving/ transmitting data”).
causing the action to be performed by the agent - these limitations are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the abstract idea with a generic agent(see MPEP § 2106.05(f)).
Claim 18 recites:
A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations -these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
Claim 19 recites:
One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations -these limitations are recited at a high-level of generality such that it amount to no more than using generic computer components to apply the judicial exception (see MPEP § 2106.05(f)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, these claims are patent ineligible.
Step 2A Prong 1
Dependent Claims
Claims 2 and 20:
the reward function is evaluated by determining, for at least some of the imitation trajectories, a measure of the similarity of the probability of those imitation trajectories occurring according to the demonstrator model and according to the imitator model - these limitations merely furthers the mathematical calculations by specifying how the similarity-based reward is computed.
Claims 3 and 21:
the demonstrator model is trained to generate a value indicative of the probability of a set of state data of one of the demonstrator trajectories occurring based on the set of state data for at least one earlier time step in that demonstrator trajectory, the demonstrator model being operative to generate the value indicative of the probability of a corresponding one of the demonstrator trajectories occurring as the product of the respective probabilities of the sets of state data of the demonstrator trajectory - these limitations encompass mathematical calculations.
Claims 4 and 22:
the imitator model is trained to generate a value indicative of the probability of a set of state data of one of the imitation trajectories occurring based on the set of state data for at least one earlier time step in that imitation trajectory, the imitator model being operative to generate the value indicative of the probability of a corresponding one of the imitation trajectories occurring as the product of the respective probabilities of the sets of state data of the imitation trajectory - these limitations encompass mathematical calculations.
Claims 5 and 23:
said jointly training the second imitator model and the policy model is performed in plurality of update steps, each update step comprising: generating one or more said imitation trajectories using the current policy model; updating the policy model using the reward function using one or more of the imitation trajectories; and updating the imitator model using one or more of the generated imitation trajectories - these limitations encompass mathematical calculations.
Claims 6 and 24:
the imitator model is updated to increase the value of an imitator reward function which characterizes the probability of at least some of the generated imitation trajectories occurring according to the imitator model - these limitations encompass mathematical calculations.
Claim 7:
the update to the policy model is performed using a maximum a posteriori policy optimization algorithm - these limitations encompass mathematical calculations.
Claim 8:
generated imitation trajectories are added to a replay buffer, and said updating of the policy model and the imitator model are performed using imitation trajectories selected from the replay buffer - these limitations encompass mathematical calculations.
Claim 9:
the demonstrator model is trained before the joint training of the imitator model and the policy model - these limitations encompass mental process and mathematical calculations.
Claim 10:
the demonstrator model is trained by a process which iteratively increases the value of a demonstrator reward function which characterizes the probability of at least some of the demonstrator trajectories occurring according to the demonstrator model - these limitations encompass mathematical calculations.
Claim 13:
performing a task by using the policy model to generate commands for controlling an agent to perform the task in an environment; at each of a plurality of time steps performing the steps of: the policy model generating action data based on the state data, whereby the policy model successively generates a sequence of sets of action data to control the agent to perform the task - these limitations encompass mental process and mathematical calculations.
Thus, the claims recite the abstract idea.
Step 2A Prong 2
Dependent Claims
Additional elements
Claim 12:
the state data comprises image data defining a plurality of images of the environment - these limitations merely limit the type of data being operated on by the abstract idea (see MPEP § 2106.05(h)).
Claim 13:
(i) obtaining state data characterizing a current state of the environment; (ii) transmitting the state data to the policy model; and (iii) transmitting the action data to the agent, the agent being operative to perform an action defined by the action data within the environment - these limitations amount to insignificant extra-solution activity of mere data gathering and outputting (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the judicial exception into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to the abstract idea.
Step 2B
Dependent Claims
Additional elements
Claim 12:
the state data comprises image data defining a plurality of images of the environment - these limitations merely limit the type of data being operated on by the abstract idea (see MPEP § 2106.05(h)).
Claim 13:
(i) obtaining state data characterizing a current state of the environment; (ii) transmitting the state data to the policy model; and (iii) transmitting the action data to the agent, the agent being operative to perform an action defined by the action data within the environment - - these limitations amount to insignificant extra-solution activity of mere data gathering , which is well-understood, routine, and conventional activity (see MPEP § 2106.05(d), “receiving/ transmitting data”).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are patent ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 11-13, and 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over Tunyasuvunakool et al. (US 2019/0126472 A1 hereinafter Tunyasuvunakool) in view of Englert et al. (“Model-based Imitation Learning by Probabilistic Trajectory Matching”, 2013, hereinafter Englert).
Regarding Claim 1, Tunyasuvunakool teaches a method of training a policy model to generate action data for controlling an agent to perform a task in an environment ([0009] a system implemented as computer programs; perform a method to train neural network used to select actions to be performed by an agent interacting with an environment; [0010] the agent is a mechanical system; the agent comprises drive mechanism to translate and/or rotate the agent in the environment under the control of the neural network; act based on respective commands they receive from the neural network (i.e., perform tasks in environment); [0053] the neural network which implements the control system 106 may be referred to as a “policy network”), the method comprising:
obtaining, for each of a plurality of performances of the task, a corresponding demonstrator trajectory comprising a plurality of sets of state data characterizing the environment at each of a plurality of corresponding successive time steps during the performance of the task ([0015] for each of a plurality of previous performances of the task, a respective dataset characterizing the corresponding performance of the task; each dataset describes a trajectory comprising a set of observations characterizing a set of successive states of the environment and the agent, and the corresponding actions performed by the agent; each previous performance of the task may be previous performance of the task by the agent under the control of an expert; this is termed an “expert trajectory” (i.e., demonstrator trajectory); each dataset may comprise data specifying explicitly the positions of the objects at times during the performance of the task (i.e., sets of state data characterizing the environment at each corresponding successive time steps during the performance of the task));
using the demonstrator trajectories to generate a demonstrator model, the demonstrator model being operative to generate, for any said demonstrator trajectory, a value indicative of the demonstrator trajectory ([0015] for each of a plurality of previous performances of the task, a respective dataset characterizing the corresponding performance of the task; each dataset describes a trajectory comprising a set of observations characterizing a set of successive states of the environment and the agent, and the corresponding actions performed by the agent; each previous performance of the task may be previous performance of the task by the agent under the control of an expert; this is termed an “expert trajectory” (i.e., demonstrator trajectory); each dataset may comprise data specifying explicitly the positions of the objects at times during the performance of the task; [0018] the neural network is trained based on the datasets, the sets of commands and the corresponding reward values; [0033] the neural network itself may take many forms; [0064] the training employs both datasets describing the performance of the task by an expert (i.e., the neural network/demonstrator model trained based on the expert trajectories), and additional datasets characterizing additional simulated performances of the task by the agent 102 under the control of the control system 206 as it learns; the training combines advantages of imitation learning and reinforcement learning; [0029] the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets - thus, generating a value characterizing expert/ demonstrator trajectory); and
jointly training an imitator model and a policy model ([0017] each set of commands results in a respective further dataset which may be called an imitation trajectory which comprises both the actions performed by the agent in implementing the commands in sequence, and a set of observations of the environment while the set of commands is performed in sequence; [0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0033] the neural network itself may take many forms; [0064] the training employs both datasets describing the performance of the task by an expert, and additional datasets characterizing additional simulated performances of the task by the agent 102 (i.e., the neural network/imitator model trained based on simulated/ imitation trajectories) under the control of the control system 206 as it learns; the training combines advantages of imitation learning and reinforcement learning; [0053] the control system 106 as a whole defines a stochastic policy πθ which outputs a sample from a probability distribution, over the action space specifying a respective probability value for any possible action in the action space; the probability distribution depends upon the input data to the control system 106; the neural network which implements the control system 106 may be referred to as a “policy network”; [0084] generative adversarial imitation learning (GAIL) and proximal policy optimization (PPO); [0086]-[0087] the GAIL technique allows an agent to interact with an environment during its training and learn from its experiences; similar to Generative Adversarial Networks (GANs), GAIL employs two networks, a policy network and a discriminator network trained together with GAN-like min-max objective function; [0090] imitation learning, using the discriminator) by:
generating a plurality of imitation trajectories, each imitation trajectory being generated by repeatedly receiving state data indicating a state of the environment, using the policy model to generate action data indicative of an action, and causing the action to be performed by the agent ([0017] each set of commands results in a respective further dataset which may be called an imitation trajectory which comprises both the actions performed by the agent in implementing the commands in sequence, and a set of observations of the environment while the set of commands is performed in sequence (i.e., repeatedly receiving state data indicating a state of the environment); [0053] the neural network which implements the control system 106 may be referred to as a “policy network”; [0073] the control system 206 generates control data 205 specifying an action, and transmits it to the physical simulation unit; [0058] the control data transmitted to a physical simulation unit 223 which has additionally received the physical state data corresponding to the initial system state from the initial state definition unit 221; based on the physical state data and the control data 205, the physical simulation unit 223 is configured to generate simulated physical state data st+1 which indicates the configuration of the environment and of the agent at an immediately following time-step which would result, starting from the initial system state, if the agent 102 performed the action defined by the control data 205; the output of the control system 206 is control data 205 for controlling an agent);
training the imitator model using the imitation trajectories, the imitator model being operative to generate, for any said imitation trajectory, a value indicative of the imitation trajectory ([0017] each set of commands results in a respective further dataset which may be called an imitation trajectory which comprises both the actions performed by the agent in implementing the commands in sequence, and a set of observations of the environment while the set of commands is performed in sequence; [0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0033] the neural network itself may take many forms; [0064] the training employs both datasets describing the performance of the task by an expert, and additional datasets characterizing additional simulated performances of the task by the agent 102 (i.e., the neural network/imitator model trained based on the imitation trajectories) under the control of the control system 206 as it learns; the training combines advantages of imitation learning and reinforcement learning; [0029] the imitation reward value may be obtained from a discriminator network; the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets - thus, generating a value characterizing expert/ demonstrator trajectory); and
training the policy model using a reward function which is a measure of the similarity of the demonstrator model and the imitator model ([0017] each set of commands results in a respective further dataset which may be called an imitation trajectory which comprises both the actions performed by the agent in implementing the commands in sequence, and a set of observations of the environment while the set of commands is performed in sequence; [0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0064] the training employs both datasets describing the performance of the task by an expert, and additional datasets characterizing additional simulated performances of the task by the agent 102 under the control of the control system 206 as it learns; the training combines advantages of imitation learning and reinforcement learning; [0028] an imitation reward value derived using the datasets, and a task reward value calculated using a task reward function; the task reward function defines task reward values based on states of the environment/ agent which result from the sets of commands; [0029] the imitation reward value may be obtained from a discriminator network; the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets; [0053] the neural network which implements the control system 106 may be referred to as a “policy network”; [0059] the discriminator network generates an output value which is indicative of how similar the action a specified by the control data 205 is to the action which a human expert would have instructed the agent to make if the environment and the agent were in the initial state - thus, the neural network/ policy model is trained using reward function based on a measure of the similarity of the demonstrator model (expert trajectory datasets) and the imitator model (simulated data/ imitation trajectories)).
However, Tunyasuvunakool fails to expressly teach wherein the value indicative of the probability of the demonstrator trajectory occurring and the value indicative of the probability of the imitator trajectory occurring.
In the same field of endeavor, Englert teaches wherein the value indicative of the probability of the demonstrator trajectory occurring and the value indicative of the probability of the imitator trajectory occurring (page 1, section 1. Introduction, para. 6 - imitation learning by probabilistic trajectory matching; page 3, section III, para. 4 - policy parameters θ such that the predicted trajectory distribution matches the observed expert trajectory distribution; page 2, section I, para. 4 - probability distributions over trajectories for representing both the demonstrated trajectories and the predicted trajectory - thus, the values indicate probability of the expert/ demonstrator trajectories and predicted/ imitation trajectories).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein the value indicative of the probability of the demonstrator trajectory occurring and the value indicative of the probability of the imitator trajectory occurring, as taught by Englert into Tunyasuvunakool. Doing so would be desirable because it would substantially speed up learning in imitation learning and it is directly applicable to learning models and policies for a highly compliant robotic arm in only a few attempts (Englert, page 6, Conclusion).
As to dependent Claim 2, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Tunyasuvunakool further teaches wherein the reward function is evaluated by determining, for at least some of the imitation trajectories, a measure of the similarity of those imitation trajectories occurring according to the demonstrator model and according to the imitator model ([0017] each set of commands results in a respective further dataset which may be called an imitation trajectory which comprises both the actions performed by the agent in implementing the commands in sequence, and a set of observations of the environment while the set of commands is performed in sequence; [0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0064] the training employs both datasets describing the performance of the task by an expert, and additional datasets characterizing additional simulated performances of the task by the agent 102 under the control of the control system 206 as it learns; the training combines advantages of imitation learning and reinforcement learning; [0028] an imitation reward value derived using the datasets, and a task reward value calculated using a task reward function; the task reward function defines task reward values based on states of the environment/ agent which result from the sets of commands; [0029] the imitation reward value may be obtained from a discriminator network; the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets; [0059] the discriminator network generates an output value which is indicative of how similar the action a specified by the control data 205 is to the action which a human expert would have instructed the agent to make if the environment and the agent were in the initial state - thus, the reward function is based on a measure of the similarity of the demonstrator model (expert trajectory datasets) and the imitator model (simulated data/ imitation trajectories). Englert further teaches wherein the measure of the similarity of the probability of those imitation trajectories occurring (page 2, section I - a probability distribution over forward models; probability distributions over trajectories for representing both the demonstrated trajectories and the predicted trajectory; page 2, section II - minimizing the KL divergence between the distribution over demonstrated trajectories and the predicted trajectory distribution of the robot when executing a policy).
As to dependent Claim 3, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Englert further teaches wherein the demonstrator model is trained to generate a value indicative of the probability of a set of state data of one of the demonstrator trajectories occurring based on the set of state data for at least one earlier time step in that demonstrator trajectory, the demonstrator model being operative to generate the value indicative of the probability of a corresponding one of the demonstrator trajectories occurring as the product of the respective probabilities of the sets of state data of the demonstrator trajectory (page 2, section I - a probability distribution over forward models; probability distributions over trajectories for representing both the demonstrated trajectories and the predicted trajectory; page 2, section II - minimizing the KL divergence between the distribution over demonstrated trajectories and the predicted trajectory distribution of the robot when executing a policy; page 3, section III A - forward model maps a state xt−1 and action ut−1 of the system to the next state xt = f(xt−1,ut−1) - thus, each successive state of the demonstrator trajectory is predicted from the preceding step and the trajectory probability factorizes as the product of the per-step state probabilities).
As to dependent Claim 4, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Englert further teaches wherein the imitator model is trained to generate a value indicative of the probability of a set of state data of one of the imitation trajectories occurring based on the set of state data for at least one earlier time step in that imitation trajectory, the imitator model being operative to generate the value indicative of the probability of a corresponding one of the imitation trajectories occurring as the product of the respective probabilities of the sets of state data of the imitation trajectory (page 2, section I - a probability distribution over forward models; probability distributions over trajectories for representing both the demonstrated trajectories and the predicted trajectory; page 2, section II - minimizing the KL divergence between the distribution over demonstrated trajectories and the predicted trajectory distribution of the robot when executing a policy; page 3, section III A - forward model maps a state xt−1 and action ut−1 of the system to the next state xt = f(xt−1,ut−1) - thus, each successive state of the imitation trajectory is predicted from the preceding step and the trajectory probability factorizes as the product of the per-step state probabilities).
As to dependent Claim 5, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Tunyasuvunakool further teaches wherein said jointly training the second imitator model and the policy model is performed in plurality of update steps, each update step comprising: generating one or more said imitation trajectories using the current policy model ([0093] policy to generate trajectories closer to the expert trajectories); updating the policy model using the reward function using one or more of the imitation trajectories ([0017] based on the reward values, the neural network is updated; [0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0028] an imitation reward value derived using the datasets, and a task reward value calculated using a task reward function; the task reward function defines task reward values based on states of the environment/ agent which result from the sets of commands; [0029] the imitation reward value may be obtained from a discriminator network; the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets; [0053] the neural network which implements the control system 106 may be referred to as a “policy network”; [0081] the update unit 231 adjusts the parameters of the control system 206 (that is, parameters of the convolutional network 208, and the neural networks 209, 210), based on the discrimination values, the reward values obtained in step 307, and the values Vϕ(st) and Vϕ(st+K); and updating the imitator model using one or more of the generated imitation trajectories ([0064] the training employs both datasets describing the performance of the task by an expert, and additional datasets characterizing additional simulated performances of the task by the agent 102 (i.e., the neural network/imitator model trained based on the imitation trajectories) under the control of the control system 206 as it learns;[0018] the neural network is trained/one or more parameters of the neural network are adjusted based on the datasets, the sets of commands and the corresponding reward values; [0028] an imitation reward value derived using the datasets, and a task reward value calculated using a task reward function; the task reward function defines task reward values based on states of the environment/ agent which result from the sets of commands; [0029] the imitation reward value may be obtained from a discriminator network; the discriminator network trained to give a value indicative of a discrepancy between results commands output by the neural network/ policy network and an expert policy inferred from the datasets; [0081] the update unit 231 adjusts the parameters of the control system 206 (that is, parameters of the convolutional network 208, and the neural networks 209, 210), based on the discrimination values, the reward values obtained in step 307, and the values Vϕ(st) and Vϕ(st+K)).
As to dependent Claim 6, Tunyasuvunakool and Englert teach all the limitations of Claim 5. Englert further teaches wherein the imitator model is updated to increase the value of an imitator reward function which characterizes the probability of at least some of the generated imitation trajectories occurring according to the imitator model (page 2, section II - imitate the expert’s behavior by minimizing the KL divergence between the distribution over demonstrated trajectories and the predicted trajectory distribution (i.e., generated imitation trajectories) of the robot when executing a policy; minimizing the KL divergence between these two trajectory distributions induces a natural cost function, that can be used by any RL algorithm to learn policies; page 2, section II B - matching the predicted trajectory of the current policy with the expert trajectory distribution via minimizing the KL divergence induces a natural cost function in a standard RL context; page 4, section III B - iteratively predicting the state distributions for a given policy π and an initial state distribution; model-based imitation learning via probabilistic trajectory matching; page 3, section III - objective in policy search is to find policy parameters θ of a policy π that minimize the long-term cost (equivalent to increased imitator reward function based on maximum likelihood of imitation/ predicted trajectories)).
As to dependent Claim 11, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Tunyasuvunakool further teaches wherein the environment is a real-world environment, the state data is data collected by at least one sensor, and the agent is an electromechanical agent arranged to move in the environment according to the action data ([0010] the agent is a mechanical system, such as, a “robot” or a vehicle comprising one or more members connected together; the agent is located within a real-world environment; the neural network may transmit commands to the agent in the form of commands (i.e., action data) which indicate “joint velocities”, that is the angular rate at which the drive mechanism(s) should move one or more of the members relative to other of the members; [0043] the image capture device captures images of some or all of the environment; [0044] the control system receives as one input the image data generated by the image capture devices).
As to dependent Claim 12, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Tunyasuvunakool further teaches wherein the state data comprises image data defining a plurality of images of the environment ([0043] the image capture device captures images of some or all of the environment; [0044] the control system receives as one input the image data generated by the image capture devices).
As to dependent Claim 13, Tunyasuvunakool and Englert teach all the limitations of Claim 1. Tunyasuvunakool further teaches wherein performing a task by using the policy model to generate commands for controlling an agent to perform the task in an environment ([0053] the neural network which implements the control system 106 may be referred to as a “policy network”; [0073] the control system 206 generates control data 205 specifying an action, and transmits it to the physical simulation unit; [0058] the control data transmitted to a physical simulation unit 223 which has additionally received the physical state data corresponding to the initial system state from the initial state definition unit 221; based on the physical state data and the control data 205, the physical simulation unit 223 is configured to generate simulated physical state data st+1 which indicates the configuration of the environment and of the agent at an immediately following time-step which would result, starting from the initial system state, if the agent 102 performed the action defined by the control data 205; the output of the control system 206 is control data 205 for controlling an agent), comprising: at each of a plurality of time steps performing the steps of: (i) obtaining state data characterizing a current state of the environment ([0058] the control data transmitted to a physical simulation unit 223 which has additionally received the physical state data corresponding to the initial system state from the initial state definition unit 221; based on the physical state data and the control data 205, the physical simulation unit 223 is configured to generate simulated physical state data st+1 which indicates the configuration of the environment and of the agent at an immediately following time-step which would result, starting from the initial system state, if the agent 102 performed the action defined by the control data 205; [0056] the initial state is defined by “physical state data” which specifies the positions and optionally velocities of all objects in the environment when they are in the initial state and the positions/velocities of all members of the agent); (ii) transmitting the state data to the policy model, the policy model generating action data based on the state data ([0053] the neural network which implements the control system 106 may be referred to as a “policy network”; [0073] the control system 206 generates control data 205 specifying an action, and transmits it to the physical simulation unit; [0058] the control data transmitted to a physical simulation unit 223 which has additionally received the physical state data corresponding to the initial system state from the initial state definition unit 221); and (iii) transmitting the action data to the agent, the agent being operative to perform an action defined by the action data within the environment ([0058] the control data transmitted to a physical simulation unit 223 which has additionally received the physical state data corresponding to the initial system state from the initial state definition unit 221; based on the physical state data and the control data 205, the physical simulation unit 223 is configured to generate simulated physical state data st+1 which indicates the configuration of the environment and of the agent at an immediately following time-step which would result, starting from the initial system state, if the agent 102 performed the action defined by the control data 205; the output of the control system 206 is control data 205 for controlling an agent); whereby the policy model successively generates a sequence of sets of action data to control the agent to perform the task ([0058] based on the physical state data and the control data 205, the physical simulation unit 223 is configured to generate simulated physical state data st+1 which indicates the configuration of the environment and of the agent at an immediately following time-step which would result, starting from the initial system state, if the agent 102 performed the action defined by the control data 205; [0073] in step 303, the control system 206 generates control data 205 specifying an action and transmits it to the physical simulation unit; [0082] in step 311, the controller 232 determines whether a termination criterion is met ; if not, the method returns to step 302, for the selection of a new initial state).
Claim 18 is a system claim corresponding to the method claim 1 above and therefore, rejected for the same reasons. Tunyasuvunakool further teaches wherein a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations ([0009] system implemented as computer programs in one or more computers; [0112] computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or any other kind of central processing unit; central processing unit will receive instructions and data from a read only memory or a random access memory or both; central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data).
Claims 19-24 are medium claims corresponding to the method claims 1-6 above and therefore, rejected for the same reasons. Tunyasuvunakool further teaches wherein one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations ([0009] system implemented as computer programs in one or more computers; [0112] computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or any other kind of central processing unit; central processing unit will receive instructions and data from a read only memory or a random access memory or both; central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Tunyasuvunakool in view of Englert, further in view of Abdolmaleki et al. (“Maximum A Posteriori Policy Optimisation”, 2018, hereinafter Abdolmaleki).
As to dependent Claim 7, Tunyasuvunakool and Englert teach all the limitations of Claim 5. However, Tunyasuvunakool and Englert fail to expressly teach wherein the update to the policy model is performed using a maximum a posteriori policy optimization algorithm.
In the same field of endeavor, Abdolmaleki teaches wherein the update to the policy model is performed using a maximum a posteriori policy optimization algorithm (page 4, section 3, 3.1 - updates the parametric policy; additional logp(θ) term is a prior over policy parameters and can be motivated by a maximum a-posteriori estimation problem; π represents a default policy towards which q is regularized/the current best policy).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein the update to the policy model is performed using a maximum a posteriori policy optimization algorithm, as taught by Abdolmaleki into Tunyasuvunakool and Englert. Doing so would be desirable because it would provide an algorithm that is highly data efficient, robust to hyperparameter choices and applicable to complex control problems, as well as improve effectiveness of MPO on a large set of continuous control problem (Abdolmaleki, page 10, Conclusion).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Tunyasuvunakool in view of Englert, further in view of Kostrikov et al. (“Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning”, 2018, hereinafter Kostrikov).
As to dependent Claim 8, Tunyasuvunakool and Englert teach all the limitations of Claim 5. However, Tunyasuvunakool and Englert fail to expressly teach wherein generated imitation trajectories are added to a replay buffer, and said updating of the policy model and the imitator model are performed using imitation trajectories selected from the replay buffer.
In the same field of endeavor, Kostrikov teaches wherein generated imitation trajectories are added to a replay buffer, and said updating of the policy model and the imitator model are performed using imitation trajectories selected from the replay buffer (page 4, section 4.1 - the agent is now incentivized to move in loops or take small actions that keep it close to the states in the expert’s trajectories; page 5, section 4.2 - explicitly learning rewards for absorbing states for expert demonstrations and trajectories produced by a policy (imitation trajectories); page 6, section 4.3 - we use an off-policy RL algorithm and perform off-policy training of the GAIL discriminator performed in the following way: instead of sampling trajectories from a policy directly, we sample transitions from a replay buffer R collected (including imitation trajectories) while performing off-policy training (i.e., updating model)).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein generated imitation trajectories are added to a replay buffer, and said updating of the policy model and the imitator model are performed using imitation trajectories selected from the replay buffer, as taught by Kostrikov into Tunyasuvunakool and Englert. Doing so would be desirable because it would improve sample inefficiency with respect to policy transitions in the environment (Kostrikov, page 8, Conclusion).
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Tunyasuvunakool in view of Englert, further in view of Kimura et al. (US 2019/0272465 A1 hereinafter Kimura).
As to dependent Claim 9, Tunyasuvunakool and Englert teach all the limitations of Claim 1. However, Tunyasuvunakool and Englert fail to expressly teach wherein the demonstrator model is trained before the joint training of the imitator model and the policy model.
In the same field of endeavor, Kimura teaches wherein the demonstrator model is trained before the joint training of the imitator model and the policy model ([0034] state prediction model 130 that is trained using the expert demonstrations; and a reward estimation module 140 that estimates a reward signal based on a state predicted by the state prediction model 130 and an actual state observed by the agent 120; [0033] during a phase of inverse reinforcement learning (IRL), the reinforcement learning system learns a reward function (built on the demonstrator state prediction model) appropriate for the environment by using the expert demonstrations that are actually performed by the expert; [0094] reward function can be learned using the expert demonstrations through the IRL phase and the learned reward function can be used in the following RL phase to learn a suitable policy for the agent to perform a given task - thus, the demonstrator model is trained before the policy training).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein the demonstrator model is trained before the joint training of the imitator model and the policy model, as taught by Kimura into Tunyasuvunakool and Englert. Doing so would be desirable because it would minimize error between visited state in the expert demonstrations and inferred state from one or more preceding visited states in the expert demonstrations (Kimura [0052]) and reduce the need for enormous computational resources (Kimura [0004]).
As to dependent Claim 10, Tunyasuvunakool and Englert teach all the limitations of Claim 1. However, Tunyasuvunakool and Englert fail to expressly teach wherein the demonstrator model is trained by a process which iteratively increases the value of a demonstrator reward function which characterizes the probability of at least some of the demonstrator trajectories occurring according to the demonstrator model.
In the same field of endeavor, Kimura teaches wherein the demonstrator model is trained by a process which iteratively increases the value of a demonstrator reward function which characterizes the probability of at least some of the demonstrator trajectories occurring according to the demonstrator model ([0066] the objective of the IRL is to find an appropriate reward function that can maximize the likelihood of the finite set of the state trajectories, which in turn is used to guide the following reinforcement learning and enable the agent to learn a suitable action policy; [0049] the state prediction model used to estimate the reward signal (r) in the following reinforcement learning, by using the expert demonstrations; [0050] the state prediction model 130 is configured to predict, for an inputted state, a state similar to the expert demonstrations that has been used to train the state prediction model; [0052] the model is trained so as to minimize an error between an visited state in the expert demonstrations and an inferred state from one or more preceding visited states in the expert demonstrations, which is maximum likelihood objective that iteratively increases the probability the model assigns to the demonstrated trajectories (i.e., reward function characterizing the probability of the demonstrator trajectories occurring according to the demonstrator model)).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have incorporated wherein the demonstrator model is trained by a process which iteratively increases the value of a demonstrator reward function which characterizes the probability of at least some of the demonstrator trajectories occurring according to the demonstrator model, as taught by Kimura into Tunyasuvunakool and Englert. Doing so would be desirable because it would minimize error between visited state in the expert demonstrations and inferred state from one or more preceding visited states in the expert demonstrations (Kimura [0052]) and reduce the need for enormous computational resources (Kimura [0004]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 CFR § 1.111(c) to consider these references fully when responding to this action.
Hafner (US 2021/0205984 A1) teaches: by combining the trained latent robot dynamics model and the trained reward function for a particular robotic task, the robot may plan, in latent space, for the best actions to complete the particular robotic task; the data may be simulated data, where the training examples are simulated training examples, where the training examples are based on simulated observations, simulated actions, and simulated rewards; the data may be real data, where the training examples are based on real observations, actions, and rewards (see [0005]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REJI KARTHOLY whose telephone number is (571)272-3432. The examiner can normally be reached on Monday - Thursday from 7:30 am to 3:30 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch, can be reached at telephone number 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/REJI KARTHOLY/Primary Examiner, Art Unit 2143
Prosecution Insights