Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
2. This action is in response to the original filing on 05/17/2024. Claims 1-22 are pending and have been considered below.
Information Disclosure Statement
3. The information disclosure statement (IDS(s)) submitted on 01/07/2025 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
4. Claims 9, 11-13, 15, and 16 are objected to because of the following informalities:
Claims 9 recites “the new state vector” where “a new state vector” was apparently intended.
Claims 11 recites “the encoded representation” where “encoded representation” was apparently intended.
Claims 12 recites “wherein the successor feature is modelled” where “wherein each of the successor features is modelled” was apparently intended.
Claims 13 recites “wherein the successor feature is a value function representation” where “wherein each of the successor features is a value function representation” was apparently intended.
Claims 15 recites “jointly training the task encoder neural network and the state feature neural network” where “jointly training task encoder neural network and state feature neural network” was apparently intended.
Claims 16 recites “using the value function representation” where “using a value function representation” was apparently intended.
Claim Rejections – 35 USC § 103
5. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
6. Claims 1-7 and 18-22 are rejected under 35 U.S.C. 103 as being unpatentable over Schaul et al. (U.S. Patent Application Pub. No. US 20190244099 A1) in view of James et al. (U.S. Patent Application Pub. No. US 20210205988 A1), and further in view of Osindero et al. (U.S. Patent Application Pub. No. US 20200265313 A1).
Claim 1: Schaul teaches a method performed by one or more computers (i.e. system implemented as one or more computers; para. [0006]) and for controlling an agent interacting with an environment (i.e. a reinforcement learning system that uses an action selection neural network to control, i.e., to select actions to be performed by, an agent that can interact with an environment to perform multiple different tasks; para. [0014]) to perform a task episode of a new task (i.e. To train the action selection neural network 102, the subsystem 104 obtains a first trajectory of transitions generated while the agent 106 was performing an episode of the first task from the continual stream of tasks; para. [0046]), the method comprising, at each of a plurality of current time steps in the task episode of the new task (i.e. Each of the actors 204 then generates a respective trajectory of H transitions by selecting actions to be performed by the agent while performing the episode of the selected task using the action selection neural network 220 in accordance with the current values of the parameters and while the action selection neural network 220 is conditioned on the goal signal gi for the selected task … Each of the H transitions in the respective trajectory can be represented as (s, a, r, s′), where s is an initial observation characterizing a state of the environment, a is an action performed by the agent in response to the observation, r is a reward received as a result of the agent performing the action, and s′ is another observation characterizing a subsequent state of the environment; para. [0058, 0059]):
obtaining an observation characterizing a current state of the environment at the time step (i.e. an observation st that characterizes a current state of an environment; para. [0054]);
obtaining a task encoding of the new task (i.e. a goal signal gi from a set of goal signals {g1, . . . , gk} (122). The goal signal gi is a vector that identifies a task i from the continuing stream of K tasks that is being performed by the reinforcement learning agent 106; para. [0036]);
for each of a plurality of training tasks and for each action in a candidate set of actions (i.e. for each task i in the subset of K′ training tasks, a network input including the initial observation st in the transition t and the goal signal gi for the task i using the action selection neural network 220 and in accordance with the current values of the parameters to generate a respective polity output that includes a Q-value Q(st, at; gi) for each action at in the set of possible actions A; para. [0063]):
determining, from the current observation, an action-specific policy for the candidate action for the training task given the current state of the environment at the time step (i.e. Given the network input, the action selection neural network 102 is configured to process the network input in accordance with the parameters to generate a policy output that defines a control policy for the agent 106 for performing the task identified by the goal signal 118. In some implementations, the network input also identifies an action a and the policy output is a Q-value, denoted as Q(s, a; gi), which is a predicted return that would result from a reinforcement learning agent performing an action a in response to an observation s that characterizes the current state of the environment given the goal signal g; para. [0037, 0054, 0063]); and
a quality score for the candidate action and for the training task (i.e. A policy output includes a respective Q value for each action in a set of possible actions that can be performed by the agent, wherein the Q value is an estimate of a return that would be received if the agent performed the action in response to the observation; para. [0063, 0080]); and
selecting an action from the candidate set using the respective quality scores for the candidate actions for the training tasks (i.e. given an observation st, each of the actors chooses an action at uniformly from a set of possible action A with probability ϵ; and with probability 1−ϵ, the actor chooses an action at defined by: a t=argmaxa Q(s t ,a;g i); para. [0058]).
Schaul does not explicitly teach obtaining a learned task encoding of the new task; generating, from the current observation and the learned task encoding for the new task, a dynamic transfer query vector for the current time step; determining, an action-specific policy vector; and determining, based on a similarity between the dynamic transfer query vector for the current time step and the action-specific policy vector.
However, James teaches obtaining a learned task encoding of the new task (i.e. a task embedding network to receive one or more demonstrations of a task and to generate a task embedding, the task embedding comprising a representation of the task, each demonstration comprising one or more observations of a performance of the task; para. [0010, 0042, 0046]); generating, from the current observation and the learned task encoding for the new task, a dynamic transfer query vector for the current time step (i.e. the task embedding 550 may be combined, fused or otherwise processed with the observation 560 to generate an action a to be performed by the robotic device 580 in accordance with the policy π. For example, a task embedding 550 in the form of a task embedding vector may be concatenated with the observation 560 (which may also be in the form of a vector) before being processed by the control network 530. For example, a concatenated vector may be generated by concatenating a task embedding vector representative of a task embedding 550 with an image feature vector obtained by processing the image data using convolutional layers of a CNN; para. [0048, 0076, 0081, 0099]);
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Schaul to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
However, Osindero teaches a dynamic transfer query vector for the current time step (i.e. The system provides the intermediate representation for the time step to the manager system, and the manager system processes the intermediate representation for the time step to generate as output a final goal vector for the time step … The system generates the initial goal vector for the time step … The system generates the final goal vector for the time step by pooling the initial goal vector generated by the goal recurrent neural network for the time step and the initial goal vectors generated by the goal recurrent neural network for one or more previous time steps; para. [0061, 0067, 0068]); determining, from the current observation, an action-specific policy vector (i.e. a set of action embedding vectors 124 for the time step. The set of action embedding vectors 124 includes an embedding vector for each action of the predetermined set of actions. An action embedding vector is an ordered collection of numerical values arranged into a vector in, by definition, an embedding space; para. [0052, 0071]); and determining, based on a similarity between the dynamic transfer query vector for the current time step and the action-specific policy vector, a quality score for the candidate action (i.e. The worker system 116 includes a modulation engine 128 that is configured to receive the action embedding vectors 124 for the time step and the goal embedding vector 130 for the time step as input and to generate as output action scores 132. Thus an action embedding vector is modulated by, that is modified dependent upon a value of, the goal embedding vector, for example using the goal embedding vector to weight an action embedding vector; para. [0051-0054]), selecting an action from the candidate set using the respective quality scores (i.e. The system selects an action from the predetermined set of actions based on the action scores generated by the worker system (210). The action scores include a separate score for each action in the predetermined set of actions. In some implementations, the system selects the action to be performed by the agent as the action having the highest corresponding score according to the action scores; para. [0063]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and James to include the feature of Osindero. One would have been motivated to make this modification because it improves action selection performance.
Claim 2: Schaul, James, and Osindero teach the method of claim 1. Schaul does not explicitly teach generating the learned task encoding of the new task for the current time step using a task encoding neural network.
However, James further teaches generating the learned task encoding of the new task for the current time step using a task encoding neural network (i.e. This processing may be performed by a trained neural network architecture. This architecture may comprise a task embedding network; para. [0042, 0046, 0094]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and Osindero to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Claim 3: Schaul, James, and Osindero teach the method of claim 2. Schaul does not explicitly teach wherein generating the learned task encoding of the new task comprises processing data characterizing the new task using the task encoding neural network to generate a vector describing features associated with the new task.
However, James further teaches wherein generating the learned task encoding of the new task comprises processing data characterizing the new task (i.e. The task embedding network 120 receives a demonstration 140 of a task. The demonstration 140 of the task may include one or more observations of a performance of the task. In the example of FIG. 1, a single demonstration 140 is received. However, in other examples which may otherwise be similar to FIG. 1, a plurality of demonstrations of a task may be received by the task embedding network 120; para. [0046]) using the task encoding neural network to generate a vector describing features associated with the new task (i.e. These examples embed tasks, wherein tasks that are similar, e.g. in visual and/or control aspects, are arranged close together in embedding space, whereas tasks that are dissimilar are arranged at a distance from one another. This task embedding space may be multi-dimensional (e.g. having a dimensionality of 10-128 … A representation of a task may be a latent representation. A latent representation is for example an inferred representation based on the observations of the task, which is for example inferred by the task embedding network 620. A task may be represented numerically, typically using a multi-dimensional numerical representation such as a tensor or a vector); para. [0041, 0087, 0091]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and Osindero to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Claim 4: Schaul, James, and Osindero teach the method of claim 3. Schaul does not explicitly teach wherein the data characterizing the new task comprises at last one of text describing the new task or an observation characterizing a goal state for the new task.
James further teaches wherein the data characterizing the new task comprises at last one of text describing the new task or an observation characterizing a goal state for the new task (i.e. the task embedding comprising a representation of the task, each demonstration comprising one or more observations of a performance of the task; para. [0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and Osindero to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Claim 5: Schaul, James, and Osindero teach the method of claim 1. Schaul further teaches wherein selecting an action to be performed by the agent based on the respective quality scores for the candidate actions for the plurality of training tasks comprises (i.e. for each task i in the subset of K′ training tasks, a network input including the initial observation st in the transition t and the goal signal gi for the task i using the action selection neural network 220 and in accordance with the current values of the parameters to generate a respective polity output that includes a Q-value Q(st, at; gi) for each action at in the set of possible actions A; para. [0063]): identifying a candidate action that has a highest quality score; and selecting the identified action (i.e. the learner 202 determines whether the performed action in the transition is the action having the highest Q-value in the policy output for the current task j; para. [0066, 0084]).
Osindero further teaches wherein selecting an action to be performed by the agent based on the respective quality scores (i.e. The worker system 116 is configured to process the inputs in accordance with current values of a set of worker system parameters to generate as output action scores 132. The system uses the action scores 132 to select the action 144 to be performed by the agent 142 for the time step; para. [0050, 0063]): identifying a candidate action that has a highest quality score; and selecting the identified action (i.e. The action scores 132 include a separate score for each action in a predetermined set of actions (i.e., the actions that may be taken by the agent 142). In some implementations, the system 100 selects the action 144 to be performed by the agent 142 as the action having the highest corresponding score according to the action scores 132; para. [0051]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and James to include the feature of Osindero. One would have been motivated to make this modification because it improves action selection performance.
Claim 6: Schaul, James, and Osindero teach the method of claim 1. Schaul does not explicitly teach wherein determining, based on a similarity between the dynamic transfer query vector for the current time step and the action-specific policy vector, a quality score for the candidate action comprises: computing an inner product between (i) the action-specific policy vector for the candidate action and (ii) the dynamic transfer query vector.
Osindero further teaches wherein determining, based on a similarity between the dynamic transfer query vector (i.e. The system provides the intermediate representation for the time step to the manager system, and the manager system processes the intermediate representation for the time step to generate as output a final goal vector for the time step (206); para. [0061, 0068, 0072]) for the current time step and the action-specific policy vector (i.e. The action score recurrent neural network 122 updates the internal state of the action score recurrent neural network 122 and generates as output a set of action embedding vectors 124 for the time step. The set of action embedding vectors 124 includes an embedding vector for each action of the predetermined set of actions. An action embedding vector is an ordered collection of numerical values arranged into a vector; para. [0052, 0071]), a quality score for the candidate action and for the training task comprises:
computing an inner product between (i) the action-specific policy vector for the candidate action for the training task and (ii) the dynamic transfer query vector (i.e. The system generates action scores for the time step (408). Specifically, the system provides the goal embedding vector for the time step and the action embedding vectors for the time step as input to the modulation engine that is configured to combine the inputs to generate as output the action scores for the time step. In some implementations, the modulation engine is configured to combine the goal embedding vector for the time step and the action embedding vectors for the time step by matrix-vector multiplication, i.e.:πt =U t w t, where Ut is a matrix where each row of the matrix is an action embedding vector for the time step, wt is the goal embedding vector for the time step, and πt is the vector of action scores for the time step; para. [0073]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and James to include the feature of Osindero. One would have been motivated to make this modification because it improves action selection performance.
Claim 7: Schaul, James, and Osindero teach the method of claim 1. Schaul does not explicitly teach wherein generating, from the current observation and the learned task encoding for the new task, a dynamic transfer query vector for the current time step comprises: processing the encoded representation of the observation and the learned task encoding of the new task using a new task state representation neural network to generate a new state vector that represents a state of the new task as of the current time step; and generating, from the new state vector, the dynamic transfer query vector for the current time step.
James further teaches wherein generating, from the current observation and the learned task encoding for the new task: processing the encoded representation of the observation (i.e. the image data it of a given observation 560 may be processed by convolutional layers of a CNN to extract image features, which may be represented as a tensor, e.g. as a vector or a multi-dimensional array. For example, the convolutional layers of the CNN may output an image feature vector; para. [0077]) and the learned task encoding of the new task (i.e. the task embedding 550 may be combined, fused or otherwise processed with the observation 560 to generate an action a to be performed by the robotic device 580 in accordance with the policy π. For example, a task embedding 550 in the form of a task embedding vector may be concatenated with the observation 560 (which may also be in the form of a vector) before being processed by the control network 530. For example, a concatenated vector may be generated by concatenating a task embedding vector representative of a task embedding 550 with an image feature vector obtained by processing the image data using convolutional layers of a CNN; para. [0048, 0076, 0081, 0099]) using a new task state representation neural network to generate a new state vector that represents a state of the new task as of the current time step (i.e. The control network 530 applies a policy to map the observations 560 of the robotic device 590 using the task embedding 550; para. [0076, 0081]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Schaul to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Osindero further teaches wherein generating, from the current observation and the learned task encoding for the new task, a dynamic transfer query vector for the current time step comprises: processing the encoded representation of the observation (i.e. The system generates an intermediate representation of the observation for the time step (204). Specifically, the system provides the observation as input to the encoding neural network, which is configured to process the observation in accordance with current values of the set of encoding neural network parameters, and to generate as output an intermediate representation of the observation for the time step. The intermediate representation is an ordered collection of numerical values (e.g., a vector or a matrix) that represents the observation for the time step in the intermediate state space; para. [0043, 0060]) and the learned task encoding of the new task using a new task state representation neural network (i.e. The system generates the latent representation for the time step by providing the intermediate representation for the time step as input to the latent representation neural network (304). The latent representation neural network is configured to process the intermediate representation for the time step in accordance with current values of the set of latent representation neural network parameters to generate as output the latent representation for the time step; para. [0046, 0066]) to generate a new state vector that represents a state of the new task as of the current time step (i.e. The latent representation for the time step is an ordered collection of numerical values (e.g., a vector or a matrix) that represents the intermediate representation and (indirectly) the observation for the time step in the latent state space; para. [0066]); and generating, from the new state vector, the dynamic transfer query vector for the current time step (i.e. The system provides the intermediate representation for the time step to the manager system, and the manager system processes the intermediate representation for the time step to generate as output a final goal vector for the time step … The system generates the initial goal vector for the time step … The system generates the final goal vector for the time step by pooling the initial goal vector generated by the goal recurrent neural network for the time step and the initial goal vectors generated by the goal recurrent neural network for one or more previous time steps; para. [0061, 0067, 0068]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul and James to include the feature of Osindero. One would have been motivated to make this modification because it improves action selection performance.
Claim 18: Schaul, James, and Osindero teach the method of claim 1. Schaul further teaches wherein the agent is a mechanical agent and the environment is a real-world environment (i.e. the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment; para. [0016]).
Claim 19: Schaul, James, and Osindero teach the method of claim 18. Schaul further teaches wherein the agent is a robot (i.e. the agent may be a robot interacting with the environment to accomplish a specific task; para. [0016]).
Claim 20: Schaul, James, and Osindero teach the method of claim 18. Schaul does not explicitly teach wherein the training tasks are learned in a simulated environment and the new task is performed in a real-world environment.
However, James further teaches wherein the training tasks are learned in a simulated environment (i.e. the set of parameters for the task embedding network and the control network may be learnt in a simulated environment using a simulated robotic device … examples as described herein may be used in a simulated-to-real context, where a goal is to learn policies for a control network within a simulation and then transfer these to a real-world environment with little or no additional training; para. [0021, 0120]) and the new task is performed in a real-world environment (i.e. a robotic device may continue to learn new tasks from single or multiple demonstrations; para. [0121]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Schaul to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Claim 21 is similar in scope to Claims 1 and is rejected under a similar rationale.
Schaul teaches a system comprising one or more computers and one or more storage devices storing instruction that when executed by the one or more computers cause the one or more computers to perform operations (i.e. system implemented as one or more computers … Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus; para. [0006, 0088]).
Claim 22 is similar in scope to Claims 1 and is rejected under a similar rationale.
Schaul teaches one or more non-transitory computer storage media storing instruction that when executed by one or more computers cause the one or more computers to perform operations (i.e. system implemented as one or more computers … Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus; para. [0006, 0088]).
7. Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Schaul in view of James, Osindero, and further in view of Qin et al. (Multi-Agent Policy Transfer via Task Relationship Modeling, arXiv, published 2022, pages 1-14).
Claim 8: Schaul, James, and Osindero teach the method of claim 1. Schaul does not explicitly teach wherein the dynamic transfer query vector is a linear combination of respective training task encodings of each of the plurality of training tasks.
However, Qin teaches wherein the dynamic transfer query vector is a linear combination (i.e. we learn new task representation as a linear combination of source task representations: equation 3; Section 2.1, 4.5, pages 2-3, 7-8) of respective training task encodings (i.e. We associate each task i with a representation zi ∈ Rm and expect it to reflect different properties of tasks; Section 2.1, page 2) of each of the plurality of training tasks (i.e. Overall, our framework first trains on several source tasks S ={Si}and then transfers the learned cooperative knowledge to unseen tasks T = {Tj} from the same task distribution; Section 2, page 2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul, James, and Osindero to include the feature of Qin. One would have been motivated to make this modification because it improves transfer and generalization to unseen tasks.
8. Claims 11 and 14-16 are rejected under 35 U.S.C. 103 as being unpatentable over Schaul in view of James, Osindero, and further in view of Barreto et al. (Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement, published 2018, pages 1-10).
Claim 11: Schaul, James, and Osindero teach the method of claim 1. Schaul does not explicitly teach wherein generating an action-specific policy vector for the candidate action for the training task comprises: processing (i) the candidate action and (ii) the encoded representation of the observation using a successor feature neural network to generate successor features for the training task and associated with the candidate action; and generating an action-specific policy vector comprising the successor features.
However, Barreto teaches wherein generating an action-specific policy vector for the candidate action for the training task comprises: processing (i) the candidate action and (ii) the encoded representation of the observation using a successor feature neural network to generate successor features for the training task (i.e. In the context of transfer, GPI makes it possible to leverage knowledge accumulated over time, across multiple tasks, to learn a new task faster. Suppose that the agent has access to n policies π1,π2,...,πn. These can be arbitrary policies, but for the sake of the argument let us assume they are solutions for tasks M1, M2, ..., Mn; Section 2.2 SF&GPI, pages 2-3) and associated with the candidate action (i.e. equation 4, Section 2.2); and generating an action-specific policy vector comprising the successor features (i.e. The SF&GPI agent adopted in the experiments is a variation of Algorithms 1 and 2 that uses Watkins’s (1989) Q(λ) to apply Q-learning with eligibility traces. The functions ˜ φ and ˜ Ψ are computed by a deep neural network whose architecture is shown in Figure 2. The network is composed of three parts. The first one uses the history of observations and actions up to time t, ht, to compute a state signal ˜st = f(ht). The construction of ˜st can itself be broken into two stages corresponding to specific functional modules: a convolutional network (CNN) to handle the pixel-based observation ot and a long short-term network (LSTM) to compute f(ht) in a recursive way. The second part of the network is composed of D + 1 specialised blocks that receive ˜st as input and compute ˜ φ(˜st,a) and ˜ ψπi(˜st,a) for all a ∈ A; Section 5.2 Agents, pages 6-7).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul, James, and Osindero to include the feature of Barreto. One would have been motivated to make this modification because it facilitates reuse of learned policies and transfer to unseen tasks without requiring a separately learned value function for every new task.
Claim 14: Schaul, James, Osindero, and Barreto teach the method of claim 11. Schaul does not explicitly teach wherein the successor feature neural network is pre-trained on the plurality of training tasks.
However, Barreto further teaches wherein the successor feature neural network is pre-trained (i.e. The SF&GPI agent adopted in the experiments is a variation of Algorithms 1 and 2 that uses Watkins’s (1989) Q(λ) to apply Q-learning with eligibility traces. The functions ˜ φ and ˜ Ψ are computed by a deep neural network whose architecture is shown in Figure 2. The network is composed of three parts. The first one uses the history of observations and actions up to time t, ht, to compute a state signal ˜st = f(ht). The construction of ˜st can itself be broken into two stages corresponding to specific functional modules: a convolutional network (CNN) to handle the pixel-based observation ot and a long short-term network (LSTM) to compute f(ht) in a recursive way. The second part of the network is composed of D + 1 specialised blocks that receive ˜st as input and compute ˜ φ(˜st,a) and ˜ ψπi(˜st,a) for all a ∈ A; Section 5.2 Agents, pages 6-7) on the plurality of training tasks (i.e. base tasks; Section 5.1, environment; page 6).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul, James, and Osindero to include the feature of Barreto. One would have been motivated to make this modification because it facilitates reuse of learned policies and transfer to unseen tasks without requiring a separately learned value function for every new task.
Claim 15: Schaul, James, Osindero, and Barreto teach the method of claim 14. Schaul does not explicitly teach the method further comprising jointly training the task encoder neural network and the state feature neural network to optimize an objective function that includes a regression loss term.
However, James further teaches the method further comprising jointly training (i.e. The parameters may have values resulting from joint training of the task embedding network and the control network on training data comprising training samples of a plurality of training tasks; para. [0103, 0104]) the task encoder neural network (i.e. a task embedding network to receive one or more demonstrations of a task and to generate a task embedding, the task embedding comprising a representation of the task, each demonstration comprising one or more observations of a performance of the task; para. [0010, 0042, 0046]) and the state feature neural network (i.e. The control network 130 of FIG. 1 also receives a plurality of successive observations 160 of a robotic device and applies a policy to map the plurality of successive observations 160 of the robotic device to respective control instructions 170 for the robotic device. The policy applied by the control network 130 is modulated across the plurality of successive observations 160 of the robotic device using the task embedding 150 from the task embedding network 120; para. [0048]) to optimize an objective function (i.e. The output of the training iteration, Li, may be used to iteratively optimise the loss function to determine parameter values for the task embedding network and the control network; para. [0111]) that includes a regression loss term (i.e. Accuracy of this classification may be estimated by performing a nearest-neighbour search within embedding space over other tasks in the batch. Although a hinge loss is used in this example, e.g. with a dot product similarity measure, other loss functions may be used in other examples, such as an L2 loss or the like; para. [0112]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the invention of Schaul to include the feature of James. One would have been motivated to make this modification because a learned task embedding provides a compact latent representation in which similar tasks located near one another and permits a control network trained on prior tasks to adapt to a new or unseen task without retraining the control network.
Claim 16: Schaul, James, Osindero, and Barreto teach the method of claim 14. Schaul does not explicitly teach generating a quality score estimate for the training task using the value function representation; and training the successor feature neural network to maximize future quality score estimates.
However, Barreto further teaches generating a quality score estimate for the training task (i.e. equation 4 where φt = φ(st,at,st+1) and ψπ(s,a) are the SFs of (s, a) under policy π; Section 2.2, page 3) using the value function representation (i.e. As discussed in Section 3.2, when using rewards as fea tures the resulting SFs are collections of value functions: ˜ ψπi = ˜ Qπi ≡ [˜Qπi 1 , ˜Qπi 2 ,..., ˜ Qπi D ]. This leads to a particu larly simple way of building the features ˜ φ while retaining transferable knowledge in ˜ Ψ. Given a set of D base tasks Mi, while solving them we only need to carry out two extra operations: compute approximations ˜ri(s,a,s) of the func tions ri(s, a,s), to be used as ˜ φ, and evaluate the resulting policies on all tasks—i.e., compute ˜ Qπi j —to build ˜; Section 4.2, page 5); and training the successor feature neural network to maximize future quality score estimates (i.e. a←argmaxb ˜Qπi i (s,b); Algorithm 2 and Section 5.2).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to modify the combination of Schaul, James, and Osindero to include the feature of Barreto. One would have been motivated to make this modification because it facilitates reuse of learned policies and transfer to unseen tasks without requiring a separately learned value function for every new task.
Allowable Subject Matter
Claims 9, 10, 12, 13, and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Florence et al. (Pub. No. US 20250144795 A1), receiving one or more observations of an environment; receiving an input text sequence that describes a task to be performed by a robot in the environment; generating an encoded representation of the input text sequence in an embedding space; generating a corresponding encoded representation of each of the one or more observations in the embedding space; generating a sequence of input tokens that comprises the encoded representation of the input text sequence and the corresponding encoded representation of each observation.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAN TRAN whose telephone number is (303)297-4266. The examiner can normally be reached on Monday - Thursday - 8:00 am - 5:00 pm MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached on 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAN H TRAN/Primary Examiner, Art Unit 2141
Prosecution Insights