DETAILED ACTION
This action is in response to the Applicant Response filed 12 September 2024 for application 18/846,412 filed 12 September 2024.
Claim(s) 3-11, 13-18, 20, 23, 25-27, 30-31 is/are currently amended.
Claim(s) 22 is/are cancelled.
Claim(s) 1-21, 22-31 is/are pending.
Claim(s) 1-21, 23-31 is/are rejected.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim(s) 1-21, 23-31 is/are objected to because of the following informalities:
Claim 1, line 3, the policy input should read “the base policy input”
Claim 1, line 20, for the set of one or more parameters should read “for each of the set of one or more parameters”
Claim 2, lines 2-3, an action selected using should read “an action
Claim 2, line 3, wherein the parameters that define the exploration strategy comprise should read “wherein the set of one or more parameters that defines the exploration strategy comprises”
Claim 3, line 2, to the policy output should read “to a policy output”
Claim 3, line 3, wherein the parameters that define the exploration strategy comprise should read “wherein the set of one or more parameters that defines the exploration strategy comprises”
Claim 4, line 2, to the policy output should read “to a policy output”
Claim 4, line 3, wherein the parameters that define the exploration strategy comprise should read “wherein the set of one or more parameters that defines the exploration strategy comprises”
Claim 7, lines 1-2, wherein the operations further comprise: should read “further comprising:”
Claim 8, lines 1-2, wherein the data characterizing should read “wherein
Claim 12, line 6, identified tuples should read “identified one or more tuples”
Claim 14, line 11, identified tuples should read “identified one or more tuples”
Claim 28, line 2, after the training should read “after
Claim 29, line 2, after the training should read “after
Claim 30, line 20, for the set of one or more parameters should read “for each of the set of one or more parameters”
Claim 31, line 18, for the set of one or more parameters should read “for each of the set of one or more parameters”
Claims 2-29 are objected to due to their dependence, either directly or indirectly, on claims 1-4, 7-8, 12, 14, 28-29
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-21, 23-31 is/are rejected under 35 U.S.C. 101, because the claim(s) is/are directed to an abstract idea, and because the claim elements, whether considered individually or in combination, do not amount to significantly more than the abstract idea, see Alice Corporation Pty. Ltd. V. CLS Bank International et al., 573 US 208 (2014).
Regarding claim 1, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of generating training data for training the base policy neural network by controlling an agent using (i) the base policy neural network and (ii) an exploration strategy that maps, in accordance with a set of one or more parameters, base policy outputs generated by the base policy neural network to actions performed by the agent to interact with an environment, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of at each of a plurality of time points: determining that criteria for updating the exploration strategy are satisfied at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: generating a meta policy input that comprises data characterizing a performance of the base policy neural network in controlling the agent at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: ... processing the meta policy input using a meta policy to generate a meta policy output that specifies respective values for each of the set of one or more parameters that define the exploration strategy, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites additional element(s) – one or more computers, agent. The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)).
The claim recites additional element(s) – base policy neural network, exploration strategy, meta policy. The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)).
The claim recites in response to determining that the criteria are satisfied: ... controlling the agent using the base policy neural network and in accordance with the exploration strategy defined by the respective values for the set of one or more parameters specified by the meta policy output which is simply applying the model recited at a high level of generality and amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer (MPEP 2106.05(f)).
The claim recites ... a base policy neural network that is configured to receive a base policy input comprising an observation of a state of an environment and to process the policy input to generate a base policy output that defines an action to be performed by an agent in response to the observation which is simply additional information regarding the model, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
one or more computers, agent amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b))
applying the model amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
base policy neural network, exploration strategy, meta policy amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h))
additional information regarding the model do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 2, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of wherein the exploration strategy is an s greedy exploration strategy that selects, as an action to be performed by the agent, an action selected using a policy output generated by the base policy neural network with probability 1 - ε and a random action with probability ε, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of mathematical concepts, then it falls within the "Mathematical Concepts" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites wherein the parameters that define the exploration strategy comprise one or more parameters that specify a value of ε which is simply additional information regarding the data, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 3, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of wherein the exploration strategy applies a softmax to the policy output generated by the base policy neural network, as drafted, is a process that, under its broadest reasonable interpretation, covers a mathematical concept. The limitation encompasses applying a softmax function.
If a claim limitation, under its broadest reasonable interpretation, covers performance of mathematical concepts, then it falls within the "Mathematical Concepts" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites wherein the parameters that define the exploration strategy comprise one or more parameters that specify a temperature parameter τ for the softmax which is simply additional information regarding the data, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 4, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of wherein the exploration strategy applies noise to the policy output generated by the base policy neural network, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of mathematical concepts, then it falls within the "Mathematical Concepts" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites wherein the parameters that define the exploration strategy comprise one or more parameters that specify how the noise is generated which is simply additional information regarding the data, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 5, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 5 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the criteria are satisfied at every N environment time points steps, wherein N is an integer greater than or equal to one.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the criteria and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the criteria do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 6, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 6 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the criteria are satisfied after a task episode associated with a predefined task is completed, wherein the task episode is completed when the agent has successfully completed the predefined task or when a terminal state of the environment is reached.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the criteria and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the criteria do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 7, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of in response to determining that training termination criteria are satisfied at the time point: controlling the agent using the base policy neural network without applying the exploration strategy, comprising: selecting actions to be performed by the agent using the base policy neural network without applying the exploration strategy, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites in response to determining that training termination criteria are satisfied at the time point: controlling the agent using the base policy neural network without applying the exploration strategy, comprising: ... receiving a respective reward in response to the agent performing each of the actions, which is simply receiving data recited at a high level of generality. This is nothing more than insignificant extra-solution activity (MPEP 2106.05(g)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
receiving data amount(s) to no more than insignificant extra-solution activity (MPEP 2106.05(g)), wherein the insignificant extra-solution activity is the well-understood routine and conventional activit(y/ies) of receiving or transmitting data over a network (MPEP 2016.05(d))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 8, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 8 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the data characterizing a performance of the base policy neural network in controlling the agent at the time point comprises data characterizing respective rewards received while controlling the agent using the base policy neural network.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the data and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 9, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 9 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the meta policy input further comprises data characterizing a difference in (i) performance of the base policy neural network in controlling the agent at the time point and (ii) performance of the base policy neural network in controlling the agent at a most recent time point at which the criteria were satisfied.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the data and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 10, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 10 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the meta policy input further comprises data identifying the time point at which the criteria are satisfied.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the data and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 11, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 11 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the training data comprises tuples that each specify at least (i) an observation characterizing a state of the environment, (ii) an action performed in response to the observation, and (iii) a reward received in response to the action being performed.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the data and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the data do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 12, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of while generating the training data, repeatedly performing training steps, wherein performing each training step comprises: identifying one or more tuples that have been generated as of the training step, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites while generating the training data, repeatedly performing training steps, wherein performing each training step comprises: ... training the base policy neural network on the identified tuples through reinforcement learning which is simply generic training to perform the abstract idea of model generation and amounts to mere instructions to apply the exception (MPEP 2106.05(f)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
generic training to perform the abstract idea amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 13, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of while generating the training data: determining that criteria are satisfied for updating the meta policy, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response: determining a meta reward based on a performance of the base policy neural network in controlling the agent since a preceding time point at which the criteria for updating the meta policy were satisfied, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response: ... updating the meta policy using the meta reward through reinforcement learning to maximize an expected time-discounted sum of meta rewards, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated
into a practical application. The claim does not recite any additional elements which integrate the
abstract idea into a practical application and, therefore, does not impose any meaningful limits on
practicing the abstract idea. Therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to
significantly more than the judicial exception. As discussed above with respect to the integration of the
abstract idea into a practical application, the claim does not recite any additional elements which
provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 14, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 14 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network.
The limitation of determining that criteria are satisfied for updating the meta policy when a threshold number of training steps have been performed since a previous time step that the meta policy was updated, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of wherein performing each training step in the threshold number of training steps comprises: identifying one or more tuples that have been generated as of the training step, wherein each tuple specifies at least (i) an observation characterizing a state of the environment, (ii) an action performed in response to the observation, and (iii) a reward received in response to the action being performed, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites wherein performing each training step in the threshold number of training steps comprises: ... training the base policy neural network on the identified tuples through reinforcement learning which is simply generic training to perform the abstract idea of model generation and amounts to mere instructions to apply the exception (MPEP 2106.05(f)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
generic training to perform the abstract idea amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 15, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 15 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 13 is applicable here since claim 15 carries out the method of claim 13 but for the recitation of additional element(s) of wherein the meta policy has previously been updated based on interactions of a different agent in the environment while controlled by the base policy neural network.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the meta policy and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the meta policy do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 16, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 16 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 13 is applicable here since claim 16 carries out the method of claim 13 but for the recitation of additional element(s) of wherein the meta policy has previously been updated based on interactions of the agent in a different environment while controlled by the base policy neural network.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the meta policy and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the meta policy do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 17, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 17 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 13 is applicable here since claim 17 carries out the method of claim 13 but for the recitation of additional element(s) of wherein the meta policy has previously been updated based on interactions of a different agent in a different environment while controlled by a different base policy neural network.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the meta policy and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the meta policy do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 18, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 18 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 13 is applicable here since claim 18 carries out the method of claim 13 but for the recitation of additional element(s) of wherein the meta reward is a difference between (i) the performance of the base policy neural network in controlling the agent at the time step at which the criteria for updating the meta policy are satisfied and (ii) the performance of the base policy neural network in controlling the agent at a preceding time step at which the criteria for updating the meta policy were satisfied.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the meta reward and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the meta reward do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 19, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 19 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 18 is applicable here since claim 19 carries out the method of claim 18 but for the recitation of additional element(s) of wherein the performance is measured based on rewards received in response to actions performed by the agent.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the model performance and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the model performance do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 20, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 20 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 20 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the meta policy has been learned through reinforcement learning based on meta rewards.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites wherein the meta policy has been learned through reinforcement learning based on meta rewards which is simply additional information regarding the meta policy, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
The claim recites additional element(s) – reinforcement learning. The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
reinforcement learning amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h))
additional information regarding the meta policy do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 21, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 21 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 20 is applicable here since claim 21 carries out the method of claim 20 but for the recitation of additional element(s) of wherein the meta rewards include meta rewards computed from performance of one or more agents in one or more environments while controlled by one or more different base policy neural networks.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the meta reward and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the meta reward do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 23, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 23 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 23 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the agent is a mechanical agent and the environment is a real-world environment.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the agent and the environment and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the agent and the environment do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 24, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 24 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 23 is applicable here since claim 24 carries out the method of claim 23 but for the recitation of additional element(s) of wherein the agent is a robot.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the agent and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the agent do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 25, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 25 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 25 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the agent and the environment and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the agent and the environment do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 26, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 26 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 26 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the agent and the environment and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the agent and the environment do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 27, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 27 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 27 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the environment is a simulated environment and the agent is a simulated agent.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the agent and the environment and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the agent and the environment do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 28, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 28 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 27 is applicable here since claim 28 carries out the method of claim 27 but for the recitation of additional element(s) of after the training, deploying the base policy neural network for use in controlling a real-world agent interacting with a real-world environment.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites after the training, deploying the base policy neural network for use in controlling a real-world agent interacting with a real-world environment, which is simply transmitting data recited at a high level of generality. This is nothing more than insignificant extra-solution activity (MPEP 2106.05(g)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
transmitting data amount(s) to no more than insignificant extra-solution activity (MPEP 2106.05(g)), wherein the insignificant extra-solution activity is the well-understood routine and conventional activit(y/ies) of receiving or transmitting data over a network (MPEP 2016.05(d))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 29, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 29 is directed to a method, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) method ... for training a base policy neural network. The Step 2A Prong One Analysis for claim 27 is applicable here since claim 29 carries out the method of claim 27 but for the recitation of additional element(s) of after the training, controlling a real-world agent interacting with a real-world environment using the base policy neural network.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites after the training, controlling a real-world agent interacting with a real-world environment using the base policy neural network which is simply applying the model recited at a high level of generality and amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer (MPEP 2106.05(f)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
applying the model amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 30, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 30 is directed to a system with a computer, which is directed to a machine, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites a(n) system.
The limitation of generating training data for training the base policy neural network by controlling an agent using (i) the base policy neural network and (ii) an exploration strategy that maps, in accordance with a set of one or more parameters, base policy outputs generated by the base policy neural network to actions performed by the agent to interact with an environment, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of at each of a plurality of time points: determining that criteria for updating the exploration strategy are satisfied at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: generating a meta policy input that comprises data characterizing a performance of the base policy neural network in controlling the agent at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: ... processing the meta policy input using a meta policy to generate a meta policy output that specifies respective values for each of the set of one or more parameters that define the exploration strategy, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites additional element(s) – system, one or more computers, one or more storage devices, instructions, agent. The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)).
The claim recites additional element(s) – base policy neural network, exploration strategy, meta policy. The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)).
The claim recites in response to determining that the criteria are satisfied: ... controlling the agent using the base policy neural network and in accordance with the exploration strategy defined by the respective values for the set of one or more parameters specified by the meta policy output which is simply applying the model recited at a high level of generality and amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer (MPEP 2106.05(f)).
The claim recites ... a base policy neural network that is configured to receive a base policy input comprising an observation of a state of an environment and to process the policy input to generate a base policy output that defines an action to be performed by an agent in response to the observation which is simply additional information regarding the model, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
system, one or more computers, one or more storage devices, instructions, agent amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b))
applying the model amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
base policy neural network, exploration strategy, meta policy amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h))
additional information regarding the model do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Regarding claim 31, the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 31 is directed to computer storage media, which is directed to an article of manufacture, one of the statutory categories.
Step 2A Prong One Analysis: The claim recites one or more computer storage media.
The limitation of generating training data for training the base policy neural network by controlling an agent using (i) the base policy neural network and (ii) an exploration strategy that maps, in accordance with a set of one or more parameters, base policy outputs generated by the base policy neural network to actions performed by the agent to interact with an environment, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of at each of a plurality of time points: determining that criteria for updating the exploration strategy are satisfied at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: generating a meta policy input that comprises data characterizing a performance of the base policy neural network in controlling the agent at the time point, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
The limitation of in response to determining that the criteria are satisfied: ... processing the meta policy input using a meta policy to generate a meta policy output that specifies respective values for each of the set of one or more parameters that define the exploration strategy, as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper.
If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea.
Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application.
The claim recites additional element(s) – one or more computer storage media, instructions, one or more computers, agent. The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)).
The claim recites additional element(s) – base policy neural network, exploration strategy, meta policy. The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)).
The claim recites in response to determining that the criteria are satisfied: ... controlling the agent using the base policy neural network and in accordance with the exploration strategy defined by the respective values for the set of one or more parameters specified by the meta policy output which is simply applying the model recited at a high level of generality and amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer (MPEP 2106.05(f)).
The claim recites ... a base policy neural network that is configured to receive a base policy input comprising an observation of a state of an environment and to process the policy input to generate a base policy output that defines an action to be performed by an agent in response to the observation which is simply additional information regarding the model, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)).
Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of:
one or more computer storage media, instructions, one or more computers, agent amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b))
applying the model amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f))
base policy neural network, exploration strategy, meta policy amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h))
additional information regarding the model do(es) not apply the exception in a meaningful way (MPEP 2106.05(e))
The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 4-5, 8-12, 20-21, 30-31 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu et al. (Learning to Explore with Meta-Policy Gradient, hereinafter referred to as “Xu”).
Regarding claim 1 (Original), Xu teaches a method performed by one or more computers (Xu, section 5.1 – hardware and software to perform methods) and for training a base policy neural network that is configured to receive a base policy input comprising an observation of a state of an environment and to process the policy input to generate a base policy output that defines an action to be performed by an agent in response to the observation (Xu, section 3.2 – teaches a neural network actor policy [base policy] that maps environment states to actions), the method comprising:
generating training data for training the base policy neural network by controlling an agent using (i) the base policy neural network and (ii) an exploration strategy that maps, in accordance with a set of one or more parameters, base policy outputs generated by the base policy neural network to actions performed by the agent to interact with an environment (Xu, sections 4-4.2 – teaches generating training data for the actor policy [base policy network] based on the reward of the actor policy and which maps the resulting state from the action taken by the actor policy and a next action), the generating comprising, at each of a plurality of time points (Xu, Algorithm 1 – teaches each time iteration t):
determining that criteria for updating the exploration strategy are satisfied at the time point (Xu, Algorithm 1 – teaches each time iteration t); and
in response to determining that the criteria are satisfied:
generating a meta policy input that comprises data characterizing a performance of the base policy neural network in controlling the agent at the time point (Xu, section 4-4.2 – taches an exploration policy [meta policy] that generates data based on the reward performance of the actor network in controlling the agent);
processing the meta policy input using a meta policy to generate a meta policy output that specifies respective values for each of the set of one or more parameters that define the exploration strategy (Xu, section 4.2 – teaches generating an exploration policy [meta policy] based on gradients of the exploration policy to update parameters of the exploration policy); and
controlling the agent using the base policy neural network and in accordance with the exploration strategy defined by the respective values for the set of one or more parameters specified by the meta policy output (Xu, section 5 – teaches experiments to control an agent using the actor network in accordance with the exploration strategy determined using the exploration policy).
Regarding claim 4 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the exploration strategy applies noise to the policy output generated by the base policy neural network, and wherein the parameters that define the exploration strategy comprise one or more parameters that specify how the noise is generated (Xu, section 3.2 – teaches an exploration strategy based on Gaussian noise).
Regarding claim 5 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the criteria are satisfied at every N environment time points, wherein N is an integer greater than or equal to one (Xu, algorithm 1 – teaches each iteration t; see also Xu section 4.2 – teaches one or a few steps).
Regarding claim 8 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the data characterizing a performance of the base policy neural network in controlling the agent at the time point comprises data characterizing respective rewards received while controlling the agent using the base policy neural network (Xu, section 4-4.2 – taches an exploration policy [meta policy] that generates data based on the reward performance of the actor network in controlling the agent).
Regarding claim 9 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the meta policy input further comprises data characterizing a difference in (i) performance of the base policy neural network in controlling the agent at the time point and (ii) performance of the base policy neural network in controlling the agent at a most recent time point at which the criteria were satisfied (, section 4-4.2 – taches an exploration policy [meta policy] that generates data based on the difference of the reward performance between a current and previous time step of the actor network in controlling the agent; see also Algorithm 1).
Regarding claim 10 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the meta policy input further comprises data identifying the time point at which the criteria are satisfied (Xu, section 4.2 – teaches determining the meta reward by running DDPG for one or a small number of steps [Meaning the meta policy has to know the number of steps at which to calculate the meta reward]).
Regarding claim 11 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the training data comprises tuples that each specify at least (i) an observation characterizing a state of the environment, (ii) an action performed in response to the observation, and (iii) a reward received in response to the action being performed (Xu, section 4.2 – teaches transition tuples from information generated by exploration policy including state, action and reward).
Regarding claim 12 (Original), Xu teaches all of the limitations of the method of claim 11 as noted above. Xu further teaches while generating the training data, repeatedly performing training steps, wherein performing each training step comprises:
identifying one or more tuples that have been generated as of the training step (Xu, section 3.2 – teaches updating the actor policy [base policy neural network] based on state/action/reward tuples); and
training the base policy neural network on the identified tuples through reinforcement learning (Xu, section 3.2 – teaches updating the actor policy [base policy neural network] based on state/action/reward tuples).
Regarding claim 20 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches wherein the meta policy has been learned through reinforcement learning based on meta rewards (Xu, section 4.2 – teaches learning the exploration policy through reinforcement learning based on meta rewards).
Regarding claim 21 (Original), Xu teaches all of the limitations of the method of claim 20 as noted above. Xu further teaches wherein the meta rewards include meta rewards computed from performance of one or more agents in one or more environments while controlled by one or more different base policy neural networks (Xu, section 4.2 – teaches learning the exploration policy based on meta rewards from performance of an agent in an environment).
Regarding claim 30 (Currently Amended), it is the system embodiment of claim 1 with similar limitations to claim 1 and is rejected using the same reasoning found in claim 1. Xu further teaches a system comprising:
one or more computers (Xu, section 5.1 – hardware and software to perform methods 0; and
one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising (Xu, section 5.1 – hardware and software to perform methods) …
Regarding claim 31 (Currently Amended), it is the computer storage media embodiment of claim 1 with similar limitations to claim 1 and is rejected using the same reasoning found in claim 1. Xu further teaches one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising (Xu, section 5.1 – hardware and software to perform methods) …
Claim(s) 2-3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Ernst, Damien (Meta-Learning of Exploration-Exploitation Strategies in Reinforcement Learning, hereinafter referred to as “Ernst”).
Regarding claim 2 (Original), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the exploration strategy is an ε greedy exploration strategy that selects, as an action to be performed by the agent, an action selected using a policy output generated by the base policy neural network with probability 1 - ε and a random action with probability ε, and wherein the parameters that define the exploration strategy comprise one or more parameters that specify a value of ε.
Ernst teaches wherein the exploration strategy is an ε greedy exploration strategy that selects, as an action to be performed by the agent, an action selected using a policy output generated by the base policy neural network with probability 1 - ε and a random action with probability ε, and wherein the parameters that define the exploration strategy comprise one or more parameters that specify a value of ε (Ernst, slide 5 – teaches an ε-greedy exploration strategy with selection probability 1- ε and random action with ε).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Ernst in order to provide short term and long term rewards in the field of meta-reinforcement learning (Ernst, slide 1 – “The agent needs to take actions so as to gather information that may help him to discover how to obtain high rewards in the long-term (exploration) and, at the same time, he has to exploit at best its current information to obtain high short-term rewards.”).
Regarding claim 3 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the exploration strategy applies a softmax to the policy output generated by the base policy neural network, and wherein the parameters that define the exploration strategy comprise one or more parameters that specify a temperature parameter τ for the softmax.
Ernst teaches wherein the exploration strategy applies a softmax to the policy output generated by the base policy neural network, and wherein the parameters that define the exploration strategy comprise one or more parameters that specify a temperature parameter τ for the softmax (Ernst, slide 5 – teaches a softmax exploration policy, including a temperature variable).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Ernst in order to provide short term and long term rewards in the field of meta-reinforcement learning (Ernst, slide 1 – “The agent needs to take actions so as to gather information that may help him to discover how to obtain high rewards in the long-term (exploration) and, at the same time, he has to exploit at best its current information to obtain high short-term rewards.”).
Claim(s) 6, 13-14, 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Xu et al. (Meta-Gradient Reinforcement Learning, hereinafter referred to as “Hasselt”)
Regarding claim 6 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the criteria are satisfied after a task episode associated with a predefined task is completed, wherein the task episode is completed when the agent has successfully completed the predefined task or when a terminal state of the environment is reached.
Hasselt teaches wherein the criteria are satisfied after a task episode associated with a predefined task is completed, wherein the task episode is completed when the agent has successfully completed the predefined task or when a terminal state of the environment is reached (Hasselt, section 1.1 – teaches an episode as the return criteria).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Hasselt in order to produce better reinforcement learning performance in the field of meta-reinforcement learning (Hasselt, section 5 – “In this work, we discussed how to learn the meta-parameters of a return function. Our meta-learning algorithm runs online, while interacting with a single environment, and successfully adapts the return to produce better performance. We demonstrated, by adjusting the meta-parameters of a state-of-the-art deep learning algorithm, that we could achieve much higher performance than previously observed on 57 Atari 2600 games from the Arcade Learning Environment.”).
Regarding claim 13 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. Xu further teaches while generating the training data:
determining that criteria are satisfied for updating the meta policy (Algorithm 1 – teaches updating at each time iteration t); and,
in response:
determining a meta reward based on a performance of the base policy neural network in controlling the agent since a preceding time point at which the criteria for updating the meta policy were satisfied (section 4-4.2 – taches an exploration policy [meta policy] that generates data based on the difference of the reward performance between a current and previous time step of the actor network in controlling the agent; see also Algorithm 1).
However, Xu does not explicitly teach updating the meta policy using the meta reward through reinforcement learning to maximize an expected time-discounted sum of meta rewards.
Hasselt teaches updating the meta policy using the meta reward through reinforcement learning to maximize an expected time-discounted sum of meta rewards (Hasselt, section 1.1 – teaches a time discounted sum of meta-rewards).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Hasselt in order to produce better reinforcement learning performance in the field of meta-reinforcement learning (Hasselt, section 5 – “In this work, we discussed how to learn the meta-parameters of a return function. Our meta-learning algorithm runs online, while interacting with a single environment, and successfully adapts the return to produce better performance. We demonstrated, by adjusting the meta-parameters of a state-of-the-art deep learning algorithm, that we could achieve much higher performance than previously observed on 57 Atari 2600 games from the Arcade Learning Environment.”).
Regarding claim 14 (Currently Amended), Xu in view of Hasselt teaches all of the limitations of the method of claim 13 as noted above. Xu further teaches wherein determining that criteria are satisfied for updating the meta policy comprises:
determining that criteria are satisfied for updating the meta policy when a threshold number of training steps have been performed since a previous time step that the meta policy was updated (Xu, Algorithm 1 – teaches updating at each time iteration t), wherein performing each training step in the threshold number of training steps comprises:
identifying one or more tuples that have been generated as of the training step, wherein each tuple specifies at least (i) an observation characterizing a state of the environment, (ii) an action performed in response to the observation, and (iii) a reward received in response to the action being performed (Xu, section 3.2 – teaches updating the actor policy [base policy neural network] based on state/action/reward tuples); and
training the base policy neural network on the identified tuples through reinforcement learning (Xu, section 3.2 – teaches updating the actor policy [base policy neural network] based on state/action/reward tuples).
It would have been obvious to one of ordinary skill in the art before the filing data of the claimed invention to combine the teachings of Xu and Hasselt for the same reasons as disclosed in claim 13 above.
Regarding claim 18 (Currently Amended), Xu in view of Hasselt teaches all of the limitations of the method of claim 13 as noted above. Xu further teaches wherein the meta reward is a difference between (i) the performance of the base policy neural network in controlling the agent at the time step at which the criteria for updating the meta policy are satisfied and (ii) the performance of the base policy neural network in controlling the agent at a preceding time step at which the criteria for updating the meta policy were satisfied (section 4-4.2 – taches an exploration policy [meta policy] that generates data based on the difference of the reward performance between a current and previous time step of the actor network in controlling the agent; see also Algorithm 1).
It would have been obvious to one of ordinary skill in the art before the filing data of the claimed invention to combine the teachings of Xu and Hasselt for the same reasons as disclosed in claim 13 above.
Regarding claim 19 (Original), Xu in view of Hasselt teaches all of the limitations of the method of claim 18 as noted above. Xu further teaches wherein the performance is measured based on rewards received in response to actions performed by the agent (Xu, section 3.1 – teaches using rewards to determine performance of the agent).
It would have been obvious to one of ordinary skill in the art before the filing data of the claimed invention to combine the teachings of Xu and Hasselt for the same reasons as disclosed in claim 18 above.
Claim(s) 7, 23-29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Zheng et al., (US 2021/0089910 A1 – Reinforcement Learning Using Meta-Learned Intrinsic Rewards, hereinafter referred to as “Zheng”).
Regarding claim 7 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach in response to determining that training termination criteria are satisfied at the time point: controlling the agent using the base policy neural network without applying the exploration strategy, comprising: selecting actions to be performed by the agent using the base policy neural network without applying the exploration strategy, and receiving a respective reward in response to the agent performing each of the actions.
Zheng teaches in response to determining that training termination criteria are satisfied at the time point (Zheng, [0082] – teaches the goal has been found):
controlling the agent using the base policy neural network without applying the exploration strategy (Zheng, [0082] – teaches once the goal is found the agent should exploit without exploration), comprising:
selecting actions to be performed by the agent using the base policy neural network without applying the exploration strategy (Zheng, [0082] – teaches once the goal is found the agent should exploit without exploration), and
receiving a respective reward in response to the agent performing each of the actions (Zheng, [0052] – teaches providing an extrinsic reward when an action is performed).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Zheng in order to reduce computational resources for training in the field of meta-reinforcement learning (Zheng, [0025] – “Implementations of the above aspects provides improved methods and systems for training reinforcement learning systems by using a meta-learned intrinsic reward system. The intrinsic reward system is able to encourage both exploratory and exploitative behavior as necessary according to the task and environment. For example, the agent can be encouraged to explore uncertain states even if the overall episodic return is lower than average in order to gather useful information to maximize the extrinsic rewards in the long run. In this way, an agent is able to learn to maximize extrinsic rewards in shorter time than conventional methods relying upon an extrinsic reward only. Therefore, the use of an intrinsic reward system as provided herein is capable of accelerating the training of an agent and thus, the training system requires fewer computational resources, e.g., reduced processor cycles, reduced wall clock time, reduced power consumption, to train an agent.”).
Regarding claim 23 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the agent is a mechanical agent and the environment is a real-world environment.
Zheng teaches wherein the agent is a mechanical agent and the environment is a real-world environment (Zheng, [0041] – teaches an electromechanical agent interacting with a real-world environment).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Zheng in order to reduce computational resources for training in the field of meta-reinforcement learning (Zheng, [0025] – “Implementations of the above aspects provides improved methods and systems for training reinforcement learning systems by using a meta-learned intrinsic reward system. The intrinsic reward system is able to encourage both exploratory and exploitative behavior as necessary according to the task and environment. For example, the agent can be encouraged to explore uncertain states even if the overall episodic return is lower than average in order to gather useful information to maximize the extrinsic rewards in the long run. In this way, an agent is able to learn to maximize extrinsic rewards in shorter time than conventional methods relying upon an extrinsic reward only. Therefore, the use of an intrinsic reward system as provided herein is capable of accelerating the training of an agent and thus, the training system requires fewer computational resources, e.g., reduced processor cycles, reduced wall clock time, reduced power consumption, to train an agent.”).
Regarding claim 24 (Original), Xu in view of Zheng teaches all of the limitations of the method of claim 23 as noted above. Zheng further teaches wherein the agent is a robot (Zheng, [0041] – teaches a robotic agent).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to combine the teachings of Xu and Zheng in order to be used for different agents in different environments to reduce computational resources for training (Zheng, [0025]).
Regarding claim 25 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility.
Zheng teaches wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility (Zheng, [0048] – teaches an electronic agent in a facility controlling electronic and/or mechanical items of equipment).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Zheng in order to reduce computational resources for training in the field of meta-reinforcement learning (Zheng, [0025] – “Implementations of the above aspects provides improved methods and systems for training reinforcement learning systems by using a meta-learned intrinsic reward system. The intrinsic reward system is able to encourage both exploratory and exploitative behavior as necessary according to the task and environment. For example, the agent can be encouraged to explore uncertain states even if the overall episodic return is lower than average in order to gather useful information to maximize the extrinsic rewards in the long run. In this way, an agent is able to learn to maximize extrinsic rewards in shorter time than conventional methods relying upon an extrinsic reward only. Therefore, the use of an intrinsic reward system as provided herein is capable of accelerating the training of an agent and thus, the training system requires fewer computational resources, e.g., reduced processor cycles, reduced wall clock time, reduced power consumption, to train an agent.”).
Regarding claim 26 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product.
Zheng teaches wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product (Zheng, [0048] – teaches an electronic agent in a facility, such as a manufacturing plant, controlling electronic and/or mechanical items of equipment).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Zheng in order to reduce computational resources for training in the field of meta-reinforcement learning (Zheng, [0025] – “Implementations of the above aspects provides improved methods and systems for training reinforcement learning systems by using a meta-learned intrinsic reward system. The intrinsic reward system is able to encourage both exploratory and exploitative behavior as necessary according to the task and environment. For example, the agent can be encouraged to explore uncertain states even if the overall episodic return is lower than average in order to gather useful information to maximize the extrinsic rewards in the long run. In this way, an agent is able to learn to maximize extrinsic rewards in shorter time than conventional methods relying upon an extrinsic reward only. Therefore, the use of an intrinsic reward system as provided herein is capable of accelerating the training of an agent and thus, the training system requires fewer computational resources, e.g., reduced processor cycles, reduced wall clock time, reduced power consumption, to train an agent.”).
Regarding claim 27 (Currently Amended), Xu teaches all of the limitations of the method of claim 1 as noted above. However, Xu does not explicitly teach wherein the environment is a simulated environment and the agent is a simulated agent.
Zheng teaches wherein the environment is a simulated environment and the agent is a simulated agent (Zheng, [0050] – teaches simulated agent and environment; see also Zheng, [0044]-[0045]).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu with the teachings of Zheng in order to reduce computational resources for training in the field of meta-reinforcement learning (Zheng, [0025] – “Implementations of the above aspects provides improved methods and systems for training reinforcement learning systems by using a meta-learned intrinsic reward system. The intrinsic reward system is able to encourage both exploratory and exploitative behavior as necessary according to the task and environment. For example, the agent can be encouraged to explore uncertain states even if the overall episodic return is lower than average in order to gather useful information to maximize the extrinsic rewards in the long run. In this way, an agent is able to learn to maximize extrinsic rewards in shorter time than conventional methods relying upon an extrinsic reward only. Therefore, the use of an intrinsic reward system as provided herein is capable of accelerating the training of an agent and thus, the training system requires fewer computational resources, e.g., reduced processor cycles, reduced wall clock time, reduced power consumption, to train an agent.”).
Regarding claim 28 (Original), Xu in view of Zheng teaches all of the limitations of the method of claim 27 as noted above. Zheng further teaches after the training, deploying the base policy neural network for use in controlling a real-world agent interacting with a real-world environment (Zheng, [0050] – teaches after training in a simulated environment, deploying the system to a real-world agent in a real-world environment).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to combine the teachings of Xu and Zheng in order to be used for different agents in different environments to reduce computational resources for training (Zheng, [0025]).
Regarding claim 29 (Original), Xu in view of Zheng teaches all of the limitations of the method of claim 27 as noted above. Zheng further teaches after the training, controlling a real-world agent interacting with a real-world environment using the base policy neural network (Zheng, [0050] – teaches after training in a simulated environment, deploying the system to a real-world agent in a real-world environment).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to combine the teachings of Xu and Zheng in order to be used for different agents in different environments to reduce computational resources for training (Zheng, [0025]).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Hasselt and further in view of Schaul et al. (Adapting Behavior for Learning Progress, hereinafter referred to as “Schaul”).
Regarding claim 15 (Currently Amended), Xu in view of Hasselt teaches all of the limitations of the method of claim 13 as noted above. However, Xu in view of Hasselt does not explicitly teach wherein the meta policy has previously been updated based on interactions of a different agent in the environment while controlled by the base policy neural network.
Shaul teaches wherein the meta policy has previously been updated based on interactions of a different agent in the environment while controlled by the base policy neural network (Schaul, section 1 – teaches distributed agents with parallel copies of an environment; Schaul, section 6 – teaches dynamically adapting parameters of a reinforcement model).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu in view of Hasselt with the teachings of Schaul in order to enabled larger scales and greater flexibility in the field of meta-reinforcement learning (Schaul, Abstract – “Determining what experience to generate to best facilitate learning (i.e. exploration) is one of the distinguishing features and open challenges in reinforcement learning. The advent of distributed agents that interact with parallel instances of the environment has enabled larger scales and greater flexibility, but has not removed the need to tune exploration to the task, because the ideal data for the learning algorithm necessarily depends on its process of learning. We propose to dynamically adapt the data generation by using a non-stationary multi-armed bandit to optimize a proxy of the learning progress. The data distribution is controlled by modulating multiple parameters of the policy (such as stochasticity, consistency or optimism) without significant overhead. The adaptation speed of the bandit can be increased by exploiting the factored modulation structure. We demonstrate on a suite of Atari 2600 games how this unified approach produces results comparable to per-task tuning at a fraction of the cost.”).
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Hasselt and further in view of Gupta et al. (Meta-Reinforcement Learning of Structured Exploration Strategies, hereinafter referred to as “Gupta”).
Regarding claim 16 (Currently Amended), Xu in view of Hasselt teaches all of the limitations of the method of claim 13 as noted above. However, Xu in view of Hasselt does not explicitly teach wherein the meta policy has previously been updated based on interactions of the agent in a different environment while controlled by the base policy neural network.
Gupta teaches wherein the meta policy has previously been updated based on interactions of the agent in a different environment while controlled by the base policy neural network (Gupta, section 1 – teaches using the same agent in different environments; see also Gupta, section 4 - experiments).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu in view of Hasselt with the teachings of Gupta in order to use prior tasks to inform how exploration should be performed in new tasks in the field of meta-reinforcement learning (Gupta, Abstract – “Exploration is a fundamental challenge in reinforcement learning (RL). Many current exploration methods for deep RL use task-agnostic objectives, such as information gain or bonuses based on state visitation. However, many practical applications of RL involve learning more than a single task, and prior tasks can be used to inform how exploration should be performed in new tasks. In this work, we study how prior tasks can inform an agent about how to explore effectively in new situations. We introduce a novel gradient-based fast adaptation algorithm – model agnostic exploration with structured noise (MAESN) – to learn exploration strategies from prior experience. The prior experience is used both to initialize a policy and to acquire a latent exploration space that can inject structured stochasticity into a policy, producing exploration strategies that are informed by prior knowledge and are more effective than random action-space noise. We show that MAESN is more effective at learning exploration strategies when compared to prior meta-RL methods, RL without learned exploration strategies, and task-agnostic exploration methods. We evaluate our method on a variety of simulated tasks: locomotion with a wheeled robot, locomotion with a quadrupedal walker, and object manipulation.”).
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Xu in view of Hasselt and further in view of Kirsh et al. (Improving Generalization in Meta-Reinforcement Learning Using Learned Objectives, hereinafter referred to as “Kirsh”).
Regarding claim 17 (Currently Amended), Xu in view of Hasselt teaches all of the limitations of the method of claim 13 as noted above. However, Xu in view of Hasselt does not explicitly teach wherein the meta policy has previously been updated based on interactions of a different agent in a different environment while controlled by a different base policy neural network.
Kirsh teaches wherein the meta policy has previously been updated based on interactions of a different agent in a different environment while controlled by a different base policy neural network (Kirsh, section 1 – teaches randomly initialized agents using meta-learned rules on entirely different environments; see also Kirsh, section 5 – experiments).
It would have been obvious to one of ordinary skill in the art before the filing date of the claimed invention to modify Xu in view of Hasselt with the teachings of Kirsh in order to generalize to new environments that are entirely different from those used for meta-training in the field of meta-reinforcement learning (Kirsh, Abstract – “Biological evolution has distilled the experiences of many learners into the general learning algorithms of humans. Our novel meta reinforcement learning algorithm MetaGenRL is inspired by this process. MetaGenRL distills the experiences of many complex agents to meta-learn a low-complexity neural objective function that decides how future individuals will learn. Unlike recent meta-RL algorithms, MetaGenRL can generalize to new environments that are entirely different from those used for meta-training. In some cases, it even outperforms human-engineered RL algorithms. MetaGenRL uses off-policy second-order gradients during meta-training that greatly increase its sample efficiency.”).
Conclusion
Any inquiry concerning this communication or earlier communication from the examiner should be directed to MARSHALL WERNER whose telephone number is (469) 295-9143. The examiner can normally be reached on Monday – Thursday 7:30 AM – 4:30 PM ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at (571) 272-7796. The fax number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARSHALL L WERNER/ Primary Examiner, Art Unit 2125
Prosecution Insights