DETAILED ACTION
This final rejection is responsive to the amendment filed on August 7, 2025. Claims 16-35 are pending. Claims 16, 29, and 34 are independent.
Claim rejections of claims 29-33 under 35 USC §101 for being directed to non-statutory subject matter are withdrawn in light of applicant’s amendment. However, rejections of claims 16-35 for being directed to an abstract idea without significantly more are maintained – see sections Claim Rejections – 35 USC §101 and Response to Arguments below.
Claim rejections under 35 USC §102 is withdrawn, however, a new grounds of rejection under 35 USC §103 has been made in light of applicant’s amendments. See sections Claim Rejections – 35 USC §103 and Response to Arguments below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on July 30, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 16-35 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 16:
Claim 16 recites a method and therefore falls within the statutory category of a process. The claim also recites processing a respective policy input through a policy model to obtain a policy output as well as determining that a sequence meets a threshold. The foregoing can practically be performed in the human mind. For instance, a person is capable of taking an input and using a sequence of equations (a model), mentally calculating a respective output. Likewise, a person is capable of comparing values with a predefined number and determining if the values meet that threshold. Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. The claim also recites a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 16 recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. Receiving data (receiving observations and obtaining a reward), electronic record keeping (maintaining a list of data in a database), and presenting offers and gathering statistics (providing an action to the actor) are well-understood, routine, conventional activities (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755; OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93, respectively), see MPEP 2106.05(d). The claim also recites a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 17:
Claim 17 further elaborates on the method of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites causing the actor to perform a respective action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 17 recites causing the actor to perform a respective action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 18:
Claim 18 further elaborates on the causing limitation of claim 17 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites causing the actor to send one or more inputs to a real-world agent and that the real-world agent is configured to receive the one or more inputs and perform the action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 18 recites causing the actor to send one or more inputs to a real-world agent and that the real-world agent is configured to receive the one or more inputs and perform the action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 19:
Claim 19 further elaborates on the causing limitation of claim 17 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites that the environment is a simulated environment and causing the actor to execute the action in the simulated environment which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 19 recites that the environment is a simulated environment and causing the actor to execute the action in the simulated environment which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 20:
Claim 20 further elaborates on the obtaining a reward limitation of claim 16. The claim also recites generating the reward. The foregoing can practically be performed in the human mind. For instance, a person is capable of using a reward function to input information into and using the reward . Therefore, these limitations fall within the "mental processes" grouping of abstract ideas. There are no additional elements recited.
Regarding claim 21:
Claim 21 further elaborates on the maintaining the sequence limitation of claim 16. The claim also recites generating a tuple with an observation, an action, and a reward. The foregoing can practically be performed in the human mind. For instance, a person is capable of observing that it is raining outside, deciding to put on a raincoat (an action), then staying dry while walking between buildings (a reward). Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 21 recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. Electronic recordkeeping (maintaining a list of data in a database) is a well-understood, routine, conventional activity (Ultramercial, 722 F.3d at 716, 112 USPQ2d at 1755), see MPEP 2106.05(d). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 22:
Claim 22 further elaborates on the policy model of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites that the policy model is a LSTM neural network and that processing the observation and reward through the model comprises maintaining a recurrent state which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 22 recites that the policy model is a LSTM neural network and that processing the observation and reward through the model comprises maintaining a recurrent state which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 23:
Claim 23 further elaborates on the training limitation of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites training the policy model using an off-policy reinforcement learning technique which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 23 recites training the policy model using an off-policy reinforcement learning technique which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 24:
Claim 24 further elaborates on the training limitation of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites adding tuples to a priority replay buffer which is nothing more than insignificant extra solution activity. The claim also recites training the policy model on tuples sampled from the priority replay buffer which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 24 recites adding tuples to a priority replay buffer which is nothing more than insignificant extra solution activity. Electronic recordkeeping (maintaining a list of data in a database) is a well-understood, routine, conventional activity (Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755), see MPEP 2106.05(d). The claim also recites training the policy model on tuples sampled from the priority replay buffer which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 25:
Claim 25 further elaborates on the processing limitation of claim 16. The claim also recites batching the policy inputs, and processing the batched inputs through the policy model to obtain a batched output. The foregoing can practically be performed in the human mind. For instance, a person is capable of mentally sorting a list of information into smaller groups, which is analogous to batching the data, then taking that batched input and, using a sequence of equations (a model), mentally calculating a respective output. Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites a policy model which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 25 recites a policy model which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 26:
Claim 26 further elaborates on the method of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites the actors do not comprise the policy model which is nothing more than generally linking the judicial exception to a particular technological environment. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 26 recites the actors do not comprise the policy model which is nothing more than generally linking the judicial exception to a particular technological environment. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 27:
Claim 27 further elaborates on the receiving the observations of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites receiving the observations as part of one or more remote procedure calls which is nothing more than insignificant extra-solution activity. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 27 recites receiving the observations as part of one or more remote procedure calls which is nothing more than insignificant extra-solution activity. Receiving data (receiving the observations) is a well-understood, routine, conventional activity (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362), see MPEP 2106.05(d). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 28:
Claim 28 further elaborates on the processing and training limitations of claim 16 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites that the processing is done on one or more first hardware accelerat which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 28 recites that the processing is done on one or more first hardware accelerators, the training is done on one or more second hardware accelerators, and the first and second accelerators are a predetermined ratio which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 29:
Claim 29 recites one or more non-transitory computer-readable storage media and therefore falls within the statutory category of a machine. The claim also recites processing a respective policy input through a policy model to obtain a policy output as well as determining that a sequence meets a threshold. The foregoing can practically be performed in the human mind. For instance, a person is capable of taking an input and using a sequence of equations (a model), mentally calculating a respective output. Likewise, a person is capable of comparing values with a predefined number and determining if the values meet that threshold. Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. The claim also recites one or more non-transitory computer-readable storage media, one or more computers, a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 29 recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. Receiving data (receiving observations and obtaining a reward), electronic record keeping (maintaining a list of data in a database), and presenting offers and gathering statistics (providing an action to the actor) are well-understood, routine, conventional activities (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755; OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93, respectively), see MPEP 2106.05(d). The claim also recites one or more non-transitory computer-readable storage media, one or more computers, a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 30:
Claim 30 further elaborates on the non-transitory computer-readable storage media of claim 29 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites causing the actor to perform a respective action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 30 recites causing the actor to perform a respective action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 31:
Claim 31 further elaborates on the causing limitation of claim 30 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites causing the actor to send one or more inputs to a real-world agent is configured to receive the one or more inputs and perform the action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 31 recites causing the actor to send one or more inputs to a real-world agent is configured to receive the one or more inputs and perform the action which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 32:
Claim 32 further elaborates on the causing limitation of claim 30 and, therefore, also falls within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites that the environment is a simulated environment and causing the actor to execute the action in the simulated environment which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 32 recites that the environment is a simulated environment and causing the actor to execute the action in the simulated environment which is nothing more than mere instructions to apply the judicial exception using a generic computer. The causing limitation of the claim is written at such a high level of generality that it “fails to recite details of how a solution to a problem is accomplished” (Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44), see MPEP 2106.05(f). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 33:
Claim 33 further elaborates on the maintain of claim 29. The claim also recites generating a tuple with an observation, an action, and a reward. The foregoing can practically be performed in the human mind. For instance, a person is capable of observing that it is raining outside, deciding to put on a raincoat (an action), then staying dry while walking between . Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 33 recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. Electronic recordkeeping (maintaining a list of data in a database) is a well-understood, routine, conventional activity (Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755), see MPEP 2106.05(d). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 34:
Claim 34 recites a system and therefore falls within the statutory category of a machine. The claim also recites processing a respective policy input through a policy model to obtain a policy output as well as determining that a sequence meets a threshold. The foregoing can practically be performed in the human mind. For instance, a person is capable of taking an input and using a sequence of equations (a model), mentally calculating a respective output. Likewise, a person is capable of comparing values with a predefined number and determining if the values meet that threshold. Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. The claim also recites one or more computers, one or more storage devices, a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 29 recites receiving observations, providing an action to the actor, obtaining a reward, and maintaining at least one tuple which is nothing more than insignificant extra-solution activity. Receiving data (receiving observations and obtaining a reward), electronic record keeping (maintaining a list of data in a database), and presenting offers and gathering statistics (providing an action to the actor) are well-understood, routine, conventional activities (Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362; Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755; OIP Techs., 788 F.3d at 1362-63, 115 USPQ2d at 1092-93, respectively), see MPEP 2106.05(d). The claim also recites one or more computers, one or more storage devices, a centralized learner system, a policy model, training the policy model, and that actions in a first subset and second subset are obtained in accordance with the policy model having a first and second set of model parameter values, respectively, which is nothing more than mere instructions to apply the judicial exception using a generic computer. The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Regarding claim 35:
Claim 35 further elaborates on the maintaining the sequence limitation of claim 34. The claim also recites generating a tuple with an observation, an action, and a reward. The foregoing can practically be performed in the human mind. For instance, a person is capable of observing that it is raining outside, deciding to put on a raincoat (an action), then staying dry while walking between buildings (a reward). Therefore, these limitations fall within the "mental processes" grouping of abstract ideas.
The judicial exception is not integrated into a practical application. The claim recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. The claim as a whole, looking at the additional elements individually and in combination does not integrate the judicial exception into a practical application. Therefore, the claim is directed to an abstract idea.
The claim does not recite additional elements that amount to significantly more than the judicial exception. In particular, claim 35 recites adding the tuple to a sequence of tuples which is nothing more than insignificant extra-solution activity. Electronic recordkeeping (maintaining a list of data in a database) is a well-understood, routine, conventional activity (Ultramercial, 772 F.3d at 716, 112 USPQ2d at 1755), see MPEP 2106.05(d). The additional elements, taken individually and in combination, do not result in the claim, as a whole, amounting to significantly more than the identified judicial exception. The claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 16-21, 23-24, 26-27, and 29-35 are rejected under 35 U.S.C. 103 as being unpatentable over Budden et al. (US11625604), hereinafter Budden, in view of Levine et al. (US20190232488), hereinafter Levine.
Regarding claim 16, Budden teaches the method:
receiving, at a centralized learner system, respective observations generated by respective actors for each environment of a plurality of environments; (Budden, column 7, lines 43-46: “The actor computing unit 112 then generates an experience tuple from the observation, the selected action, and the transition data.” And column 8, lines 12-15: “The actor computing unit 112 then determines a priority for the experience tuple, and stores the experience tuple 124 in association with the priority 122 in the replay memory 130” – The replay memory is a part of the reinforcement learning system, see Fig. 1, and is analogous to the centralized learner system. The actor computer unit generating experience tuple from the observation and sending it to the replay memory is analogous to receiving observations generated by the actors for each environment.)
processing, by the centralized learner system for each environment, a respective policy input that includes the respective observation for the environment through a policy model having a plurality of model parameter values to obtain a respective policy output for the actor that defines a control policy for performing a task in the environment; (Budden, column 6, lines 14-18: “In some implementations, the action selection neural network 110 can select an action to be performed by the agent 102 in response an observation by using an action selection policy (exploration policy).” – The action selection neural network is a part of the reinforcement learning system, see Fig. 1, and is therefore analogous to processing by the centralized learner system. The action selection neural network is analogous to the policy model, where a neural network has a plurality of parameter values to obtain an output. The action selection neural network selecting an action to be performed by the agent in response to an observation using an action selection policy is analogous to processing the policy input through a policy model to obtain a policy output that defines a control policy.)
providing, from the centralized learner system to the respective actor for each of the environments, a respective action determined from the control policy defined by the respective policy output for the environment; (Budden, column 7, lines 29-34: “The actor computing unit 112 is configured to receive an observation characterizing a current state of the environment instance 132 and to select an action to be performed by the agent 102 using the action selection neural network replica 118 and in accordance with current values of the network parameters.” – As noted above, the action selection neural network is part of the reinforcement learning system and is therefore analogous to providing from the centralized learner system. The action selection neural network replica, control policy, determining the action for the actor computing unit is analogous to providing the respective action to the actor for each environment.)
obtaining, for each of the environments, a respective reward for the respective actor for the environment generated as a result of the provided action being performed in the environment; (Budden, column 7, lines 41-43: “The transition data also includes a reward which is a numeric value that is received from the environment as a result of the agent 102 performing the selected action.” – The transition data including a reward as a result of the agent performing the action is analogous to obtaining a reward as a result of the provided action being performed in the environment.)
maintaining, by the centralized learner system for each environment, a respective sequence of tuples, at least one tuple comprising a respective observation, an action, and a reward obtained in response to the actor performing the action in the environment; (Budden, column 7, lines 43-50: “The actor computing unit 112 then generates an experience tuple from the observation, the selected action, and the transition data. An experience tuple, which can be denoted as
S
t
,
A
t
,
r
t
+
1
,
S
t
+
1
, includes the current observation
S
t
, the selected action
A
t
, and the reward
r
t
+
1
, and the next observation
S
t
+
1
that characterizes the next state of the environment after the selected action was performed.” – The actor computing unit is a part of the reinforcement learning system and is therefore analogous to maintaining by the centralized learner system.)
determining, by the centralized learner system, that a maintained sequence meets a threshold condition; and (Budden, column 10, lines 30-33: “To maintain the capacity of replay memory within a threshold, the learner computing unit 120 can determine whether criteria for removing any experience tuples from the shared memory are satisfied.” – The learner computer unit is a part of the reinforcement learning system and is therefore analogous to determining by the centralized learner system while determining whether the criteria are satisfied is analogous to determining that a maintained sequence meets a threshold condition.)
in response, training, by the centralized learner system, the policy model on the maintained sequence, (Budden, column 9, lines 1-5: “The learner computing unit 120 (hereafter referred to as the learner 120) determines, using the sampled experience tuples, an update to the network parameters of the action selection neural network 110 using a reinforcement learning technique.” – The learner computing unit is a part of the reinforcement learning system and is therefore analogous to training by the centralized learner system.)
Budden does not explicitly teach:
wherein actions in a first subset of the maintained sequence of tuples are obtained in accordance with the policy model having a first set of model parameter values, and actions in a second subset of the maintained sequence of tuples that are different from the first subset are obtained in accordance with the policy model having a second set of model parameters that (i) are different from the first set of model parameter values and (ii) have been updated by the centralized learner system.
However, Levine teaches:
wherein actions in a first subset of the maintained sequence of tuples are obtained in accordance with the policy model having a first set of model parameter values, and actions in a second subset of the maintained sequence of tuples that are different from the first subset are obtained in accordance with the policy model having a second set of model parameters that (i) are different from the first set of model parameter values and (ii) have been updated by the centralized learner system. (Levine, paragraph 0061: “The training engine 114 iteratively trains one or more parameters of the policy network 124 utilizing techniques such as those described herein (e.g., those related to Q-learning such as the NAF and/or DDPG variants). At each iteration of training, the training engine 114 may generate updated policy parameters utilizing a group of one or more instances of experience data of the replay buffer 122.” – The training engine is analogous to the centralized learner system. The group of one or more instances corresponds to a subset of the maintained sequence of tuples, see e.g. paragraph 0006. Generating the updated policy parameters utilizing a group of one or more instances after each iteration indicates that the actions are obtained in accordance with the first set of model parameter values. Since this is an iterative process, the next iteration would have a set of model parameters that is different than the first which are used to obtain the actions of the second subset of the maintained sequence of tuples. See paragraph 0051 for the trainer thread.)
Levine is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Budden, which already teaches the method of reinforcement training using a centralized learner system but does not explicitly teach training the policy model using actions from a first subset and second subset where the actions are obtained according to the policy model having different parameters, to include the teachings of Levine which does teach training the policy model using actions from a first subset and second subset where the actions are obtained according to the policy model having different parameters in order to facilitate "rapid learning of optimal approaches, or policies, for performing particular physical tasks using the one or more robots." (Levine, paragraph 0002)
Regarding claim 17, Budden and Levine teach the method of claim 16, as cited above.
Budden further teaches:
causing the actor to perform, in an environment, a respective action defined by the respective policy output provided to the actor. (Budden, column 1, lines 51-57: “The system has a distributed architecture that decouples acting from learning: multiple actor computing units ( or actors) interact with their own instances of the environment by selecting actions according to a shared actions selection neural network, and accumulate the resulting experience in a shared experience replay memory;” – The actors selecting actions according to a shared action selection neural network is analogous to causing the actor to perform a respective action.)
Regarding claim 18, Budden and Levine teach the method of claim 17, as cited above.
Budden further teaches:
wherein the environment is a real-world environment, and wherein causing the actor to perform, in the environment, the respective action defined by the respective policy output provided to the actor comprises: (Budden, column 3, lines 43-45: “In some other implementations, the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment.”)
causing the actor to send one or more inputs to a real-world agent in the real-world environment corresponding to the respective action, wherein the real-world agent is configured to receive the one or more inputs from the actor and perform the respective action in the real- world environment. (Budden, column 4, lines 23-30: “In other words, the actions can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. Action data may additionally or alternatively include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment.”)
Regarding claim 19, Budden and Levine teach the method of claim