DETAILED ACTION
This action is in response to amendments and remarks filed 04/30/2026. Claims 1, 3-13, and 15-18 are pending and have been examined.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/30/2026 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1, 3-13, and 15-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “a first task request parameter defining a number of the executable resource task request portions of the resource task request and a second task request parameter defining a timing for executing the executable resource task request portions” in its sixth limitation. While the instant specification discloses “In the depicted embodiment, each policy head 302 may be dedicated to a particular portion of a resource request, and be responsible for selecting from a set of actions related to that portion of the resource request” ([0083]), it fails to disclose “a first task request parameter defining a number of the executable resource task request portions of the resource task request” or “a second task request parameter defining a timing for executing the executable resource task request portions”. Thus, claim 1 contains new matter not described in the instant specification, and fails to comply with the written description requirement. This deficiency is present in substantially similar independent claims 13 and 17, and is inherited by all dependent claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) Claims 1, 3-13, and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over Burhani et al. (TRADE PLATFORM WITH REINFORCEMENT LEARNING, published 12/5/2019, US 2019/0370649 A1), hereafter referred to as Burhani, in view of Van Seijen et al. (SCALABILITY OF REINFORCEMENT LEARNING BY SEPARATION OF CONCERNS, filed 6/26/2017, US 2018/0165602 A1), hereafter referred to as Van Seijen.
Regarding claim 1, Burhani teaches [a] computer-implemented system for automatic generation of resource task requests, the system comprising:
a communication interface; at least one processor; memory in communication with the at least one processor; and software code stored in the memory, which when executed at the at least one processor causes the system to: In accordance with an aspect, there is provided a computer-implemented system for training an automated agent. The system includes a communication interface, at least one processor, memory in communication with the at least one processor, and software code stored in the memory. The software code, when executed at the at least one processor causes the system to: instantiate an automated agent that maintains a reinforcement learning neural network and generates, according to outputs of the reinforcement learning neural network, signals for communicating resource task requests” (Burhani, [0004]).
provid[ing], to the reinforcement learning neural network, state data reflective of a current state of an environment in which resource task requests are made: “Reinforcement learning is a category of machine learning that configures agents, such the automated agents 180 described herein, to take actions in an environment to maximize a notion of a reward. The processor 104 is configured with machine executable instructions to instantiate an automated agent 180 that maintains a reinforcement learning neural network 110 … Reward system 126 is configured to receive control the reinforcement learning network 110 to process input data (state data) in order to generate output signals. Input data may include trade orders, various feedback data (e.g., rewards), or feature selection data, or data reflective of completed tasks ( e.g., executed trades), data reflective of trading schedules, etc. Output signals may include signals for communicating resource task requests, e.g., a request to trade in a certain security” (Burhani, [0052]). Resource task requests are made in the agent environments.
generat[ing] a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads: “The software code, when executed at the at least one processor causes the system to: instantiate an automated agent that maintains a reinforcement learning neural network and generates, according to outputs (task request parameters) of the reinforcement learning neural network, signals for communicating resource task requests” (Burhani, [0004]).
While Burhani fails to disclose the further limitations of claim 1, Van Seijen teaches a method, comprising:
maintain[ing] a reinforcement learning neural network having an output layer with a plurality of policy heads, each of the policy heads configured to be trained based on a separate reward:
“Let the reward function of the environment be
R
e
n
v
. The target function of the deep network (neural network) can be regularized by splitting the reward function into n reward functions, weighted by
w
i
:
PNG
media_image1.png
104
550
media_image1.png
Greyscale
and training a separate reinforcement-learning agent on each of these reward functions” (Van Seijen, [0239]).
“Different agents can share multiple lower-level layers of a deep Q-network (reinforcement learning neural network), the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (output) of the current state under a different Q,. A single vector 8 can be used for the parameters of this network. Each head can be associated with a different reward function” (Van Seijen, [0243]).
“Options are temporally-extended actions that, like HRA's heads, can be trained in parallel based on their own (intrinsic) reward functions (separate reward[s])” (Van Seijen, [0251])
provid[ing], to the plurality of policy heads of the reinforcement learning neural network, a corresponding plurality of rewards corresponding to at least one prior resource task request generated based on outputs of the reinforcement learning neural network:
“Actions a are taken at discrete time steps according to policy
π
, which maps states to actions. For example, actions a may be taken at discrete time steps t=0, 1, 2, ... according to a policy
π
” (Van Seijen, [0074]). At each time step, a task request is made to select an action.
“In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (signals from resource task requests))” (Van Seijen, [0064]).
“In an example HRA model, consider a Markov decision process (MDP) that models an agent interacting with an environment at discrete time steps t. It has a state set S, an action set A, transition probability function
P
:
S
×
A
×
S
→
[
0,1
]
and environment reward function
R
e
n
v
:
S
×
A
→
R
. At time step t, the agent observes states
s
t
∈
S
, and takes action
a
t
∈
A
(signal from prior task request). The agent observes the next state
s
t
+
1
, drawn from the transition probability function P, and a reward
r
t
=
R
e
n
v
(
s
t
,
a
t
)
.” (Van Seijen, [0235]). The value of the reward function is based on the previous action performed in response to a prior task request.
“Each head can be associated with a different reward function” (Van Seijen, [0243]).
obtain[ing] a plurality of task request parameters relating to executable resource task request portions of a resource task request associated with a resource, each task request parameter selected by a corresponding policy head of the plurality of policy heads provided with the corresponding plurality of rewards
“Different agents can share multiple lower-level layers of a deep Q-network, the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (task request parameters) of the current state under a different Qi” (Van Seijen, [0243]).
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent can give its values for the actions (task request parameters) of the current state to an aggregator” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (executable resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]). Collecting each individual fruits comprises an executable portion of the fruit collection task.
… the plurality of task request parameters including
a first task request parameter defining a number of the executable resource task request portions of the resource task request “each piece of fruit 102 may be assigned to a specific agent (policy head) whose only learning objective is to estimate the optimal action-value function for reaching that piece of fruit 102 (resource task request portion). This agent sees a reward of+ 1 only if its assigned fruit 102 is reached and otherwise sees no reward. The state-space for this agent can ignore all other fruit 102 because they are irrelevant for its value function. An aggregator can then make the final action selection from among the agents of each piece of fruit 102” (Van Seijen, [0068]). The resource task of collecting a plurality of fruits is broken down into several portions, one for each fruit, each with a corresponding policy head. Each head selects actions (task request parameters) only relevant to their own task portion. Thus, each head produces a task request parameter that defines one of the executable resource task request portions.
and a second task request parameter defining a timing for executing the executable resource task request portions: “At time t, each agent (policy head) i, observes state
Y
t
… At each time t, each agent i can also select environment action
B
t
i
, and communication action
c
t
i
according to policy
π
i
. Action (task request parameter)
a
t
is fed to the environment, which responds with an updated state
x
t
+
1
” (Van Seijen, [0076])
generat[ing] a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent (policy head) can give its values for the actions (task request parameters) of the current state to an aggregator. ” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]).
Burhani and Van Seijen relate to deep reinforcement learning and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Burhani to generate task request signals from a plurality of policy heads with unique outputs, as disclosed by Van Seijen. Van Seijen’s multi-head framework allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
Regarding claim 3, the rejection of claim 1 is incorporated. Van Seijen further teaches a method, wherein each of the plurality of rewards is associated with a corresponding subgoal of the resource task requests:
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (resource task / goal) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0070]). The agent’s goal is to reach each piece of fruit.
“In the experiments, the performance of DQN was compared with HRA. The learning objective for DQN gave a+1 reward for each piece of fruit and used y=0.95. For HRA, the reward function was decomposed into ten different reward functions: one per possible fruit locations (sub-goal[s])” (Van Seijen, [0266]). Each of the plurality of rewards is associated with finding one particular fruit, a subgoal.
Van Seijen relates to multi-head deep reinforcement learning and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Burhani and Van Seijen to apply a reward to each of the plurality of policy heads, as disclosed by Van Seijen. Van Seijen’s multi-head framework uses a decomposed reward function, where each subfunction learns only a subset of the features. This allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
Regarding claim 4, the rejection of claim 3 is incorporated. Van Seijen further teaches a method, wherein the providing the plurality of rewards includes providing to each of the plurality of policy heads a subset of the plurality of rewards selected for that policy head: “Different agents can share multiple lower-level layers of a deep Q-network, the collection of agents can be viewed alternatively as a single agent with multiple heads (plurality of policy heads), with each head producing the action-values of the current state under a different Qi. A single vector 8 can be used for the parameters of this network. Each head can be associated with a different reward function” (Van Seijen, [0243]).
Van Seijen relates to multi-head deep reinforcement learning and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Burhani and Van Seijen to apply a reward to each of the plurality of policy heads, as disclosed by Van Seijen. Van Seijen’s multi-head framework uses a decomposed reward function, where each subfunction learns only a subset of the features. This allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
Regarding claim 5, the rejection of claim 1 is incorporated. Burhani further teaches a method, wherein the reinforcement learning neural network is maintained in an automated agent: “The software code, when executed at the at least one processor causes the system to: instantiate an automated agent that maintains a reinforcement learning neural network” (Burhani, [0004]).
Regarding claim 6, the rejection of claim 5 is incorporated. Burhani further teaches a method, wherein the outputs include at least one output defining an action to be taken by the automated agent: “The automated agent 180 generates, according to outputs of its reinforcement learning neural network, signals for communicating resource task requests (action[s] to be taken by the automated agent) for a given resource (e.g., a given security). For example, the automated agent 180 may receive a trade order for a given security as input data and then generate signals for a plurality of resource task requests corresponding to trades for child trade order slices of that security. Such signals may be communicated to a trading venue by way of communication interface 106” (Burhani, [0090]).
Regarding claim 7, the rejection of claim 6 is incorporated. Burhani further teaches a method, wherein the outputs include at least one output defining a parameter of the action: “Output signals may include signals for communicating resource task requests, e.g., a request to trade in a certain security“ (Burhani, [0052]). A request to trade a security must contain some sort of parameter for it to be a proper request, such as (yes/no).
Regarding claim 8, the rejection of claim 1 is incorporated. Van Seijen further teaches a method, wherein the generating includes combining at least two of the outputs: “A strategy for constructing a learning objective can be to decompose the reward function of the environment into n different reward functions. Each reward function can be assigned to a separate reinforcement learning agent. These agents can learn in parallel on the same sample sequence by using off-policy learning (e.g., using a Horde architecture). An aggregator can generate or select an action to take (task request) with respect to the environment. This can be referred to as an environment action and can define a set of all possible actions that can be taken with respect to the environment. Each agent can give its values for the actions of the current state to an aggregator. In an example, the aggregator can select one of the received actions as the environment action. For example, the aggregator can combine two more received action values (outputs) into a single action value for each action (for example, by averaging over all agents)” (Van Seijen, [0234]); “the aggregator uses the values of all heads to select its action” (Van Seijen, [0251]).
Van Seijen relates to multi-head deep reinforcement learning and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Burhani and Van Seijen to combine the outputs of policy heads, as disclosed by Van Seijen. Van Seijen’s multi-head framework uses a decomposed reward function, where each head subfunction learns only a subset of the features. This allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
Regarding claim 9, the rejection of claim 8 is incorporated. Van Seijen further discloses a method, wherein the output layer is interconnected with a plurality of hidden layers of the reinforcement learning neural network: “The HRA neural network 2420 includes an input layer 2422, one or more hidden layers 2424, and a plurality of heads 2426, each with their own reward function … The heads 2426 inform the output 2428 (e.g., using a linear combination) (output layer)” (Van Seijen, [0262]).
Van Seijen relates to multi-head deep reinforcement learning and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Burhani and Van Seijen to use a neural network with hidden layers, as disclosed by Van Seijen. Hidden layers are a component of the HRA network described by Van Seijen. HRA has numerous benefits over comparable methods, including greater exploitation of available domain knowledge, advantage estimating random policy, outperforming other methods and human performance, resisting getting stuck in local optima, and learning quickly. See Van Seijen, [0252], [0256], [0268], [0269], [0293], [0298], and [0299].
Regarding claim 10, the rejection of claim 1 is incorporated. Burhani further teaches a method, wherein the resource task request signal encodes a request to trade a security: “Output signals may include signals for communicating resource task requests, e.g., a request to trade in a certain security” (Burhani, [0052]).
Regarding claim 12, the rejection of claim 1 is incorporated. Burhani further teaches a method, wherein the environment includes a trading venue: “As depicted, at each time step (t0 , t1 , . .. tn), platform 100 receives task data 300, e.g., directly from a trading venue or indirectly by way of an intermediary. Task data 300 includes data relating to tasks completed in a given time interval” (Burhani, [0069]).
Regarding claim 13, Burhani teaches [a] computer computer-implemented method for automatically generating resource task requests, the method comprising:
providing, to the reinforcement learning neural network, state data reflective of a current state of an environment in which resource task requests are made: “Reinforcement learning is a category of machine learning that configures agents, such the automated agents 180 described herein, to take actions in an environment to maximize a notion of a reward. The processor 104 is configured with machine executable instructions to instantiate an automated agent 180 that maintains a reinforcement learning neural network 110 … Reward system 126 is configured to receive control the reinforcement learning network 110 to process input data (state data) in order to generate output signals. Input data may include trade orders, various feedback data (e.g., rewards), or feature selection data, or data reflective of completed tasks ( e.g., executed trades), data reflective of trading schedules, etc. Output signals may include signals for communicating resource task requests, e.g., a request to trade in a certain security” (Burhani, [0052]). Resource task requests are made in the agent environments.
generating a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads: “The software code, when executed at the at least one processor causes the system to: instantiate an automated agent that maintains a reinforcement learning neural network and generates, according to outputs (task request parameters) of the reinforcement learning neural network, signals for communicating resource task requests” (Burhani, [0004]).
While Burhani fails to disclose the further limitations of claim 1, Van Seijen teaches a method, comprising:
maintaining a reinforcement learning neural network having an output layer with a plurality of policy heads, each of the policy heads configured to be trained based on a separate reward:
“Let the reward function of the environment be
R
e
n
v
. The target function of the deep network (neural network) can be regularized by splitting the reward function into n reward functions, weighted by
w
i
:
PNG
media_image1.png
104
550
media_image1.png
Greyscale
and training a separate reinforcement-learning agent on each of these reward functions” (Van Seijen, [0239]).
“Different agents can share multiple lower-level layers of a deep Q-network (reinforcement learning neural network), the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (output) of the current state under a different Q,. A single vector 8 can be used for the parameters of this network. Each head can be associated with a different reward function” (Van Seijen, [0243]).
“Options are temporally-extended actions that, like HRA's heads, can be trained in parallel based on their own (intrinsic) reward functions (separate reward[s])” (Van Seijen, [0251])
providing, to the plurality of policy heads of the reinforcement learning neural network, a corresponding plurality of rewards corresponding to at least one prior resource task request generated based on outputs of the reinforcement learning neural network:
“Actions a are taken at discrete time steps according to policy
π
, which maps states to actions. For example, actions a may be taken at discrete time steps t=0, 1, 2, ... according to a policy
π
” (Van Seijen, [0074]). At each time step, a task request is made to select an action.
“In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (signals from resource task requests))” (Van Seijen, [0064]).
“In an example HRA model, consider a Markov decision process (MDP) that models an agent interacting with an environment at discrete time steps t. It has a state set S, an action set A, transition probability function
P
:
S
×
A
×
S
→
[
0,1
]
and environment reward function
R
e
n
v
:
S
×
A
→
R
. At time step t, the agent observes states
s
t
∈
S
, and takes action
a
t
∈
A
(signal from prior task request). The agent observes the next state
s
t
+
1
, drawn from the transition probability function P, and a reward
r
t
=
R
e
n
v
(
s
t
,
a
t
)
.” (Van Seijen, [0235]). The value of the reward function is based on the previous action performed in response to a prior task request.
“Each head can be associated with a different reward function” (Van Seijen, [0243]).
obtaining a plurality of task request parameters relating to executable resource task request portions of a resource task request associated with a resource, each task request parameter selected by a corresponding policy head of the plurality of policy heads provided with the corresponding plurality of rewards
“Different agents can share multiple lower-level layers of a deep Q-network, the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (task request parameters) of the current state under a different Qi” (Van Seijen, [0243]).
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent can give its values for the actions (task request parameters) of the current state to an aggregator” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (executable resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]). Collecting each individual fruits comprises an executable portion of the fruit collection task.
… the plurality of task request parameters including
a first task request parameter defining a number of the executable resource task request portions of the resource task request “each piece of fruit 102 may be assigned to a specific agent (policy head) whose only learning objective is to estimate the optimal action-value function for reaching that piece of fruit 102 (resource task request portion). This agent sees a reward of+ 1 only if its assigned fruit 102 is reached and otherwise sees no reward. The state-space for this agent can ignore all other fruit 102 because they are irrelevant for its value function. An aggregator can then make the final action selection from among the agents of each piece of fruit 102” (Van Seijen, [0068]). The resource task of collecting a plurality of fruits is broken down into several portions, one for each fruit, each with a corresponding policy head. Each head selects actions (task request parameters) only relevant to their own task portion. Thus, each head produces a task request parameter that defines one of the executable resource task request portions.
and a second task request parameter defining a timing for executing the executable resource task request portions: “At time t, each agent (policy head) i, observes state
Y
t
… At each time t, each agent i can also select environment action
B
t
i
, and communication action
c
t
i
according to policy
π
i
. Action (task request parameter)
a
t
is fed to the environment, which responds with an updated state
x
t
+
1
” (Van Seijen, [0076])
generat[ing] a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent (policy head) can give its values for the actions (task request parameters) of the current state to an aggregator. ” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]).
Burhani and Van Seijen relate to deep reinforcement learning and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Burhani to generate task request signals from a plurality of policy heads with unique outputs, as disclosed by Van Seijen. Van Seijen’s multi-head framework allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
All limitations of claims 15-16 are disclosed by claims 3-4. Thus, the analysis of claims 15-16 mirrors that of claims 3-4 and claims 15-16 are rejected under the same rationale used for claims 3-4.
Regarding claim 17, Burhani teaches [a] non-transitory computer-readable storage medium storing instructions which when executed adapt at least one computing device to: “Throughout the foregoing discussion, numerous references will be made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a server can include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions” (Burhani, [0193]).
Burhani’s instructions to adapt at least one computing device to:
provide state data reflective of a current state of an environment in which resource task requests are made to the reinforcement learning neural network: “Reinforcement learning is a category of machine learning that configures agents, such the automated agents 180 described herein, to take actions in an environment to maximize a notion of a reward. The processor 104 is configured with machine executable instructions to instantiate an automated agent 180 that maintains a reinforcement learning neural network 110 … Reward system 126 is configured to receive control the reinforcement learning network 110 to process input data (state data) in order to generate output signals. Input data may include trade orders, various feedback data (e.g., rewards), or feature selection data, or data reflective of completed tasks ( e.g., executed trades), data reflective of trading schedules, etc. Output signals may include signals for communicating resource task requests, e.g., a request to trade in a certain security” (Burhani, [0052]). Resource task requests are made in the agent environments.
generate a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads: “The software code, when executed at the at least one processor causes the system to: instantiate an automated agent that maintains a reinforcement learning neural network and generates, according to outputs (task request parameters) of the reinforcement learning neural network, signals for communicating resource task requests” (Burhani, [0004]).
While Burhani fails to disclose the further limitations of claim 1, Van Seijen teaches instructions to:
maintain a reinforcement learning neural network having an output layer with a plurality of policy heads, each of the policy heads configured to be trained based on a separate reward:
“Let the reward function of the environment be
R
e
n
v
. The target function of the deep network (neural network) can be regularized by splitting the reward function into n reward functions, weighted by
w
i
:
PNG
media_image1.png
104
550
media_image1.png
Greyscale
and training a separate reinforcement-learning agent on each of these reward functions” (Van Seijen, [0239]).
“Different agents can share multiple lower-level layers of a deep Q-network (reinforcement learning neural network), the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (output) of the current state under a different Q,. A single vector 8 can be used for the parameters of this network. Each head can be associated with a different reward function” (Van Seijen, [0243]).
“Options are temporally-extended actions that, like HRA's heads, can be trained in parallel based on their own (intrinsic) reward functions (separate reward[s])” (Van Seijen, [0251])
provide to the plurality of policy heads of the reinforcement learning neural network, a corresponding plurality of rewards corresponding to at least one prior resource task request generated based on outputs of the reinforcement learning neural network:
“Actions a are taken at discrete time steps according to policy
π
, which maps states to actions. For example, actions a may be taken at discrete time steps t=0, 1, 2, ... according to a policy
π
” (Van Seijen, [0074]). At each time step, a task request is made to select an action.
“In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (signals from resource task requests))” (Van Seijen, [0064]).
“In an example HRA model, consider a Markov decision process (MDP) that models an agent interacting with an environment at discrete time steps t. It has a state set S, an action set A, transition probability function
P
:
S
×
A
×
S
→
[
0,1
]
and environment reward function
R
e
n
v
:
S
×
A
→
R
. At time step t, the agent observes states
s
t
∈
S
, and takes action
a
t
∈
A
(signal from prior task request). The agent observes the next state
s
t
+
1
, drawn from the transition probability function P, and a reward
r
t
=
R
e
n
v
(
s
t
,
a
t
)
.” (Van Seijen, [0235]). The value of the reward function is based on the previous action performed in response to a prior task request.
“Each head can be associated with a different reward function” (Van Seijen, [0243]).
obtain a plurality of task request parameters relating to executable resource task request portions of a resource task request associated with a resource, each task request parameter selected by a corresponding policy head of the plurality of policy heads provided with the corresponding plurality of rewards
“Different agents can share multiple lower-level layers of a deep Q-network, the collection of agents can be viewed alternatively as a single agent with multiple heads (policy heads), with each head producing the action-values (task request parameters) of the current state under a different Qi” (Van Seijen, [0243]).
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent can give its values for the actions (task request parameters) of the current state to an aggregator” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (executable resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]). Collecting each individual fruits comprises an executable portion of the fruit collection task.
… the plurality of task request parameters including
a first task request parameter defining a number of the executable resource task request portions of the resource task request “each piece of fruit 102 may be assigned to a specific agent (policy head) whose only learning objective is to estimate the optimal action-value function for reaching that piece of fruit 102 (resource task request portion). This agent sees a reward of+ 1 only if its assigned fruit 102 is reached and otherwise sees no reward. The state-space for this agent can ignore all other fruit 102 because they are irrelevant for its value function. An aggregator can then make the final action selection from among the agents of each piece of fruit 102” (Van Seijen, [0068]). The resource task of collecting a plurality of fruits is broken down into several portions, one for each fruit, each with a corresponding policy head. Each head selects actions (task request parameters) only relevant to their own task portion. Thus, each head produces a task request parameter that defines one of the executable resource task request portions.
and a second task request parameter defining a timing for executing the executable resource task request portions: “At time t, each agent (policy head) i, observes state
Y
t
… At each time t, each agent i can also select environment action
B
t
i
, and communication action
c
t
i
according to policy
π
i
. Action (task request parameter)
a
t
is fed to the environment, which responds with an updated state
x
t
+
1
” (Van Seijen, [0076])
generate a resource task request signal defining the resource task request and the associated plurality of task request parameters selected by the plurality of policy heads
“An aggregator can generate or select an action (task request signal) to take with respect to the environment. This can be referred to as an environment action (task request signal) and can define a set of all possible actions that can be taken with respect to the environment. Each agent (policy head) can give its values for the actions (task request parameters) of the current state to an aggregator. ” (Van Seijen, [0234]).
“FIG. 1 illustrates an example layout 100 for this introductory example, including three pieces of fruit 102 (resource[s]) and the robot 104 with arrows 106 indicating potential directions of movement within a grid of possible positions 108. In this example, the goal of the robot 104 is to reach each piece of fruit (resource task) 102 scattered across the possible positions 108 as quickly as possible (e.g., in the fewest possible actions (task request signal[s])). In reinforcement learning, an agent controlling the robot 104 aims to maximize a return, Gt, which is the expected discounted sum of rewards … The possible actions (task request parameters) of the robot 104 include moving in different directions and a "no movement" (i.e., no-op) action” (Van Seijen, [0064]).
Burhani and Van Seijen relate to deep reinforcement learning and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Burhani to generate task request signals from a plurality of policy heads with unique outputs, as disclosed by Van Seijen. Van Seijen’s multi-head framework allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010] and [0072].
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Burhani et al. (TRADE PLATFORM WITH REINFORCEMENT LEARNING, published 12/5/2019, US 2019/0370649 A1), hereafter referred to as Burhani, in view of Van Seijen et al. (SCALABILITY OF REINFORCEMENT LEARNING BY SEPARATION OF CONCERNS, filed 6/26/2017, US 2018/0165602 A1), hereafter referred to as Van Seijen, and further in view of Baldacci et al. (Market making and incentives design in the presence of a dark pool: a deep reinforcement learning approach, 2019, arXiv:1912.01129v), hereafter referred to as Baldacci.
Regarding claim 11, the rejection of claim 10 is incorporated. While Burhani and Van Seijen fail to disclose the further limitations of the claim, Baldacci discloses a method, wherein the outputs include at least one output indicating whether the request to trade a security should be made in a lit pool or a dark pool:
“To our knowledge, most of studies treat the issue of trading in dark pools mainly from the point of view of optimal liquidation: a trader wishing to buy or sell a large number of shares of one or several stocks (securit[ies]) and needing to find an optimal order placement strategy between the lit and dark pools, see for example [15]. In this paper, we rather focus on the behavior of a market maker, acting on both lit and dark venues” (Baldacci, page 2, paragraph 4).
“We now turn to the description of our numerical method to solve (4.17), the optimization procedure consists of two stages. At the first stage, we optimize the controls of the market maker for all possible values of the incentives given by the exchange. At the second stage, we use an actor-critic approach, to obtain both the optimal controls and the value function of the exchange” (Baldacci, page 15, paragraph 2)
“The first step to tackle our principal-agent problem is to find optimal volumes
L
*
” (Baldacci, page 15, paragraph 5)
“We approximate the best response function
L
*
by a neural network
l
[
ω
l
]
… The neural network
l
[
ω
l
]
takes as inputs principal’s incentives and the market maker’s current inventory” (Baldacci, page 16, paragraph 3)
“We now move to the problem of the principal” (Baldacci, page 18, paragraph 2)
“We use an algorithm known in reinforcement learning literature as the actor-critic method. The core of this approach is the representation of the value function and optimal controls with deep neural networks. The learning procedure itself consists of two stages: value function update (also called critic update) and controls update (actor update)” (Baldacci, page 18, paragraph 5)
“5.2 Numerical Results” (Baldacci, page 20, paragraph 1)
PNG
media_image2.png
663
907
media_image2.png
Greyscale
(Baldacci, page 20, Figure 3). This figure shows optimal bid / ask volumes for lit / dark pools as an output of this model.
“One can see that the market maker splits his orders equitably between the lit and dark pools when his inventory is near zero. However, when he has a very positive (resp. negative) inventory, he has a large imbalance on the ask (resp. bid) side of the lit pool, to liquidate his position in the dark pool” (Baldacci, page 20, paragraph 4)
Examiner’s note: This model determines how many trades should be conducted in a lit vs. a dark pool.
Baldacci relates to deep reinforcement learning for market trading and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the existing combination to output information indicating whether a trade should be conducted in a lit or dark pool, as disclosed by Baldacci. Dark pools have gained a significant market share over traditional lit pools, with many major exchanges having both dark pools and lit pools. Baldacci’s method efficiently approximates both the optimal controls of a market maker trying to buy and sell in both pools, and the optimal incentives of the dual-pool exchange, determining the optimal trading volume for each type of pool for the market maker. See Baldacci, Abstract; page 2, paragraph 2; page 2, paragraph 4; page 3 paragraph 2.
Regarding claim 18, the rejection of claim 3 is incorporated. Van Seijen further teaches a method, wherein the plurality of sub-goals of the resource task requests are determined based on the state data reflecting the current state of the environment: “For HRA, the reward function was decomposed into ten different reward functions: one per possible fruit locations (sub-goal[s]). The network included an input layer of length 110, encoding the agent's position (state data) and whether there is a piece of fruit on each location (state data)” (Van Seijen, [0266])
Van Seijen relates to multi-head deep reinforcement learning and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Burhani and Van Seijen to determine sub-goals based on current state data, as disclosed by Van Seijen. Van Seijen’s multi-head framework uses a decomposed reward function, where each subfunction learns only a subset of the features. By earning different rewards for traveling to empty spaces versus collecting fruits, the network can constantly learn via backpropagation. Van Seijen’s system allows for a smoother overall value function that can be approximated by a lower-dimensional representation, allowing for more effective learning. Additionally, this process reduces the overall state space, increases convergence speed, and reduces computing resources consumed. See Van Seijen, [0010], [0072], and [0264].
Response to Arguments
The following responses address arguments and remarks made in the instant remarks dated 4/30/2026.
112 Rejections
In light of the instant remarks, previous rejections under 35 U.S.C. 112(a) have been withdrawn. However, in light of the instant amendments, new rejections under 35 U.S.C. 112(a) have been determined.
103 Rejections
On pages 6-7 of the instant remarks, the Applicant argues that the references currently relied upon fail to disclose the amended claims:
“Van Seijen is directed to decomposing a single-agent learning problem into simpler problems addressed
by multiple agents and aggregating, using an aggregator, the different solutions from the multiple agents to determine a single action to take with respect to an environment (see Abstract)
The Examiner appears to analogize the output values of each agent to resource task request parameters.
Applicant disagrees with this analogy, since, as recited, the task request parameters define the number of
executable portions of the resource task request and the timing for executing the portions. Further, Van
Seijen does not describe resource task request parameters relating to executable resource task request
portions of a resource task request
At best, Van Seijen describes a single task (an action to be taken) at a particular moment in time. Further,
Applicant submits that the outputs from the different agents cannot be analogized to executable resource
task request portions, since the outputs from the different agents are of a different nature. In Van Seijen,
the outputs are combined to determine a single action for an agent and are not individually executed
according to timing information
Applicant further submits that it is not clear how Burhani or Hernandez disclose the above-noted feature.”
Regarding the Applicant’s arguments above, the Examiner respectfully disagrees.
Regarding amended claim limitation “obtain a plurality of task request parameters relating to executable resource task request portions of a resource task request associated with a resource”, Van Seijen discloses a reinforcement learning task (executable resource task) for collecting all of a set of fruits (resource[s]) (Van Seijen, [0064]). This task can be decomposed into portions of single-fruit collection subtasks that are to be executed, and corresponding actions (task request parameters) selected by policy heads to complete the (sub)tasks (Van Seijen, [0234] & [0243]). The executable resource task (collecting fruit) is executed through the execution and completion of lower-level subtasks collecting individual fruits.
Regarding “a first task request parameter defining a number of the executable resource task request portions of the resource task request”, Van Seijen discloses policy heads, wherein each head defines one portion of the fruit collection task, and produces actions (task request parameter[s]) responsive to the corresponding subtask (Van Seijen, [0068]). As mentioned above, the task request portions are executable. Additionally, actions of the single-fruit heads are used to produce final actions directly executed by the agent in the environment in the service of the aforementioned task goals.
Regarding “a second task request parameter defining a timing for executing the executable resource task request portions”, Van Seijen discloses policy heads able to produce actions (task request parameters) at each time step (Van Seijen, [0076]), thus defining the timing of actions made in service of completing (at least a portion of) a task request.
The pertinent information disclosed by Van Seijen is commensurate with the claim language. See the 103 rejections section for more detail. No rejections are withdrawn on this basis.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Li et al. (SYSTEMS AND METHODS FOR REPOSITIONING VEHICLES IN A RIDE-HAILING PLATFORM, filed 2/26/2021, US 20220277329 A1) discloses a method of temporally extending the duration of actions in a reinforcement learning system
Kartal et al. (SYSTEM AND METHOD FOR DEEP REINFORCEMENT LEARNING, published 5/7/2020, US 2020/0143206 A1) teaches a method of using a parallelized asynchronous multi-head reinforcement learning model
Laruelle et al. (Optimal split of orders across liquidity pools: a stochastic algorithm approach, 2010, arXiv:0910.1166v3) teaches a method of determining an optimal way to split trades across lit and dark pools
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights