Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16th, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In the previous Office Action issued April 29th, 2026 (hereinafter “the previous Office Action”), claims 1-3, 6-9 , 11-17, 19, and 21-22 were pending.
This action is in response to the amendment and remarks filed July 29th, 2026. In the amendment, claims 1, 9, and 21 were amended, no claims were canceled, and no claims were added. Thus, claims 1-3, 6-9 , 11-17, 19, and 21-22 are pending.
The objections to the drawings, set forth in the previous Office Action, have been withdrawn in view of Applicant’s amendments and remarks.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-3, 7-9, 11-17, 19, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Viola et al. (US 20190266449), hereinafter Viola, in view of Ilhan et al. (“Teaching on a Budget in Multi-Agent Deep Reinforcement Learning”), hereinafter Ilhan, further in view of Zheng et al. (“Self-Supervised Mixture-of-Experts by Uncertainty Estimation”), hereinafter Zheng, in view of Clements et al. (“Estimating Risk and Uncertainty in Deep Reinforcement Learning”), hereinafter Clements.
Regarding Claim 1:
Viola discloses:
A computer-implemented system for training a learning agent, the system comprising: at least one processor, memory in communication with the at least one processor, and software code stored in the memory, which when executed by the at least one processor causes the system to:
Viola, Abstract, “Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a reinforcement learning system.”
[0063], “Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.”
Viola discloses a computer system, a processor, memory, and computer instructions.
instantiate the learning agent that maintains a reinforcement learning neural network
Viola, [0003], “In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.”
Viola discloses a reinforcement learning system with an agent that interacts with an environment. Thereby, Viola is construed as disclosing instantiating the learning agent at a high level of generality.
receive state date reflective of a state of an environment explored by the learning agent
Viola, [0006], “an agent interacting with an environment… and in selecting actions to be performed in response to observations of the environment”
Viola discloses an agent interacting with an environment which corresponds to the learning agent exploring an environment and receiving state data.
Viola does not explicitly disclose:
wherein the reinforcement learning neural network comprises a plurality of hidden layers including a layer having a plurality of heads
each of the heads for generating predictions of action values for actions that can be taken by the learning agent
calculate an uncertainty metric upon processing the state data
wherein the uncertainty metric is a function of the variance between expected values for each potential action of the learning agent
upon determining that the uncertainty metric exceeds a pre-defined threshold:
send a request signal requesting an action suggestion from a demonstrator
receive a suggestion signal reflective of the action suggestion from the demonstrator
send an action signal to implement the action suggestion
However, in the same field, analogous art Ilhan teaches:
calculate an uncertainty metric upon processing the state data
Ilhan, p. 4, col. 2, “it measures its uncertainty µ…”
Ilhan, p. 4, col. 2, “Consequently, it will be able to measure how uncertain it is in a state with observation o by measuring the error
G
^
o
;
θ
-
G
(
o
)
2
when determining if it needs advice or is capable of giving it.”
Ilhan discloses measuring uncertainty µ upon processing the state data.
wherein the uncertainty metric measures epistemic uncertainty indicative of the learning agent’s lack of confidence in selecting an action
Ilhan, p. 4, col. 2, “At each timestep, each agent
i
chooses its action according to teacher-student augmented action selection procedure defined in Algorithm 1. If it has a remaining asking budget
b
a
s
k
, it measures its uncertainty µ using its internal models
G
and
G
^
; if µ is higher than a predefined asking threshold
τ
a
s
k
the agent broadcasts its advice requests to other agents as described in Algorithm 2.”
Ilhan discloses that when uncertainty µ [the uncertainty metric measures epistemic uncertainty] is higher than a predefined threshold, the agent asks for advice from other agents [indicative of the learning agent’s lack of confidence in selecting an action].
upon determining that the uncertainty metric exceeds a pre-defined threshold:
Ilhan, p. 4, col. 2, “if µ is higher than a predefined threshold”
send a request signal requesting an action suggestion from a demonstrator
Ilhan, p. 4, col. 2, “the agent broadcasts its advice requests to other agents”
receive a suggestion signal reflective of the action suggestion from the demonstrator
Ilhan, p. 4, col. 2, “If agent receives any advice…”
Receiving advice corresponds to receiving a suggestion signal reflective of the action suggestion.
send an action signal to implement the action suggestion
Ilhan, p. 4, col. 2, “If the agent receives any advice, it determines which one to follow… then executes such action.”
Executing the action corresponds to implementing the action suggestion.
Viola, Ilhan, and the instant application are analogous art because they are all directed to reinforcement learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola with Ilhan by providing an uncertainty metric, allowing a deep reinforcement learning agent to receive advice at appropriate times (Ilhan, p. 2, col. 1). Doing so effectively reduces time constraints on training and overall sample-complexity required (Ilhan, p. 1, col. 1) as it address the practical resource concerns of constant attention and communication cost between agent and demonstrators (Ilhan, p. 2, col. 2).
Viola in view of Ilhan do not explicitly disclose:
wherein the reinforcement learning neural network comprises a plurality of hidden layers and at least one layer having a plurality of heads, the plurality of heads branching from the plurality of hidden layers, each of the plurality of hidden layers being shared by the plurality of heads
each of the heads for separately generating predictions of action values for actions that can be taken by the learning agent
wherein the uncertainty metric is a function of the variance between expected values for each of the potential actions from the plurality of heads of the learning agent
However, in the same field, analogous art Zheng teaches:
wherein the reinforcement learning neural network comprises a plurality of hidden layers and at least one layer having a plurality of heads, the plurality of heads branching from the plurality of hidden layers, each of the plurality of hidden layers being shared by the plurality of heads
PNG
media_image1.png
337
1053
media_image1.png
Greyscale
Zheng, p. 5935, “In this work, we develop Self-Supervised Mixture-of Experts (SUM) for multi-task RL (see Figure 1).”
p. 5937, “In all cases, we use fully-connected network (see Figure1), where hidden layer and head layer sizes are denoted by (N, M).”
On p. 5935 and in view of FIG. 1 above, Zheng discloses a SUM for multi-task reinforcement learning [reinforcement learning neural network]. Further on p. 5937, Zheng discloses their network is fully-connected, where they have hidden layer and head layers of sizes N and M [comprises a plurality of hidden layers and at least one layer having a plurality of heads]. Lastly, FIG. 1 depicts the multi-head actor to be fully connected to the hidden layer, with each head being connected to the hidden layer [the plurality of heads branching from the plurality of hidden layers, each of the plurality of hidden layers being shared by the plurality of heads].
each of the heads for separately generating predictions of action values for actions that can be taken by the learning agent
Zheng, p. 5936, “Note that each actor head in multi-head DDPG is considered as an expert. Following that, each actor head produces an action candidate
a
t
k
while at the same time its paired critic head produces respective
Q
-value
Q
k
and
Q
-variance
σ
k
2
, which represents how uncertain the expert is about its evaluation of the action
a
t
k
”
Zheng discloses each actor head in the multi-head produces an action candidate [each of the heads for separately generating predictions of action values for actions that can be taken by the learning agent].
Viola, Ilhan, Zheng, and the instant application are analogous art because they are all directed to reinforcement learning.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola and Ilhan with Zheng in order to achieve robust generalization. “Well-calibrated predictive uncertainty estimation assists the agent in robustly averting over-confident incorrect predictions. Utilizing the multi-head architecture simulates Deep Ensembles technique (Lakshminarayanan, Pritzel, and Blundell 2017), averaging predictions over multiple models, which significantly improves uncertainty quality and robustness by variance reduction. In this case, critics can produce more accurate Q-value estimates by ‘ensemble’ improving the generalization ability against overfitting. Furthermore, when applying to unknown tasks, highly specialized experts with relatively lower uncertainty shows reliably higher confidence to tackle specific states, which benefits effective expert scheduling enhancing the overall generalization capacity across multiple tasks” (Zheng, p. 5935-5936).
Viola in view of Ilhan, in view of Zheng do not explicitly disclose:
wherein the uncertainty metric is a function of the variance between expected values for each of the potential actions from the plurality of heads of the learning agent
However, in the same field, analogous art Clements teaches:
wherein the uncertainty metric is a function of the variance between expected values for each of the potential actions from the plurality of heads of the learning agent
Clements, p. 4, “Specifically…the variance of the Bayesian posterior distribution can be used to estimate the epistemic uncertainty.”
p. 14, “The mean of the distributions for each action give the expectation value corresponding to performing that action, and the standard deviation between the two network’s predictions averaged over the quantiles yields the uncertainty.”
On p. 4, Clements teaches that the epistemic uncertainty is estimated by the variance of the Bayesian posterior distribution [the uncertainty metric is a function of the variance between expected values for each of the potential actions from the plurality of heads of the learning agent]. Further on p. 14, Clements further discloses the uncertainty is obtained using the expected value corresponding to the performing the action [between expected values for each of the potential actions] and the networks predictions [from the plurality of heads of the learning agent].
Viola, Ilhan, Zheng, Clements, and the instant application are analogous art because they are all directed to reinforcement learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, and Zheng with Clements in order to build a neural network containing more than a single hidden layer and more than a single output. Doing so enables a neural network using epistemic uncertainty to more efficiently explore an environment with poorly known states (Clements, p. 1, para. 3), allowing them to better generalize to a wide variety of initial conditions different from what they were trained on (Clements, p. 7, para. 4).
Regarding Claim 2:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented system of claim 1, and Ilhan further discloses:
wherein the demonstrator comprises an automated agent
Ilhan, p. 1, col. 2, “the learning agent is assisted by an expert agent…it is possible for one of the peers to be human…”
Ilhan discloses that the expert agent [demonstrator] can be a human and therefore is construed as disclosing the expert agent is an automated expert agent by default.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to use an automated agent in order optimize communication. “According to this method, an expert teacher constantly monitors a student agent’s learning process and provides it with action advices at appropriate times, limited with a budget, considering the practical concerns regarding attention and communication” (Ilhan, p. 2, col. 2). An automated agent is able to constantly monitor the learning agent, whereas a human would not be able to constantly monitor the learning agent. Therefore, the learning of the agent is optimized as it is able to effectively learn.
Regarding Claim 3:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented system of claim 2, and Ilhan further discloses:
wherein the automated agent has a policy that differs from a policy of the learning agent
Ilhan, p. 4, col. 2, “every agent is learning simultaneously and no longer have fixed policies. Moreover, they now have different levels of knowledge about the task.”
Every automated agent has a policy that differs from each other including the learning agent.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to use an automated agent with a policy that differs from the policy of the learning agent in order to increase robustness of the learning. “Moreover, they now have different levels of knowledge about the task, and have no information on each other’s expertise since they can not have any access to internal information and they hold on assumptions about roles of being teacher or student” (Ilhan, p. 4, col. 2). Since the agents do not have access to each other’s internal information, this increases the system’s robustness as it is able to preserve data privacy.
Regarding Claim 7:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented system of claim 1, and Viola further discloses:
further comprising a network communication interface for transmitting signals through a network, and the request signal is sent by way of the network communication interface
Viola, [0068], “the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.”
Viola discloses a communication network [network communication interface] for digital data communication [for transmitting signals through a network]. Therefore, Viola is construed as disclosing that the request signal is sent by way of the network.
Regarding Claim 8:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented system of claim 7, and Viola further discloses:
wherein the action signal is sent by way of the network communication interface
Viola, [0068], “the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.”
Viola disclose a network communication and therefore is construed as disclosing that the action signal is sent by way of the network communication.
Regarding Claim 9:
Claim 9 is a computer-implemented method claim corresponding to the computer-implemented system of claim 1 and is rejected for at least the same reasons as given in the rejection of claim 1.
Regarding Claim 11:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 9, and Clements further discloses:
wherein the calculating the uncertainty metric comprises: receiving, from each of the plurality of heads, a predicted action value
Clements, p. 15, “The outputs from both heads are averaged to produced the value estimates used by the policy.”
Clements’ neural network outputs from each head are used to produce value estimates corresponding to prediction action values.
computing a variance of the predicted action values received from the plurality of heads
Clements, p. 4, “the variance of the Bayesian posterior distribution can be used to estimate the epistemic uncertainty.”
Clements teaches computing the variance values from the distribution can be used to estimate epistemic uncertainty.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Clements to include the variance in computing the uncertainty metric in order to increase the robustness. Doing so enables a neural network using epistemic uncertainty to more efficiently explore an environment with poorly known states (Clements, p. 1, para. 3), allowing them to better generalize to a wide variety of initial conditions different from what they were trained on (Clements, p. 7, para. 4). Additionally, by computing the variance, the system will know how the data points differ and thereby know the uncertainty.
Regarding Claim 12:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 9, and Clements further discloses:
wherein each of the plurality of heads minimizes a loss function associated with that head
Clements, p. 3, para. 5, “This loss can be minimized stochastically for each new value…”
Clements discloses using minimizing a loss and therefore is construed as disclosing minimizing a loss function for each head in the neural network.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Clements to minimize loss function in order to achieve greater accuracy. By minimizing the loss function of the respective head of the neural network, more accurate results are achieved.
Regarding Claim 13:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 9, and Ilhan further discloses:
further comprising determining whether the demonstrator is available
Ilhan, p. 4, col. 2, “Agents who have any remaining giving budget bgive attempt to respond to this request.”
Agents that have remaining giving budget are considered available. Conversely, agents that have depleted their advice-giving budget are considered unavailable.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to provide a budget that determines whether or not a demonstrator is available to give advice (Ilhan, p. 4, col. 2). Doing so ensures that instead of waiting indeterminately for a demonstrator, the reinforcement learning agent can execute based on the received advice or continue following its own policy (Ilhan, p. 4, col. 2).
Regarding Claim 14:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 13, and Ilhan further discloses:
further comprising maintaining an advice budget for the demonstrator and the determining comprises determining whether the advice budget is depleted
Ilhan, p. 3, col. 1, “teacher agents also have a form of budget bgive which is the number of advices they can provide to limit the amount they can exchange…”
p. 4, col. 2, “Upon getting a request, the agent first checks if it has any remaining giving budget bgive.”
Teacher agents [demonstrators] maintain an advice budget, in addition to checking if the budget is depleted before giving advice.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to provide a budget that determines whether or not a demonstrator is available to give advice (Ilhan, p. 4, col. 2). Doing so ensures that instead of waiting indeterminately for a demonstrator, the reinforcement learning agent can execute based on the received advice or continue following its own policy (Ilhan, p. 4, col. 2).
Regarding Claim 15:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 9, and Ilhan further discloses:
further comprising selecting the demonstrator from a plurality of demonstrators
Ilhan, p. 2, col. 1, “getting advice from multiple teachers by combining the advices using a voting based selection.”
Ilhan discloses multiple demonstrators, and they are selected a by voting based selection.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to provide a method for selecting a demonstrator amongst a plurality of demonstrators. Doing so is important because there are distinguishing qualities between an expert and a good teacher, the best performing demonstrators are not always the best teachers (Ilhan, p. 2, col. 1).
Regarding Claim 16:
Claim 16 is a computer-implemented method claim corresponding to the computer-implemented system of claim 2 and is rejected for at least the same reasons as given in the rejection of claim 2.
Regarding Claim 17:
Claim 17 is a computer-implemented method claim corresponding to the computer-implemented system of claim 3 and is rejected for at least the same reasons as given in the rejection of claim 3.
Regarding Claim 19:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer-implemented method of claim 9, and Ilhan further discloses:
further comprising updating a policy of the learning agent based on the action suggestion
Ilhan, p. 5, Algorithm 2: If agent receives any advice, it determines which one to follow, updates its own policy, and executes the action. If no advice is received, it continues following its own policy.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan to provide a method for updating the learning agent’s policy based on the action suggestion. Doing so drives the learning agents continued exploration of its environment (Ilhan, p. 1, col. 2).
Regarding Claim 21:
Claim 21 is a computer-implemented method claim corresponding to the computer-implemented system of claim 1 and is rejected for the same reasons as given in the rejection of claim 1, with the exception of the additional following limitations.
Ilhan further discloses:
A computer-implemented method for determining epistemic uncertainty of a learning agent, the method comprising:
Ilhan, p. 4, col. 2, “We propose to use RND [Random Network Distillation] technique as a metric to measure agents’ uncertainties as an alternative for nonlinear function approximation.”
p. 2, col. 2, “Random Network Distillation (RND) [20] is a technique built to provide an intrinsic curiosity reward for RL agents to enhance their exploration capabilities in complex environments.”
Ilhan discloses measuring an agent’s uncertainty [determining epistemic uncertainty of a learning agent]. The agents are a part of reinforcement learning which can only be performed by a computer, therefore Viola is construed as disclosing a computer [computer-implemented method].
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements further with Ilhan by providing an uncertainty metric, allowing a deep reinforcement learning agent to receive advice at appropriate times (Ilhan, p. 2, col. 1). Doing so effectively reduces time constraints on training and overall sample-complexity required (Ilhan, p. 1, col. 1) as it address the practical resource concerns of constant attention and communication cost between agent and demonstrators (Ilhan, p. 2, col. 2).
Zheng further discloses:
maintaining, by the learning agent, a neural network comprising a plurality of hidden layers and at least one last layer having a plurality of heads, the plurality of heads branching from the plurality of hidden layers, each of the plurality of hidden layers being shared by the plurality of heads
PNG
media_image1.png
337
1053
media_image1.png
Greyscale
Zheng, p. 5935, “In this work, we develop Self-Supervised Mixture-of Experts (SUM) for multi-task RL (see Figure 1).”
p. 5937, “In all cases, we use fully-connected network (see Figure1), where hidden layer and head layer sizes are denoted by (N, M).”
On p. 5935 and in view of FIG. 1 above, Zheng discloses an agent SUM for multi-task reinforcement learning [maintaining, by the learning agent, a neural network]. Further on p. 5937, Zheng discloses their network is fully-connected, where they have hidden layer and head layers of sizes N and M [comprising a plurality of hidden layers and at least one layer having a plurality of heads]. Lastly, FIG. 1 in view of p. 5937 depicts the multi-head actor to be fully connected to the hidden layer, with each head being connected to the hidden layer [the plurality of heads branching from the plurality of hidden layers, each of the plurality of hidden layers being shared by the plurality of heads].
each of the heads separately generating predictions of action values for actions that can be taken by the learning agent
Zheng, p. 5936, “Note that each actor head in multi-head DDPG is considered as an expert. Following that, each actor head produces an action candidate
a
t
k
while at the same time its paired critic head produces respective
Q
-value
Q
k
and
Q
-variance
σ
k
2
, which represents how uncertain the expert is about its evaluation of the action
a
t
k
”
Zheng discloses each actor head in the multi-head produces an action candidate [each of the heads for separately generating predictions of action values for actions that can be taken by the learning agent].
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola and Ilhan with Zheng in order to achieve robust generalization. “Well-calibrated predictive uncertainty estimation assists the agent in robustly averting over-confident incorrect predictions. Utilizing the multi-head architecture simulates Deep Ensembles technique (Lakshminarayanan, Pritzel, and Blundell 2017), averaging predictions over multiple models, which significantly improves uncertainty quality and robustness by variance reduction. In this case, critics can produce more accurate Q-value estimates by ‘ensemble’ improving the generalization ability against overfitting. Furthermore, when applying to unknown tasks, highly specialized experts with relatively lower uncertainty shows reliably higher confidence to tackle specific states, which benefits effective expert scheduling enhancing the overall generalization capacity across multiple tasks” (Zheng, p. 5935-5936).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Viola in view of Ilhan, in view of Zheng, in view of Clements as applied to claim 1 above, and further in view of Jibowu et al. (US 20180336636), hereinafter Jibowu.
Regarding Claim 6:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] computer implemented system of claim 1, but do not explicitly disclose:
wherein the environment is an electronic trading platform
However, in the same field, analogous art Jibowu teaches:
wherein the environment is an electronic trading platform
Jibowu, [0008], “The system may execute the instructions to receive environmental data representative of an environment comprising a plurality of financial states…”
Jibowu discloses that their reinforcement learning agent explores a complex domain such as an electronic trading platform environment including a financial state.
Viola, Ilhan, Zheng, Clements, Jibowu and the instant application are analogous art because they are all directed to reinforcement learning.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements with Jibowu in order to train a deep reinforcement learning agent in the domain of an electronic trading platform. Doing so is beneficial because a user may revise their financial policy over time so they can learn which types of transactions yield the greatest benefit for themselves over time (Jibowu, [0035]).
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Viola in view of Ilhan, in view of Zheng, in view of Clements as applied to claim 1 above, and further in view of Hou et al. (“An Evolutionary Transfer Reinforcement Learning Framework for Multiagent Systems”), hereinafter Hou.
Regarding Claim 22:
As discussed above, Viola, Ilhan, Zheng, and Clements teach [the] system of claim 1, but do not explicitly disclose:
wherein the demonstrator is selected based at least on a location of the learning agent within the environment
However, in the same field, analogous art Hou teaches:
wherein the demonstrator is selected based at least on a location of the learning agent within the environment
Hou, p. 2, col. 2, “Specifically, meme external evolution happens whenever the current agent identifies an appropriate teacher agent via a meme selection process.”
On p. 2, col. 2, Hou teaches that an appropriate teacher agent [demonstrator] is identified and selected [selected] for a current agent [learning agent].
pp. 6-7, “1) Meme Selection…In our design, the decision of which peer an agent should learn from depends on both the agents’ superior historical success, namely elitist and the agents’ confidence in solving the request for a particular given situation, namely similarity. This meme selection process in eTL [evolutionary transfer reinforcement learning framework] is thus a fusion of the “imitate-from-elitist” and “like-attracts-like” principles.
On one hand, the elitist is maintained as a centralized resource to express the judgement of agents’ behaviors in accordance with the past knowledge…
On the other hand, similarity does not solely focus on the knowledge of the past, but the prediction on future reward under similar situations. This scheme does not impose higher performances in the partners but rather focuses on the sharing of relevant experiences in dealing with similar previously encountered situations. As a consequence, the similarity scheme narrows down the relevant elitist to learn from.”
On pp. 6-7, Hou specifies that the teacher agent to be learned from is identified from a group of agents based on elitist and similarity traits. The similarity trait identifies agents that have encountered similar situations in the same environment. The similarity trait corresponds to a location of the learning agent within the environment because the similarity trait identifies teacher agents that have encountered similar situations in the same environment as the learning agent.
Viola, Ilhan, Zheng, Clements, Hou, and the instant application are analogous art because they are all directed to reinforcement learning agents.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Viola, Ilhan, Zheng, and Clements with Hou to select a teaching agent/demonstrator based on similarity in order to increase the robustness of selecting a teaching agent. “On one hand, the elitist is maintained as a centralized resource to express the judgement of agents’ behaviors in accordance with the past knowledge…although the elitist has a higher performance than the partners, it might not be the most suitable candidate in solving the given unseen circumstance. On the other hand, similarity does not solely focus on the knowledge of the past, but the prediction on future reward under similar situations. This scheme does not impose higher performances in the partners but rather focuses on the sharing of relevant experiences in dealing with similar previously encountered situations” (Hou, pp. 6-7). Hou discloses that choosing a demonstrator based on their performance may not always result in solving unseen circumstances. The similarity score does not focus on performance, but focuses on experiences with similar situations in which the agent is moving through the environment. This allows the learning agent to select a better demonstrator since it takes into account the demonstrator’s performance and as well as its experience with similar situations in the environment.
Response to Arguments
Applicant's arguments filed July 29th, 2026, have been fully considered but they are not persuasive.
35 U.S.C. § 103:
Remarks, pp. 7-10. Applicant’s arguments with respect to claims 1, 9, and 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PHUNG whose telephone number is (703) 756-1499. The examiner can normally be reached Monday-Thursday: 9:00AM-4:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KAMRAN AFSHAR can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.H.P./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights