Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-2, 4-6, 9, 12-13, 15-18, & 21-22 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hasan, M. M., et al. (2019). Dynamic multi-objective optimisation using deep reinforcement learning: benchmark, algorithm and an application to identify vulnerable zones based on water quality. Engineering Applications of Artificial Intelligence, 86, 107–135. https://doi.org/10.1016/j.engappai.2019.08.014, hereinafter “Hasan”.
Claim 1:
Hasan teaches a computer implemented method of
training an action selection policy neural network defining an action selection policy used to select actions to be performed by an agent to control the agent to perform a task in an environment (i.e. pg. 108, In this work, we proposed a novel algorithm called parity Q deep Q network (PQDQN) which is able to identify the vulnerable zones based on water quality resilience in a dynamic multi-objective environment. The agent in the partially observable Markov decision process detects the changes in the dynamic setting and finds the vulnerable zones), the task having multiple associated objectives (i.e. pg. 120, The meta-policy of selecting a policy is defined by the parity value which is obtained by the objective relation mapping (ORM) in a dynamic environment to satisfy or closely satisfy the objectives), the method comprising: obtaining data defining an updated version of the action selection policy for selecting an action for the agent in response to an observation of a state of the environment (i.e. pg. 120, to identify the meta-policy in the changing environment the followings need to be done: a. detecting changes by re-evaluating detectors or changing component), by using a reinforcement learning technique based on rewards received subsequent to selected actions (i.e. pg. 120, In the vector reward space, an action a in the state s under any non-dominated policy can be denoted as ⃗𝑄∗(𝑠,𝑎). Here, the fact is the obtained reward is a vector where the agent does not learn a single policy but a set of policies at the same time); obtaining data defining a second action selection policy for selecting an action for the agent in response to an observation of a state of the environment (i.e. pg. 120, “it is also observable that the policy is selected based on the individual input of the DQNs which are the representatives of the state and action for a particular objective”, wherein it is noted that these DQNs represent the state–actions portfolio that needs to be selected by an optimal policy that may be defined by observations in the changing environment); determining a first policy projection value dependent on a measure of a difference between the updated version of the action selection policy and the action selection policy (i.e. pg. 119, “State–action networks learn the Q-values for each action 𝑎1 given a state 𝑆1, at some particular time step 𝑡”, wherein the BRI for a first policy projection encompasses the Q-values that are mathematical estimates that determine the expected future rewards an agent will receive for taking a specific action in a given state); determining a second policy projection value dependent on a measure of a difference between the second action selection policy and the action selection policy (i.e. pg. 120, “a parity value (i.e. dynamic weight) has been introduced before summing up the Q-values from DQNs that ensures equilibrium between objective”, wherein the BRI for a second policy projection value encompasses how each objective may have a dynamic Q-value as the environment changes); determining a combined objective value from a weighted combination of the first policy projection value and the second policy projection value (i.e. pg. 122, it forwards the best compromising objectives after satisfying the constraints. This module balances the Q values to achieve the possible optimal Q values for the different objectives by averaging them or multiplying with the preference values); and training the action selection policy neural network by adjusting the parameters of the action selection policy neural network to optimize the combined objective value (i.e. pg. 132, Our proposed generic algorithm (i.e.PQDQN) enables decomposition of problems into sub problems and make a relation and map with different objectives to find out compromising solutions that adhere to the POF. It also provides a method for robust manipulation of priorities after the training which may ignore a specific behaviour or objective provided by DQN).
Claim 2:
Hasan teaches the method of claim 1 wherein obtaining the data defining the updated version of the action selection policy: maintaining a Q-value neural network configured to process an observation of a state and an action for the agent to generate a Q-value (i.e. pg. 120, “The values learned in this scenario can be denoted as ⃗𝑄(𝑠,𝑎) of vectors, which are used to estimate the optimal ⃗ 𝑄∗(𝑠,𝑎) sets”, wherein s represents an observation of state); training the Q-value neural network by reinforcement learning, using the reinforcement learning technique based on the rewards, to optimize a first, task-related objective function (i.e. pg. 109, Fig. 3 shows a multi-objective deep reinforcement learning model where an agent takes an optimal action (i.e. policy) for a state in an environment and earns reward points (e.g. vector rewards for multi objective cases)); and using the Q-value neural network to obtain the data defining the updated version of the action selection policy (i.e. pg. 120-121, In the deep layer, the weights of the neural network are adjusted based on the backpropagation procedure (Baldi and Sadowski, 2018). The obtained Q value by the agent is the average value from the DQNs that characterise all the objectives. Therefore, the selected objectives are the representation of the compound structure of all the objectives. Needless to mention, this value is made by the Q values in a finite horizon. In other words, this can be represented as the most compromising solutions that the agent could achieve in a particular episode).
Claim 4:
Hasan teaches the method of claim 1, further comprising obtaining training data by, for each of one or more time steps: obtaining an observation of the state of the environment (i.e. pg. 109, Multi-objective reinforcement learning (MORL) was proposed by (Zoltán et al., 1998). MORL deals with the decision-making problems in uncertain situations where the agent learns by interacting and taking feedback within the environment); processing the observation using the action selection policy neural network to generate a policy output (i.e. pg. 109, Fig. 3 shows a multi-objective deep reinforcement learning model where an agent takes an optimal action (i.e. policy) for a state in an environment); selecting an action to be performed by the agent in response to the observation using the policy output (i.e. pg. 110, This framework provides a standard model to design the system with probabilistic, nondeterministic and controlled behaviour which helps to analyse the states and necessary actions for each state); causing the agent to perform the selected action and, in response, receiving a reward characterizing progress made on the task (i.e. pg. 109, multi-objective reinforcement learning depends on the use of reward mechanism; such as scalar reward and vector reward which are used for single and multi-objective problem respectively); and obtaining the data defining the updated version of the action selection policy using the reinforcement learning technique based on the rewards received subsequent to the actions selected using the policy output (i.e. pg. 109, it is observable that the target network 𝑄′(𝜃′) is iterating until it gets the highest expected reward over time that can easily fit within the dynamic environment according to our requirements).
Claim 5:
Hasan teaches the method of claim 4,
comprising iteratively obtaining the training data, and training the action selection policy neural network (i.e. pg. 109, it is observable that the target network 𝑄′(𝜃′) is iterating until it gets the highest expected reward over time that can easily fit within the dynamic environment according to our requirements).
Claim 6:
Hasan teaches the method of claim 1, wherein the data defining the second action selection policy comprises a dataset of transitions each comprising an observation characterizing a state of the environment at a time step, an action that was performed at the time step, and a reward received subsequent to performing the action (i.e. pg. 113, “This environment consists of 10 rows and 9 columns with three different types of cells such as water cells where the vessel can traverse, sea ground cells that cannot be traversed as the edges of the grid and treasure cells that provide different rewards and by reaching these cells, an episode ends. The agent controls a submarine that searches for treasures under the sea. The objectives of the agent are to find out the highest valued treasure within minimum time”, wherein an example shows how observations of environmental change, such as moving through squares, and the actions with subsequent rewards, such as finding treasure or not, make up data defining a policy for an agent) ; and wherein obtaining the data defining the updated version of the action selection policy uses the reinforcement learning technique based on the rewards in the dataset (i.e. pg. 113, “o illustrate the problem mathematically, the components of the environment are given below:– A submarine (i.e. black coloured) that works as an agent,– the treasure values,– time,– rewards and– health meter for the dynamic DST environment (attack by enemy) Considering the objectives, the agent needs to hunt the highest treasures in minimum time. However, for every step, the agent is penalised by −1 point. So these two aims consist the conflicting objectives such as maximisation and minimisation for all the considered environment”, wherein the agent policy is iteratively updated as by the actions and appropriate rewards).
Claim 9:
Hasan teaches the method of claim 4
wherein the data defining the second action selection policy comprises data from a model policy output of an action selection model configured to process an input from an observation representing a state of the environment and to generate the model policy output for selecting an action for the agent (i.e. pg. 119-120, “Each group comprises several nodes that correspond to the number of possible actions which leads to governing the policy… it is observable that the agent determines the compromising solutions based on the balance of the several objectives. These Q values are forwarded by the DQNs which consist of the set of state and action values for a particular episode that is generated by the emulator.”, wherein it is noted that the meta-policy governs an agents specific steps or actions required to reach a final goal) .
Claim 12:
Hasan teaches the method of claim 4
wherein the data defining the second action selection policy is derived from a second Q-value neural network configured to process an observation of a state and an action for the agent to generate a second Q-value, the method further comprising training the second Q-value neural network by reinforcement learning using the training data to optimize a second, task-related objective function (i.e. pg. 124, “Table 8 shows the training steps until convergence over 100 agents. From Tables 7 and 8, it is clear that our developed PQDQN earns highest expected rewards than MPDQN, MPQ and MO-MCTS in all the cases. However, the proposed algorithm takes reasonably higher steps compared to MO-MCTS in the Dynamic DST (attack by enemy) environment.”, wherein in an example where an agent’s objective is to find treasure, a second objective to avoid being destroyed by an enemy may be considered in the multi-policy DQN (MPDQN)).
Claim 13:
Hasan teaches the method of claim 12 further comprising: maintaining a further Q-value neural network configured to process an observation of a state and an action for the agent to generate a further Q-value, and training the further Q-value neural network by reinforcement learning using the training data to optimize a further, task- related objective function (i.e. pg. 122, This module balances the Q values to achieve the possible optimal Q values for the different objectives by averaging them or multiplying with the preference values (e.g., if any). This action is performed based on the balancing of all the objectives); using the further Q-value neural network to obtain data defining a second updated version of the action selection policy for selecting an action for the agent in response to an observation of a state of the environment (i.e. pg. 122, Thus, the agent interacts within the environment and learns the optimum values which lead to select the policy as discussed in the earlier section.); determining a third policy projection value dependent on a measure of a difference between the second updated version of the action selection policy and the action selection policy; and determining the combined objective value from a weighted combination of the first policy projection value, the second policy projection value, and the third policy projection value (i.e. pg. 122, The target of our proposed algorithm is to detect the changes and then tracking the changing optima (i.e. local optima or ideally the global optima) over time. From the below algorithm 1, it is noticeable that we need to provide vector rewards for each action and prioritise the objectives (i.e. if needed as like as the DST-enemy attack environ ment to prioritise health more than treasure). Therefore, unlike the previous one, this is a 3 objectives environment. After that, the agent needs to convey the state–action pair into a deep Q network and get the highest achieved Q value (multiply with the preference value—if any) in the stack for each episode.).
Claim 15:
Hasan teaches the method of claim 1.
wherein the weighted combination of the first policy projection value and the second policy projection value comprises a combination of the first policy projection value with a first weight and a combination of the second policy projection value with a second weight, the method further comprising adjusting the first and second weights to optimize the reward or return from the environment (i.e. pg. 118, “This grid-world is formed into states (i.e. zones with stations), actions (i.e. which zone to traverse), transition models (i.e. the selection from one zone to another) and rewards (i.e. achieved by identifying the resilient zone). The solution for this MOMDP is an optimal policy that relies on the vector rewards which determines the critical as well as the resilient zones. These rewards are connected to the objectives to find out the optimal policy. Since, the dynamic nature and partially observable MOMDP, the agent requires an experience replay as to store the past observations to make the best possible decisions which are close to the Pareto Front”. Wherein the experience replay stabilize the learning process by adjusting the model weighs to minimize the difference between the outcomes).
Claim 16:
Hasan teaches the method of claim 1, wherein the weighted combination of the first policy projection value and the second policy projection value is defined by a weight vector (i.e. pg. 120, However, in the context of dynamic multi-objective optimisation, we need to tweak the Q learning so that it can work with vector rewards. Thus, we need to extend the equation that can handle vector operations. Therefore, in the finite horizon or an episodic environment, the reward function ⃗𝑅 = 𝑆 ×𝐴×𝑆 is a vector of n rewards rather than a scalar with an element for each objective), the method further comprising: processing the observation and the weight vector using the action selection policy neural network to generate the policy output (i.e. pg. 120, In the vector reward space, an action 𝑎 in state 𝑠 under any non dominated policy can be denoted as ⃗𝑄∗(𝑠,𝑎));
and adjusting the weight vector to optimize the reward or return from the environment (i.e. pg. 120, As far as the dynamic environment is concerned, we have intro duced a meta-policy (i.e. governing the policies in the policy life-cycle) that defines which policy needs to be counted and prioritise the objectives… a parity value (i.e. dynamic weight) has been introduced before summing up the Q-values from DQNs that ensures equilibrium between objectives).
Claim 17:
Hasan teaches the method of claim 16, further comprising randomly sampling values for the weight vector during the training of the action selection policy neural network (i.e. pg. 120, “in the dynamic environment, the location of the optimum moves either deterministically or stochastically. This move can also be linear, non-linear, periodical or random over time during optimisation”, wherein the nature of an environment may lead to random sampling).
Claim 18:
Hasan teaches the method of claim 16, further comprising automatically adjusting the weight vector to optimize the rewards (i.e. pg. 120, “The meta-policy of selecting a policy is defined by the parity value which is obtained by the objective relation mapping (ORM) in a dynamic environment to satisfy or closely satisfy the objectives”, wherein it is noted that balancing the objectives using objective relation mapping is based on a dynamic weight (parity value)).
Claim 21:
Claim 21 is the system claim reciting similar limitations to claim 1 and is rejected for similar reasons
Claim 22:
Claim 22 is the medium claim reciting similar limitations to claim 1 and is rejected for similar reasons
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 7-8, 10-11, 14, is/are rejected under 35 U.S.C. 103 as being unpatentable over Hasan, M. M., et al. (2019). Dynamic multi-objective optimisation using deep reinforcement learning: benchmark, algorithm and an application to identify vulnerable zones based on water quality. Engineering Applications of Artificial Intelligence, 86, 107–135. https://doi.org/10.1016/j.engappai.2019.08.014, hereinafter “Hasan”, and in light of Abdolmaleki, A., Tobias, S. J., Degrave, J., Bohez, S., Tassa, Y., Belov, D., Heess, N., & Riedmiller, M. (2018). Relative Entropy Regularized Policy Iteration. ArXiv.org. https://arxiv.org/abs/1812.02256, hereinafter Abdolmaleki.
Claim 3:
Hasan teaches the method of claim 2
wherein the action selection policy neural network is configured to generate a policy output π (a|s) for selecting an action a to be performed by the agent in a state s of the environment, and wherein using the Q-value neural network to obtain the data defining the updated version of the action selection policy comprises multiplying π (a|s) by exp
PNG
media_image1.png
44
144
media_image1.png
Greyscale
, where Q (s, a) is the Q-value from the Q-value neural network for action a and state s (i.e. pg. 117, To solve the RL in this context, the Markov Decision Process (MDP) has been defined as the collection of the following components: • States: 𝑆 • Actions: 𝐴(𝑠),𝐴 • Transition model: 𝑇(𝑠,𝑎,𝑠}) ∼ 𝑃(𝑠}|𝑠,𝑎) • Rewards: 𝑅(𝑠),𝑅(𝑠,𝑎),𝑅(𝑠,𝑎,𝑠}) • Policy: 𝜋(𝑠) → 𝛼𝜋∗ is the optimal policy) and n is a temperature parameter (i.g. pg. 116-117, “this test case also provides the dynamics where the values are not static over time. These data are changing which depends on natural factors such as weather, temperature), to obtain the data defining the updated version of the action selection policy (i.e. pg. 123, “To reach all the states including the unvisited nodes, a 𝜀-greedy exploration policy has been implemented with annealing from 1 to 0.05.”, wherein a temperature parameter may be optimized by annealing).
While Hasan teaches a generic deep learning reinforcement algorithm for policy selection for an agent that includes considering a temperature parameter, Hasan may not explicitly teach that the
the action selection policy comprises multiplying π (a|s) by exp
PNG
media_image1.png
44
144
media_image1.png
Greyscale
.
However, Abdolmaleki teaches
the action selection policy comprises multiplying π (a|s) by exp
PNG
media_image1.png
44
144
media_image1.png
Greyscale
(i.e. pg. 4, “As it turns out, its solution can be obtained in closed form, and consists of a softmax over Q-values:
q
i
j
=
q
(
a
i
,
s
j
)
=
e
x
p
(
Q
π
k
s
j
,
a
i
η
)
/
Z
(
j
)
,
The temperature η corresponding to the constraint can be found automatically by solving the following convex dual function alongside our policy optimization:”, wherein it is noted that a policy improvement may used or any off-policy method for learning Q-values could be used here, as long as it provides sufficiently accurate value estimates).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add the action selection policy comprises multiplying π (a|s) by exp
PNG
media_image1.png
44
144
media_image1.png
Greyscale
, with Hasan’s multi-objective policy optimization, and a particular KL divergence policy evaluation equation for optimization, as taught by Abdolmaleki. One would have been motivated to combine the optimization methods of Abdolmaleki with multi-objective policy algorithm of Hasan as both are in the similar art of providing policy optimization and the combination provides an improved accuracy for value estimates when using learning for Q-values.
Claim 7:
Hasan teaches the method of claim 6.
Hasan teaches
wherein determining the second policy projection value comprises sampling one or more observations of states of the environment from the dataset, sampling one or more actions corresponding to the sampled observations from the dataset, and averaging a logarithm of a policy output from the action selection policy neural network for each sampled state and action pair (i.e. pg. 122, The proposed solution consists of two different blocks which represent a common RL setting such as selecting states and actions. The agent interacts with the environment and selects the actions based on the ORM… From Fig. 17, it is observable that the Q values are selected based on the states and actions. The state is selected based on the DQN networks and this value is sent to a stack where the agent looks for the best Q value… This module balances the Q values to achieve the possible optimal Q values for the different objectives by averaging them or multiplying with the preference values (e.g., if any)).
While Hasan teaches averaging a policy output from the action selection policy neural network for each sampled state and action pair, Hasan may not explicitly teach that what is averaged is
A logarithm.
However, Abdolmaleki teaches
averaging a logarithm of a policy output from the action selection policy neural network for each sampled state and action pair (i.e. pg. 4, we obtained an improved sample-based distribution over actions. Next, we want to generalize this sample-based solution over state and action space– which is required when we want to select better actions in unseen situations during control. For this, we solve a weighted supervised learning problem π(k+1) = argmax πθ K j N i qij log πθ(ai|sj),).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to averaging a logarithm of a policy output from the action selection policy neural network for each sampled state and action pair , with Hasan’s multi-objective policy optimization, and a fitting for an improved policy using a KL divergence, as taught by Abdolmaleki. One would have been motivated to combine the optimization methods of Abdolmaleki with multi-objective policy algorithm of Hasan as both are in the similar art of providing policy optimization and provide effective regularization that addresses both concerns is to limit the overall change in the parametric policy.
Claim 8:
Hasan and Abdolmaleki teach the method of claim 6.
Abdolmaleki further teaches comprising
averaging the logarithm of the policy output for each sampled state and action pair weighted by a state-action advantage value for the sampled state and action pair (i.e. pg. 4, Here, the first constraint forces the weights to stay close to the last policy probabilities, i.e. bounds the average relative entropy, or average KL, since samples ai are drawn from π(k). The second constraint ensures that weights are normalized. The solution will be new weights, given through the categorical probabilities qij = q(ai|sj), such that the expected Q-value increases while constraining the reduction in entropy (to prevent the weights from collapsing onto one action immediately)).
Claim 10:
Hasan teaches the method of claim 9
wherein determining the second policy projection value comprises sampling one or more observations of states of the environment from the training data, determining one or more actions corresponding to the sampled observations according to the action selection policy defined by the action selection policy neural network (i.e. pg. 120 To identify the meta-policy in the changing environment the followings need to be done: a. detecting changes by re-evaluating detectors or changing com ponents b. detecting changes based on the algorithmic behaviours c. balancing the objectives using objective relation mapping based on a dynamic weight (parity value)).
While Hasan teaches a sampled state and action pair corresponding to observations of states of the environment, Hasan may not explicit teach
for each sampled state and action pair, determining logarithm of a ratio of the model policy output from the action selection model for the sampled state and for the action, to the policy output from the action selection policy neural network for the sampled state and for the action.
However, Abdolmaleki teaches
for each sampled state and action pair, determining logarithm of a ratio of the model policy output from the action selection model for the sampled state and for the action, to the policy output from the action selection policy neural network for the sampled state and for the action (i.e. pg. 4, “the first constraint forces the weights to stay close to the last policy probabilities, i.e. bounds the average relative entropy, or average KL, since samples ai are drawn from π(k). The second constraint ensures that weights are normalized. The solution will be new weights, given through the categorical probabilities qij = q(ai|sj), such that the expected Q-value increases while constraining the reduction in entropy (to prevent the weights from collapsing onto one action immediately)”, wherein it is noted that each action and state pair q(ai|sj) undergoes relative entropy of average KL which is the average of the logarithm of the ration between the two distributions).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to for each sampled state and action pair, determining logarithm of a ratio of the model policy output from the action selection model for the sampled state and for the action, to the policy output from the action selection policy neural network for the sampled state and for the action, with Hasan’s multi-objective policy optimization, and the average KL algorithm for q(ai|sj) pairs, as taught by Abdolmaleki. One would have been motivated to combine the optimization methods of Abdolmaleki with multi-objective policy algorithm of Hasan as both are in the similar art of providing policy optimization as the combination ensures that weights are normalized to prevent the weights from collapsing onto one action immediately.
Claim 11:
Hasan and Abdolmaleki teach the method of claim 10.
Abdolmaleki further teaches wherein determining the second policy projection value further comprises averaging, over the determined states and actions, a product of a logarithm of the policy output network for the sampled state and for the action and an exponential function of the logarithm of the ratio (i.e. pg. 4, “Note that In our case q(a|s) is a non-parametric and samples based distribution, and we can solve this constraint optimization in close form for each sample state s”, wherein it is noted that an exponential function of the logarithm of the ratio for the Q pair helps the agent prioritize actions with higher expected rewards to derive an improved version of the policy
PNG
media_image2.png
38
359
media_image2.png
Greyscale
. )
Claim 14:
Hasan teaches the method of claim 1.
While Hasan teaches first and second policy projection values, Hasan may not explicitly each
However, Abdolmaleki teaches
wherein the first policy projection value and the second policy projection value each comprise a measure of a KL divergence (i.e. pg. 4, Here, the first constraint forces the weights to stay close to the last policy probabilities, i.e. bounds the average relative entropy, or average KL, since samples ai are drawn from π(k). The second constraint ensures that weights are normalized. The solution will be new weights, given through the categorical probabilities qij = q(ai|sj), such that the expected Q-value increases while constraining the reduction in entropy (to prevent the weights from collapsing onto one action immediately)).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to add wherein the first policy projection value and the second policy projection value each comprise a measure of a KL divergence, with Hasan’s multi-objective policy optimization, and a particular KL divergence policy evaluation equation for optimization of Q values, as taught by Abdolmaleki. One would have been motivated to combine the optimization methods of Abdolmaleki with multi-objective policy algorithm of Hasan as both are in the similar art of providing policy optimization and the combination provides an improved accuracy for value estimates when using learning for Q-values.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
U.S. Patent Application Publication NO. 20180012137 “Wright” teaches in para. [0047], a “policy evaluation phase, samples gathered following π are used by an on-policy algorithm to derive its value function, Q.sub.π. Then, in the policy improvement phase, Q.sub.πis used to calculate the improved policy, π′, by identifying any states where π took a sub-optimal action, π(s.sub.t)≠arg max.sub.a∈A Q.sub.π(s.sub.t,a). The process is then repeated for π′ and it is guaranteed to asymptotically converge upon Q* and π”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID H TAN whose telephone number is (571)272-7433. The examiner can normally be reached M-F 7:30-4:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.T./Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights