DETAILED ACTION
This office action is in response to amendments filed on 03/03/2026.
Claims 1, 5-8, 11, and 15-18 have been amended. Claims 1-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Objection to the specification:
In light of applicant’s amendment to the specification (pg. 2), the objection to the specification has been withdrawn.
Rejections Under 35 USC § 101:
Applicant's arguments regarding the rejections under 35 USC § 101 (pg. 11-20) have been fully considered but they are not persuasive.
Regarding Step 2A, Prong One, applicant argues that the independent claim is not directed to a mental process because it recites features which are claimed as being performed on a computing device and thus cannot practically be performed in the human mind. Examiner respectfully notes that, per MPEP 2106.04(a)(2)(III), “[c]laims can recite a mental process even if they are claimed as being performed on a computer.” In this case, as can be seen in the rejection below, while the steps of “defining… a parameter value” and “determining… a policy” may be claimed as being performed on a computing device, they can be considered mental processes as defining a parameter value based on an observed state and determining a policy based on the parameter value are judgements which can practically be performed in the human mind. Performing these steps using a generic computing device and a generic machine learning algorithm, as claimed, amounts to mere instructions to implement the mental processes in a computing environment.
Regarding Step 2A, Prong Two and Step 2B, applicant argues that the amended independent claim integrates the abstract idea into a practical application by providing a technological improvement to sequential decision-making systems. Applicant specifically argues that the implementation of a reinforcement learning agent coupled to a guidance module, prediction module, and optimization module improves upon sequential decision-making systems by reducing the time and computational costs associated with performing sequential operations. Applicant points to specification paragraph 0086, which asserts that these improvements are achieved by the guidance function, which can “limit the size of a parameter value search space” by “accounting for domain-specific data.” However, the cited paragraph does not explain what it means to account for domain-specific data, how doing so limits the size of a parameter value search space, or how this reduces time and computational costs, nor do applicant’s remarks. According to MPEP 2106.05(a), “if the specification explicitly sets forth an improvement but in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art), the examiner should not determine the claim improves technology.” Further, examiner notes that the guidance function relied upon in this argument is not actually required by the independent claim due to its presence in the Markush grouping introduced by the phrase “at least one of”.
The rejections under 35 USC § 101 have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Prior Art Rejections:
Applicant's arguments regarding the prior art rejections (pg. 20-23) have been fully considered but they are not persuasive. Applicant argues that the features of the amended independent claims are not disclosed or suggested by the cited references. Applicant specifically points to the amended limitation which recites “defining, by the computing device, a parameter value of the optimization module based on at least one of an observed state of the subsystem determined by the reinforcement learning agent, a guidance signal generated by a guidance function implemented by the guidance module to account for domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem, or…”
Applicant argues that the cited references Bohn and Alonso do not disclose use of a guidance function, and thus fail to teach the above limitation. Examiner respectfully notes that, on account of being an alternative set forth as part of the Markush grouping introduced by the phrase “at least one of”, the guidance function is not required by the claim. As can be seen in the rejection below, Bohn teaches a different alternative from the Markush grouping (“an observed state of the subsystem determined by the reinforcement learning agent”). Therefore, Bohn and Alonso disclose or suggest every element of the as-amended independent claim.
Applicant additionally argues that the cited reference Francon does not overcome the deficiencies of Bohn and Alonso. Examiner notes that, as explained above, Bohn and Alonso disclose or suggest every element of the as-amended independent claim. However, Francon is relied upon to remedy the potential deficiencies of Bohn and Alonso with regard to claims 7-8 and 17, which applicant also addresses. Applicant argues that Francon does not mention a guidance function, and that examiner’s comparison between the claimed guidance function and Francon’s surrogate model is erroneous because a guidance function is used to determine a parameter value of an optimization module, while a surrogate model approximates an underlying function. Examiner notes that, as is explained in the rejection below, Francon’s surrogate model is used to evaluate candidate policies and guide the optimization of the policy parameters of a sequential decision-making system. Therefore, the surrogate model disclosed by Francon falls within the broadest reasonable interpretation of the claimed guidance function. Francon is not relied upon in the rejection of the independent claims, and adequately remedies the potential deficiencies of Bohn and Alonso with regard to dependent claims 7-8 and 17.
The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1:
Step 1: The claim is directed to a method, which falls within the statutory category of a
process.
Step 2A Prong 1: The claim is directed to an abstract idea. Specifically, the claim recites:
defining, [by the computing device], a parameter value of the optimization module based on at least one of an observed state of the subsystem determined by the reinforcement learning agent, a guidance signal generated by a guidance function implemented by the guidance module to account for domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem, or a reward provided to the prediction module based on the observed state; (Abstract idea – mental process. Defining a parameter value based on an observed state of the subsystem can practically be performed in the human mind or with the aid of pen and paper, for example, by visually observing the state of the subsystem and mentally defining a parameter value. The courts have recognized that claims can recite a mental process even if they are claimed as being performed on a computer. See MPEP 2106.04(a)(2)(III).)
determining, [using a machine learning algorithm of the reinforcement learning agent], a policy that is defined by the optimization module based on the parameter value and a predicted future state of the subsystem that is predicted by the prediction module based on the reward, the policy comprising a suggested action to be performed by the subsystem to achieve a defined goal; (Abstract idea – mental process. Determining a suggested action policy based on the parameter value and a predicted future state can practically be performed in the human mind or with the aid of pen and paper, for example, by considering the parameter value and predicted state and mentally determining a suitable policy. The courts have recognized that claims can recite a mental process even if they are claimed as being performed on a computer. See MPEP 2106.04(a)(2)(III).)
Step 2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination. Specifically, the claim recites the additional elements:
implementing, by a computing device, a reinforcement learning agent in a subsystem of the sequential decision-making system, the reinforcement learning agent being coupled to a guidance module, a prediction module and an optimization module of the subsystem; (Reinforcement learning is standard in the field of machine learning, and implementing reinforcement learning by a generic computing device on a generic subsystem coupled to generic guidance, prediction, and optimization modules amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
defining the parameter value by the computing device (Using the computing device to define the parameter value amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
determining the policy using a machine learning algorithm of the reinforcement learning agent (Using a generic machine learning algorithm to determine the policy amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
implementing, by the computing device, the policy to cause the subsystem to perform the suggested action. (Implementing the policy to perform an action is a necessary output step which is incidental to the primary process, and thus amounts to adding insignificant extra-solution activity (necessary output) to the judicial exception – see MPEP2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, the claim recites the additional elements:
implementing, by a computing device, a reinforcement learning agent in a subsystem of the sequential decision-making system, the reinforcement learning agent being coupled to a guidance module, a prediction module and an optimization module of the subsystem; (Reinforcement learning is standard in the field of machine learning, and implementing reinforcement learning by a generic computing device on a generic subsystem coupled to generic guidance, prediction, and optimization modules amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
defining the parameter value by the computing device (Using the computing device to define the parameter value amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
determining the policy using a machine learning algorithm of the reinforcement learning agent (Using a generic machine learning algorithm to determine the policy amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).)
implementing, by the computing device, the policy to cause the subsystem to perform the suggested action. (Implementing the policy to perform an action is a necessary output step which is incidental to the primary process, and thus amounts to adding insignificant extra-solution activity (necessary output) to the judicial exception – see MPEP2106.05(g).)
Claims 2-20:
Claim 2 recites The method to provide reinforcement learning of claim 1, wherein implementing the reinforcement learning agent in the subsystem comprises: implementing, by the computing device, the prediction module to predict the predicted future state based on the observed state and the reward; and implementing, by the computing device, the optimization module to define the policy based on the parameter value and the predicted future state. Predicting the future state based on the observed state and the reward and defining the policy based on the parameter value and the predicted state are judgements/evaluations which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Implementation of the prediction module and optimization module by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f).Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 3 recites The method to provide reinforcement learning of claim 1, wherein defining the parameter value of the optimization module comprises: defining, by the computing device, an objective function parameter value or a constraint parameter value of the optimization module. Defining an objective function parameter value or constraint parameter value is a judgement/evaluation which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Defining the parameter by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 4 recites The method to provide reinforcement learning of claim 1, wherein defining the parameter value of the optimization module comprises: tuning, by the computing device, an objective function parameter or a constraint parameter of the optimization module to respectively define an objective function parameter value or a constraint parameter value of the optimization module. Tuning an objective function parameter value or constraint parameter value is a judgement/evaluation which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Tuning the parameter by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 5 recites The method to provide reinforcement learning of claim 1, wherein determining the policy comprises: determining, using the machine learning algorithm, the policy based on a second observed state corresponding to a second subsystem of the sequential decision-making system. Learning the policy based on a second observed state corresponding to a second subsystem is a judgement/evaluation which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Learning the policy by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 6 recites The method to provide reinforcement learning of claim 1, further comprising: determining, using the machine learning algorithm, a second policy corresponding to a second subsystem of the sequential decision-making system, the second policy respectively comprising a second suggested action to be performed by the second subsystem to achieve the defined goal. Learning a second suggested action policy for a second subsystem is a judgement/evaluation which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Learning the second policy by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 7 recites The method to provide reinforcement learning of claim 1, further comprising: determining, using the machine learning algorithm, the parameter value based on domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem. Learning the parameter value based on domain-specific data is a judgement/evaluation which can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process). See MPEP 2106.04(a)(2)(III). Learning the parameter value by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 8 recites The method to provide reinforcement learning of claim 1, further comprising: implementing, by the computing device, an evolutionary search algorithm that uses the guidance function to define the parameter value based on the domain-specific data associated with the domain of at least one of the sequential decision-making system or the subsystem. Defining the parameter using an evolutionary search algorithm with a guidance function based on domain-specific data can practically be performed in the human mind or with the aid of pen and paper (i.e. mental process), for example, by recursively mentally generating candidate parameter values, evaluating their optimality using a simple guidance function which can be computed mentally based on domain specific data, and mentally identifying the optimal parameter value based on the guidance function’s output. Implementing the evolutionary search algorithm by the computing device amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea – see MPEP 2106.05(f). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 9 recites The method to provide reinforcement learning of claim 1, wherein the sequential decision-making system comprises a distributed control system, and wherein implementing the policy to cause the subsystem to perform the suggested action comprises: implementing, by the computing device, the policy in the distributed control system to cause the subsystem to perform the suggested action in the distributed control system. This limitation merely qualifies the environment in which the judicial exception is performed as a distributed control system, and thus amounts to generally linking the use of a judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). Therefore, the claim does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim 10 recites The method to provide reinforcement learning of claim 1, wherein the optimization module comprises a convex optimization model. A convex optimization model, in its broadest reasonable interpretation, is directed to a mathematical concept. See MPEP 2106.04(a)(2)(I). Therefore, the claim merges with the abstract idea recited in claim 1 and does not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claims 11-17 are system claims containing substantially the same elements as method claims 1-7, respectively, and are rejected on the same grounds under 35 U.S.C. 101 as claims 1-7, respectively, mutatis mutandis. The additional components of A computing device, comprising: a memory device to store computer-readable instructions thereon; and at least one processing device configured through execution of the computer-readable instructions to: are interpreted as a general-purpose computer and mere instructions to apply the judicial exception on the computer. Therefore, the claims do not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claims 18-20 are product claims containing substantially the same elements as method claims 1-3, respectively, and are rejected on the same grounds under 35 U.S.C. 101 as claims 1-3, respectively, mutatis mutandis. The additional components of A non-transitory computer-readable medium embodying at least one program that, when executed by at least one computing device, directs the at least one computing device to: are interpreted as a general-purpose computer and mere instructions to apply the judicial exception on the computer. Therefore, the claims do not recite additional elements that are sufficient to amount to significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 9-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over
Bøhn et al. (hereinafter Bøhn), “Optimization of the Model Predictive Control Meta-Parameters Through Reinforcement Learning” in view of
Alonso et al. (hereinafter Alonso), “Distributed and Localized Model Predictive Control. Part I: Synthesis and Implementation”.
Regarding Claim 1,
Bøhn teaches A method to provide reinforcement learning for a sequential decision-making system, comprising:
implementing, by a computing device, a reinforcement learning agent in [a subsystem of] the sequential decision-making system, the reinforcement learning agent being coupled to a guidance module, a prediction module, and an optimization module [of the subsystem]; (Pg. 2, section II-A: “The MPC [Model Predictive Control] receives as arguments the current state of the plant,
x
-
t
, exogenous input variables (e.g. reference signals),
p
^
t
, as well as the prediction horizon,
N
t
, for the OCP [optimal control problem]. We label the MPC control law (for a given horizon selected by the horizon policy
π
θ
N
N
, more on this in Section III-B3) as:
u
t
:
t
+
N
t
-
1
M
,
x
^
t
:
t
+
N
t
=
π
θ
M
M
(
x
-
t
,
p
^
t
,
N
t
)
where the first return value is the optimal input sequence, the second return value is the predicted optimal state trajectory, and
θ
M
are the tunable parameters of the MPC scheme.” Pg. 8, section IV.A: “The dynamics of the system are highly nonlinear, and further the system is unstable, meaning that a controller is necessary to guide the system to stable conditions and then to maintain the stability.” Pg. 1, section I: “In this paper we propose the novel idea of tuning the meta-parameters of the MPC scheme using reinforcement learning (RL).” MPC is a sequential decision-making system which predicts the state trajectory (i.e. prediction module) and determines the optimal input sequence (i.e. optimization module) for a controller which guides the system to stability (i.e. a guidance module). The parameters of the MPC are tuned by (i.e. coupled to) a reinforcement learning agent.)
defining, by the computing device, a parameter value of the optimization module based on at least one of: an observed state of the subsystem determined by the reinforcement learning agent, a guidance signal generated by a guidance function implemented by the guidance module to account for domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem, or a reward provided to the prediction module based on the observed state; (Pg. 1, section I: “The main tunable parameter in these regards is the prediction horizon [
N
], which essentially controls how far into the future the MPC evaluates the optimality of its solution.” Pg. 6, section III-B4: “We collect all the parameters of the meta-parameter-deciding recomputation and horizon policies described above, and the parameters of the controllers into a single parameter vector
θ
=
θ
c
,
θ
N
,
α
,
θ
M
,
θ
L
,
Σ
M
,
Σ
M
L
T
, and define the complete policy
π
θ
, whose input is the state
s
and output is the action
a
=
[
c
,
N
,
u
M
,
u
M
L
]
.” The RL agent’s action defines the prediction horizon
N
(i.e. parameter value of the optimization module) based on the input state
s
(i.e. observed state determined by the reinforcement learning agent).)
determining, using a machine learning algorithm of the reinforcement learning agent, a policy that is defined by the optimization module based on the parameter value and a predicted future state [of the subsystem] that is predicted by the prediction module based on the reward, the policy comprising a suggested action to be performed [by the subsystem] to achieve a defined goal; and (Pg. 4, section III-A: “[T]he control law
π
θ
M
,
L
C
S
(22) that determines
u
t
consists of
π
θ
M
M
(20) — which depends on the state and exogenous variables
x
-
i
,
p
^
i
and the prediction horizon
N
i
at the last MPC computation — and
π
θ
M
L
M
,
L
(21), which depends on the current state
x
t
and the
(
t
-
i
)
t
h
element of the MPC’s predicted state trajectory, and
x
t
s
which depends on
p
^
t
.” Pg. 3, section II.B: “The objective in RL is to develop a policy
π
θ
, i.e. a function that maps from states to actions (here parameterized by
θ
) that maximizes the expected discounted sum of rewards. In this paper we use the policy gradient algorithm proximal policy optimization (PPO)… In general, policy gradient algorithms updates a parameterized stochastic policy directly in the parameter space, by sampling actions from the policy’s action distribution and observing the outcomes in terms of states and rewards. Parameters are adjusted to increase the likelihood of actions leading to high rewards using gradient ascent with gradients from the policy gradient theorem…” A policy gradient algorithm (i.e. machine learning algorithm of the RL agent) is used to determine the policy
π
θ
M
,
L
C
S
which determines the next control input
u
t
(i.e. a suggested action) that is optimal (i.e. to achieve a defined goal) based on the prediction horizon
N
(i.e. the parameter value) and the
(
t
-
i
)
t
h
element of the MPC’s predicted state trajectory (i.e. a predicted future state). The elements of the predicted state trajectory depend on the prediction horizon N which is set by the RL agent (i.e. the predicted future state is based on the reward).)
Bøhn does not appear to explicitly disclose
a subsystem of the sequential decision-making system
implementing, by the computing device, the policy to cause the subsystem to perform the suggested action.
However, Alonso teaches a subsystem of the sequential decision-making system (Pg. 3, section 2: “The system is composed of N interconnected subsystems (each having one or more states), so the state, control, and disturbance inputs can be suitably partitioned as
[
x
]
i
,
[
u
]
i
, and
[
w
]
i
for each subsystem
i
…”)
implementing, by the computing device, the policy to cause the subsystem to perform the suggested action. (Pg. 1, Abstract: “DLMPC is a distributed closed-loop model predictive control (MPC) scheme wherein only local state and model information needs to be exchanged between subsystems for the computation and implementation of control actions.” A control action, having been computed (i.e. suggested by the policy), is implemented (i.e. performed) by the subsystem.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Bøhn and Alonso. Bøhn teaches using reinforcement learning to optimize parameters of model predictive control (MPC) for sequential decision making. Alonso teaches a scheme for distributed model predictive control (MPC) in large-scale distributed control systems. One of ordinary skill would have motivation to combine Bøhn and Alonso because “[m]odel predictive control (MPC) is a powerful optimizing control technique, capable of controlling a wide range of systems with high control proficiency while respecting system constraints” (Bøhn, pg. 1, section I), “However, the need to control increasingly large-scale, distributed, and networked systems has limited [MPC’s] applicability” (Alonso, pg. 1, section I). With Alonso’s scheme, “we are able to distribute the computation via ADMM [22], thus allowing the online computation of closed-loop MPC policies to be carried out in a scalable localized manner” (Alonso, pg. 2, section I).
Regarding Claim 2, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Bøhn also teaches wherein implementing the reinforcement learning agent in the subsystem comprises:
implementing, by the computing device, the prediction module to predict the predicted future state based on the observed state and the reward; and (Pg. 2, section II-A: “The MPC [Model Predictive Control] receives as arguments the current state of the plant,
x
-
t
, exogenous input variables (e.g. reference signals),
p
^
t
, as well as the prediction horizon,
N
t
, for the OCP [optimal control problem]. We label the MPC control law (for a given horizon selected by the horizon policy
π
θ
N
N
, more on this in Section III-B3) as:
u
t
:
t
+
N
t
-
1
M
,
x
^
t
:
t
+
N
t
=
π
θ
M
M
(
x
-
t
,
p
^
t
,
N
t
)
where the first return value is the optimal input sequence, the second return value is the predicted optimal state trajectory, and
θ
M
are the tunable parameters of the MPC scheme.” The predicted optimal state trajectory
x
^
t
:
t
+
N
t
(i.e. predicted future state) is computed based on the current state
x
-
t
and the prediction horizon
N
t
set by the RL agent (i.e. based on the reward).)
implementing, by the computing device, the optimization module to define the policy based on the parameter value and the predicted future state. (Pg. 4, section III-A: “the control law
π
θ
M
,
L
C
S
(22) that determines
u
t
consists of
π
θ
M
M
(20) — which depends on the state and exogenous variables
x
-
i
,
p
^
i
and the prediction horizon
N
i
at the last MPC computation — and
π
θ
M
L
M
,
L
(21), which depends on the current state
x
t
and the
(
t
-
i
)
t
h
element of the MPC’s predicted state trajectory, and
x
t
s
which depends on
p
^
t
.” The control law determines the next control input
u
t
(i.e. defines the policy) based on the prediction horizon
N
(i.e. the parameter value) and the
(
t
-
i
)
t
h
element of the MPC’s predicted state trajectory (i.e. the predicted future state).)
Regarding Claim 3, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Bøhn also teaches wherein defining the parameter value of the optimization module comprises: defining, by the computing device, an objective function parameter value or a constraint parameter value of the optimization module. (Pg. 1, section I: “Other parameters of the MPC scheme are also subject to tuning, e.g. discretization step size, objective functions, optimality tolerances and constraints.”)
Regarding Claim 4, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Bøhn also teaches wherein defining the parameter value of the optimization module comprises: tuning, by the computing device, an objective function parameter or a constraint parameter of the optimization module to respectively define an objective function parameter value or a constraint parameter value of the optimization module. (Pg. 1, section I: “Other parameters of the MPC scheme are also subject to tuning, e.g. discretization step size, objective functions, optimality tolerances and constraints.”)
Regarding Claim 5, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Alonso also teaches wherein determining the policy comprises: determining, using the machine learning algorithm, the policy based on a second observed state corresponding to a second subsystem of the sequential decision-making system. (Pg. 4, section 2: “the closed-loop control policy at sub-controller
i
can be computed using only states, control actions, and system models collected from
d
-hop incoming neighbors of subsystem
i
in the communication topology
G
(
A
,
B
)
.” The control policy at a sub-controller
i
(i.e. the policy) is computed (i.e. determined) based on the observed state collected from a neighboring sub-controller (i.e. a second subsystem).)
Regarding Claim 6, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Alonso also teaches further comprising: determining, using the machine learning algorithm, a second policy corresponding to a second subsystem of the sequential decision-making system, the second policy respectively comprising a second suggested action to be performed by the second subsystem to achieve the defined goal. (Pg. 1, Abstract: “DLMPC is a distributed closed-loop model predictive control (MPC) scheme wherein only local state and model information needs to be exchanged between subsystems for the computation and implementation of control actions.” Pg. 4, section 2: “the closed-loop control policy at sub-controller
i
can be computed using only states, control actions, and system models collected from
d
-hop incoming neighbors of subsystem
i
in the communication topology
G
(
A
,
B
)
.” The control policy and action computed (i.e. determined) at a sub-controller other than sub-controller
i
is a second policy comprising a second suggested action corresponding to a second subsystem.)
Regarding Claim 9, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Alonso also teaches wherein the sequential decision-making system comprises a distributed control system, and wherein implementing the policy to cause the subsystem to perform the suggested action comprises: implementing, by the computing device, the policy in the distributed control system to cause the subsystem to perform the suggested action in the distributed control system. (Pg. 1, Abstract: “we present the Distributed and Localized Model Predictive Control (DLMPC) algorithm for large-scale linear systems. DLMPC is a distributed closed-loop model predictive control (MPC) scheme wherein only local state and model information needs to be exchanged between subsystems for the computation and implementation of control actions.” DLMPC performs sequential decision making in a distributed control system where each subsystem in the distributed control system computes and implements (i.e. performs) control actions.)
Regarding Claim 10, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Alonso also teaches wherein the optimization module comprises a convex optimization model. (Pg. 9, section 3.3.2: “These results allow us to solve the DLMPC problem (6) using standard convex optimization methods, and further preserves the locality structure of the original problem under the given assumptions.”)
Claims 11-16 are system claims containing substantially the same elements as method claims 1-6, respectively. Bøhn and Alonso teach the elements of claims 1-6, as shown above.
Alonso also teaches A computing device, comprising: a memory device to store computer-readable instructions thereon; and at least one processing device configured through execution of the computer-readable instructions to: (Examiner notes that this limitation is interpreted as implementation of the disclosed method in a generic computing environment. Pg. 13, section 5: “All code needed to replicate these experiments was implemented using the SLS toolbox [27] and is available at https://github.com/unstable-zeros/dl-mpc-sls.” Code implementation using the SLS toolbox necessitates a computing environment.)
Claims 18-20 are product claims containing substantially the same elements as method claims 1-3, respectively. Bøhn and Alonso teach the elements of claims 1-3, as shown above.
Alonso also teaches A non-transitory computer-readable medium embodying at least one program that, when executed by at least one computing device, directs the at least one computing device to: (Examiner notes that this limitation is interpreted as implementation of the disclosed method in a generic computing environment. Pg. 13, section 5: “All code needed to replicate these experiments was implemented using the SLS toolbox [27] and is available at https://github.com/unstable-zeros/dl-mpc-sls.” Code implementation using the SLS toolbox necessitates a computing environment.)
Claims 7-8 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Bøhn in view of Alonso and further in view of
Francon et al. (hereinafter Francon), U.S. Patent Application Publication US 20200311556 A1.
Regarding Claim 7, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Bøhn and Alonso do not appear to explicitly disclose further comprising: determining, using the machine learning algorithm, the parameter value based on domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem.
However, Francon teaches further comprising: determining, using the machine learning algorithm, the parameter value based on domain-specific data associated with a domain of at least one of the sequential decision-making system or the subsystem. (Abstract: “A surrogate-assisted evolutionary optimization method, ESP, discovers decision strategies in real-world applications. Based on historical data, a surrogate is learned and used to evaluate candidate policies with minimal exploration cost.” [0039]: “There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this historical data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes.” The policy parameters of a sequential decision-making system are optimized (i.e. the parameter value is determined) based on a surrogate model trained on historical data (i.e. based on domain-specific data associated with the domain of the decision-making system).)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Bøhn, Alonso, and Francon. Bøhn teaches using reinforcement learning to optimize parameters of model predictive control (MPC) for sequential decision making. Alonso teaches a scheme for distributed model predictive control (MPC) in large-scale distributed control systems. Francon teaches learning a surrogate model based on historical data to guide policy optimization in sequential decision-making. One of ordinary skill would have motivation to combine Bøhn, Alonso, and Francon because “[e]xtended into sequential decision making, ESP is highly sample efficient, has low variance, and low regret, making the policies reliable and safe. As an unexpected result, the surrogate also regularizes decision making, making it sometimes possible to discover good policies even when direct evolution fails. ESP is therefore a promising approach to improving decision making in many real world applications where historical data is available” (Francon, 0092).
Regarding Claim 8, Bøhn and Alonso teach The method to provide reinforcement learning of claim 1, as shown above.
Bøhn and Alonso do not appear to explicitly disclose further comprising: implementing, by the computing device, an evolutionary search algorithm that uses the guidance function to define the parameter value based on the domain-specific data associated with the domain of at least one of the sequential decision-making system or the subsystem.
However, Francon teaches further comprising: implementing, by the computing device, an evolutionary search algorithm that uses the guidance function to define the parameter value based on the domain-specific data associated with the domain of at least one of the sequential decision-making system or the subsystem. (Abstract: “A surrogate-assisted evolutionary optimization method, ESP, discovers decision strategies in real-world applications. Based on historical data, a surrogate is learned and used to evaluate candidate policies with minimal exploration cost.” [0039]: “There is now significant historical data available on decision making in organizations, consisting of the decision problem, what decisions were made, and how desirable the outcomes were. Using this historical data, it is possible to learn a surrogate model, and with that model, evolve a decision strategy that optimizes the outcomes.” An evolutionary optimization method (i.e. evolutionary search algorithm) which uses a surrogate model (i.e. guidance function) trained on historical data (i.e. based on domain-specific data associated with the domain of the decision-making system) is utilized to guide the optimization of the policy parameters (i.e. define the parameter value) of a sequential decision-making system.)
Claim 17 is a system claim containing substantially the same elements as method claim 7. Bøhn, Alonso, and Francon teach the elements of claim 7, as shown above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN M ROHD whose telephone number is (571)272-6445. The examiner can normally be reached Mon-Thurs 8:00-6:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.M.R./Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights