DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the amendment filed on Feb. 2nd, 2026. The amendments are linked to the original application filed on March 2nd, 2022.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on Feb. 2nd, 2026 has been entered.
Response to Amendment
The Examiner thanks the applicant for the remarks, edits and arguments.
Regarding Claim Rejections – 35 USC 103
Applicant Remarks:
The applicant argues that the prior art Bøhn and Richards fail to disclose the amendments made to the independent claims. Further the applicant believes that Richards discloses a process that fails to teach the use of an Actor-Critic model as described in the claims. Instead, the applicant argues, Richards discloses a meta-learning process but not in a reinforcement learning context.
Next the applicant argues that the claimed arts fail to disclose a system which uses an embedding network along with actor and critic models. The applicant has amended the independent claims to disclose that all three of these three networks are a part of the whole system as claimed. Therefore, the applicant believes that the proposed prior art fails to disclose a system as claimed and they believe that the current claims overcome the prior art and the rejection under 35 U.S.C. 103 should be withdrawn.
Examiner Response:
The applicant argues that Richards fails to disclose a reinforcement learning process as claimed. The examiner would like to point to Fig. 1 of Richards and note that this article does disclose a reinforcement learning process. The system disclosed in Richards is able to use input data of a current state of an environment and produce an action based on this input data. The disclosed model will then produce feedback for the ensemble of models. This process is similar to the core concepts of reinforcement learning, where an agent is set in an environment and the actor will generate actions based on the input data. The actor will receive feedback, a reward or punitive measures, based on the change in the environment.
The examiner has considered the amendments made to the independent claims. The examiner has noted that the current art proposed by the examiner does fail to disclose all of the elements added to the amended claims. In particular, as stated by the applicant, the arts now fail to disclose a system that uses multiple networks to generate an output or process to control an industrial process. The examiner no longer relies on the prior arts to completely disclose the independent claims. However, after each amendment, the examiner is required to perform a complete and through search of the amended claims to see if the comply with 35 U.S.C. 102 and 35 U.S.C. 103. After completing this search, the examiner has found subject matter which also discloses a model that contains an embedding, actor and critic models. The examiner believes that the combinations of arts would lead a person of ordinary skill of the art to disclose the claimed invention under 35 USC 103. Therefore, the current amended claims are rejected under 35 U.S.C. 103, see 103 rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al, (Liu et al, “SFC Embedding Meets Machine Learning: Deep Reinforcement Learning Approaches”, 2021, hereinafter “Liu”) and Rakelly et al, (Rakelly et al, “Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables”, 2019, hereinafter, “Rakelly”) in view of Richards et al, (Richards et al, "Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems", 2019, hereinafter "Richards").
Regarding claim 1, Liu discloses, “training, using an embedding neural network, an actor-critic deep reinforcement learning (DRL) algorithm wherein the embedding neural network processing context data including input-output process data and historical process data from the industrial process to generate an embedding input for the DRL algorithm;” (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927; “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models.)
“generating, by the embedding neural network, a multidimensional vector based on the training of the DRL, wherein the multidimensional vector is lower in dimensions as compared to the processing context data, summarizing the processing context data to represent dynamics of the industrial process and a control objective,” (Proposed A3C-Based SFC Embedding Approach, pp. 1928; “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all of the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data.)
“providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;” (Figure 1, pp. 1927; “As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network.)
“training a meta-reinforcement learning agent (meta-RL agent) using a meta-RL training algorithm, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, and the meta-RL agent is trained to collect a set of parameters from at least one process controllers, wherein the set of parameters are associated with the new industrial process; and” (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929; This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters.)
Liu fails to explicitly disclose, “A method of meta-reinforcement learning (MRL) for process control of an industrial process run by a process control system (PCS) including at least one process controller coupled to actuators that is configured for controlling processing equipment, comprising:” and “adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process;”.
However, Rakelly discloses, “A method of meta-reinforcement learning (MRL) for process control of an industrial process run by a process control system (PCS) including at least one process controller coupled to actuators that is configured for controlling processing equipment, comprising:” (Meta-RL Problem Statement, pp. 3; “Similar to previous meta-RL formulations, we assume a distribution of tasks p(T ), where each task is a Markov decision process (MDP), consisting of a set of states, actions, a transition function, and a bounded reward function. We assume that the transition and reward functions are unknown, but can be sampled by taking actions in the environment. Formally, a task
T
=
p
s
0
,
p
s
t
+
1
s
t
,
a
t
,
r
(
s
t
,
a
t
)
consists of an initial state distribution
p
s
0
, transition distribution
p
s
t
+
1
s
t
,
a
t
, and reward function
r
(
s
t
,
a
t
)
. This problem definition encompasses task distributions with varying transition functions (e.g., robots with different dynamics) and varying reward functions (e.g., navigating to different locations). This article discloses a meta-reinforcement learning system that is able to take in environment states and produce an action or policy by using an Actor-Critic model.)
“adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process;” (Algorithm 3 PEARL Meta-Testing, pp. 5; This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model)
Liu and Rakelly fail to explicitly disclose, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.”.
However, Richards discloses, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.” (Model Ensembling as a Proxy for Feedback Offline, pp. 4; "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data
T
j
j
=
1
M
to roughly capture the distribution of
f
(
∙
,
∙
,
w
)
over possible values of the disturbance w. Specifically, we fit a model
f
^
j
(
x
,
u
;
ψ
j
)
with parameters
ψ
j
each trajectory
T
j
, and use this as a proxy for
f
(
x
,
u
,
w
j
)
in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model
f
^
j
is dependent on the trajectory data
T
j
, while
{
r
i
}
i
-
1
N
can be chosen freely. The collection
{
f
^
j
}
i
=
1
N
is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]." This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly and Richards. Liu teaches a system which use multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that is able to take dynamic data and train models to output proper polices. Richards teaches a system using Meta-Reinforcement Learning to train actuators and control systems for stability in arial vehicles. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment, “We further observe in Fig. 4 that, regardless of the control gains, using our features y(
q
,
q
˙
,
θ
y
) in the adaptive controller yields the lowest tracking error. Moreover, using our features with our meta-learned gains yields the lowest tracking error in all but one case." (Richards, Results and Discussion, pp. 8).
Regarding claim 7, Liu discloses, “training, using the embedding neural network, the DRL algorithm by processing context data comprising input-output process data and historical process data from an industrial process run by a process control system (PCS) to generate a multidimensional vector by the embedding neural network wherein the multidimensional vector is lower in dimensions as compared to the processing context data to represent dynamics of the industrial process and a control objective;” (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927; “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models.) and (Proposed A3C-Based SFC Embedding Approach, pp. 1928; “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all of the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data.)
“providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;” (Figure 1, pp. 1927; “As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network.)
“training a meta-reinforcement learning agent (meta-RL agent) to collect a set of parameters from the at least one process controller, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, and wherein the set of parameters are associated with the new industrial process; and” (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929; This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters.)
Liu fails to explicitly disclose, “A process controller, comprising: a data processing system comprising at least one processor and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm and an embedding neural network configured for:” and “using the multidimensional vector as an input to the actor-critic DRL policy to adapt the at least one process controller to a new industrial process,”.
However, Rakelly discloses, “A process controller, comprising: a data processing system comprising at least one processor and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm and an embedding neural network configured for:” (Experimental setup, pp. 6; “We evaluate PEARL on six continuous control environments focused around robotic locomotion, simulated via the MuJoCo simulator (Todorov et al., 2012). These locomotion task families require adaptation across reward functions (walking direction for Half-Cheetah- Fwd-Back, Ant-Fwd-Back, Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal location for Ant-Goal- 2D) or across dynamics (random system parameters for Walker-2D-Params).” The system in this article was used to perform and evaluate robotic movements. This teaches a system which would be connected to a generic computing device to store and execute functions.)
“using the multidimensional vector as an input to the actor-critic DRL policy to adapt the at least one process controller to a new industrial process,” (Algorithm 3 PEARL Meta-Testing, pp. 5; This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model)
Liu and Rakelly fail to explicitly disclose, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.”
However, Richards discloses, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.” (Model Ensembling as a Proxy for Feedback Offline, pp. 4; "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data
T
j
j
=
1
M
to roughly capture the distribution of
f
(
∙
,
∙
,
w
)
over possible values of the disturbance w. Specifically, we fit a model
f
^
j
(
x
,
u
;
ψ
j
)
with parameters
ψ
j
each trajectory
T
j
, and use this as a proxy for
f
(
x
,
u
,
w
j
)
in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model
f
^
j
is dependent on the trajectory data
T
j
, while
{
r
i
}
i
-
1
N
can be chosen freely. The collection
{
f
^
j
}
i
=
1
N
is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]." This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy.)
Regarding claim 13, Richards discloses, “wherein the meta-RL agent is used to tune a proportional integral derivative controller.” (Figure 1, pp. 1; "Our meta-learning is control oriented rather than regression-oriented; specifically, we: 1) collect input-output data on the true system, 2) train a parametric adaptive controller in closed-loop simulation to adapt well to each model of an ensemble constructed from past input-output data, and 3) test our adaptive controller on the real system." This system uses Meta-reinforcement learning to train flight information for nonlinear planar rotorcraft in adverse conditions.)
Regarding claim 14, Richards discloses, “wherein proportional integral tuning is performed in a closed-loop without system identification.” (Figure 1, pp. 1; "Our meta-learning is control oriented rather than regression-oriented; specifically, we: 1) collect input-output data on the true system, 2) train a parametric adaptive controller in closed-loop simulation to adapt well to each model of an ensemble constructed from past input-output data, and 3) test our adaptive controller on the real system." The adaptive controllers can be trained in closed loop simulations.)
Regarding claim 15, Liu discloses, “train, using the embedding neural network, the actor-critic DRL algorithm wherein the embedding neural network processing context data including input-output process data and historical process data from an industrial process to generate via the embedding neural network, a multidimensional vector which is lower in dimensions as compared to the processing context data, and the multidimensional latent vector summarizing the context data to represent dynamics of the industrial process and a control objective,” (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927; “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models.) and (Proposed A3C-Based SFC Embedding Approach, pp. 1928; “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all of the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data.)
“providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;” (Figure 1, pp. 1927; “As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network.)
“train a meta-reinforcement learning agent (meta-RL agent) using a meta-RL training algorithm, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, wherein the meta-RL agent is trained to collect a set of parameters from the at least one process controller, wherein the set of parameters are associated with the new industrial process; and” (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929; This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters.)
Liu fails to explicitly disclose, “A system comprising: one or more processors and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm, and an embedding neural network configured to:” and “adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process, and”.
However, Rakelly discloses, “A system comprising: one or more processors and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm, and an embedding neural network configured to:” (Experimental setup, pp. 6; “We evaluate PEARL on six continuous control environments focused around robotic locomotion, simulated via the MuJoCo simulator (Todorov et al., 2012). These locomotion task families require adaptation across reward functions (walking direction for Half-Cheetah- Fwd-Back, Ant-Fwd-Back, Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal location for Ant-Goal- 2D) or across dynamics (random system parameters for Walker-2D-Params).” The system in this article was used to perform and evaluate robotic movements. This teaches a system which would be connected to a generic computing device to store and execute functions.)
“adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process, and” (Algorithm 3 PEARL Meta-Testing, pp. 5; This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model)
Liu and Rakelly fail to explicitly disclose, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.”
However, Richards discloses, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.” (Model Ensembling as a Proxy for Feedback Offline, pp. 4; "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data
T
j
j
=
1
M
to roughly capture the distribution of
f
(
∙
,
∙
,
w
)
over possible values of the disturbance w. Specifically, we fit a model
f
^
j
(
x
,
u
;
ψ
j
)
with parameters
ψ
j
each trajectory
T
j
, and use this as a proxy for
f
(
x
,
u
,
w
j
)
in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model
f
^
j
is dependent on the trajectory data
T
j
, while
{
r
i
}
i
-
1
N
can be chosen freely. The collection
{
f
^
j
}
i
=
1
N
is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]." This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy.)
Regarding claim 16, Richards discloses, “wherein the meta-RL agent is trained offline across a distribution of simulated processes.” (Introduction, pp. 2; "In this work, we acknowledge this distinction between regression-oriented and control-oriented learning, and propose a control-oriented method to learn a parametric adaptive controller that performs well in closed-loop at test time. Critically, our method (outlined in Fig. 1) focuses on offline learning from past trajectory data. We formalize training the adaptive controller as a semi-supervised, bi-level meta learning problem, with the average integrated tracking error across chosen reference trajectories as the meta-objective." This system utilizes offline data which is from past trajectories to help train the meta learning agent.)
Regarding claim 17, Richards discloses, “wherein the meta-RL agent is configured to produced closed-loop behavior on one or more systems.” (Related work, pp. 2; "In this paper, we focus on offline meta-learning targeting a downstream closed-loop control objective, to train adaptive controllers for nonlinear systems." This system teaches closed-loop training of adaptive controllers for nonlinear systems.)
Regarding claim 18, Richards discloses, “wherein the meta-RL agent is configured to be deployed on novel systems, wherein novel systems are systems without any online training.” (Experiments, pp. 5; "We evaluate our method in simulation on a Planar Fully-Actuated Rotorcraft (PFAR) with degrees of freedom
q
≔
x
,
y
,
ϕ
governed by the nonlinear equations of motion where (x, y) is the position of the center of mass in the inertial frame,¢ is the roll angle, g = (O, 9:81, O) m/s2 is the gravitational acceleration in vector form, R(
ϕ
) is a rotation matrix,
f
e
x
t
is some unknown external force, and u = (
u
1
,
u
2
,
u
3
) are the normalized thrust along the body x-axis, thrust along the body y-axis, and torque about the center of mass, respectively." This system was run in simulation with the above variables defined. This system would be designed to run and execute on PFAR aircraft.) And (Model Ensembling as a Proxy for Feedback Offline, pp. 4; “In practice, we cannot simulate the true dynamics f or sample an actual disturbance trajectory w(t) offline. Instead, we can more reasonably assume we have past data collected with some other, possibly poorly tuned controller. In particular, we make the following assumptions: We have access to trajectory data
T
j
j
=
1
M
, such that [see Equation 10] where
x
k
(
j
)
∈
R
n
and
u
k
(
j
)
∈
R
m
were the state and control input, respectively, at time
t
k
(
j
)
. Moreover,
u
k
(
j
)
was applied in a zero-order hold over
[
t
k
j
,
t
k
+
1
j
)
,
i
.
e
.
,
u
t
=
u
t
k
j
f
o
r
a
l
l
t
∈
[
t
k
(
j
)
,
t
k
+
1
(
j
)
)
along each trajectory
T
j
. During the collection of trajectory data
T
j
, the disturbance w(t) took on a fixed, unknown value
w
j
. The second point is inspired by both meta-learning literature, where it is usually assumed the training data can be segmented according to the latent task, and adaptive control literature, where it is usually assumed that any unknown parameters are constant or slowly time-varying. These assumptions can be generalized to any collection of measured time-state-control transition tuples that can be segmented according to some latent task; in (10) we consider when such tuples can be grouped into trajectories, since this is a natural manner in which data is collected from dynamical systems.” This system recognizes that not all elements are able to be taught offline. To remedy this they are able to simulate true dynamics in the specified way above. This teaches that some aspects are designed to be taught offline.)
Regarding claim 19, Richards discloses, “wherein in a control policy using the meta reinforcement learning agent is performed online.” (Related Work, pp. 2; "Overall, these works concern learning a fixed model-controller pair. Instead, with offline meta-learning, we train an adaptive controller that can update its internal representation of the dynamics on line." This system is designed to adjust controls online while in flight. Some training can be done offline but this would need to execute online for real time flight adjustments.)
Regarding claim 20, Richards discloses, “wherein for each task, a trajectory is collected using a meta-policy.” (Bi-Level Meta-Learning, pp. 3; In meta-learning, we instead have a collection of loss functions
l
i
i
=
1
M
, training datasets
D
i
t
r
a
i
n
i
=
1
M
, and evaluation data sets
D
i
e
v
a
l
i
=
1
M
, where each i corresponds to a task. Moreover, during each task i, we can apply an adaptation mechanism Adapt:
θ
,
D
i
t
r
a
i
n
↦
φ
i
to map so-called meta-parameters
θ
and the task-specific training data
D
i
t
r
a
i
n
to task-specific parameters
φ
i
.” This system teaches that each task i is used to help train the meta-RL algorithm.)
Claims 2-6 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, Rakelly and Richards in view of Bøhn et al, (Bøhn et al, “Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimization”, 2019, hereinafter, “Bøhn”).
Regarding claim 2, Bøhn discloses, “wherein the DRL algorithm comprises a policy network, wherein the policy network is configured for taking a multidimensional vector variables and a current state of the new industrial process as inputs, and outputting a control action configured for the actuators to control the processing equipment.” (Method, pp. 527; “The objective is to control the UAV’s attitude, so a natural choice of controlled variables are the roll, pitch and yaw angles. However, the yaw angle of the aircraft is typically not controlled directly, but through the yaw-rate that depends on the roll angle. In addition, it is desirable to stay close to some nominal airspeed to ensure energy efficient flight, to avoid stall, and to maintain control surface effectiveness which is proportional to airspeed squared. The RL controller is therefore tasked with controlling the roll and pitch angles, φ and θ, and the airspeed Va to desired reference values.” This system uses the DRL network to train different actuators on a UAV. The input information consists of the observation vector, which is disclosed above, which contains information related to feedback and different measurements.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly, Richards and Bøhn. Liu teaches a system which use multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that is able to take dynamic data and train models to output proper polices. Richards teaches a system using Meta-Reinforcement Learning to train actuators and control systems for stability in arial vehicles. Bøhn teaches the use of Deep reinforcement learning to help train actuators and control systems on UAVs. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment with a system that applies reinforcement learning to real world UAV flight controls, “A comparison of the two controllers is shown in Figure 3 on a scenario involving fairly aggressive maneuvers, which both are able to execute. Figure 2 and 3 illustrate an interesting result, the RL controller is able to eliminate steady state errors. While the PID controller has integral action to mitigate steady-state errors, the control law of the RL controller is only a function of the last few states and references. This might suggest that the RL controller has learned some feed-forward action, including nominal inputs in each equilibrium state, thus removing steady-state errors in most cases. Another possibility is that steady-state errors are greatly reduced through the use of high-gain feedback, but the low control variation shown for severe turbulence in Table III indicates that the gain is not excessive. Future work should include integral error states in the observations and evaluate the implications on training and flight performance.” (Bøhn, Evaluation of Controller, pp. 530)
Regarding claim 3, Bøhn discloses, “wherein a policy neural network comprises an actor-neural network, and wherein the training further comprises training the policy neural network using a distribution of different processes or control objective models to determine a latent representation of each process.” (The Proximal Policy Optimization Algorithm, pp. 527; “PPO is a model-free, on-policy, actor-critic, policy-gradient method. It aims to retain the reliable performance of TRPO algorithms, which guarantee monotonic improvements by considering the Kullback-Leibler (KL) divergence of policy updates, while only using first-order optimization. In this section, π is the policy network (that is, the control law) which is optimized wrt.” The system uses the policy network to help improve the current state of the policy.)
Regarding claim 4, Bøhn discloses, “wherein the processing context data further comprises online output data obtained from the PCS, wherein the PCS comprises a physical PCS or a simulated PCS.” (Method, pp. 527; "The policy network takes the state, s, as its input, i.e. the observation vector, and outputs an action, a, consisting of the elevator, aileron and throttle setpoints." The context data used to train the system and while in use obtain data from the physical components of the UAV to help stabilize the flight.)
Regarding claim 5, Bøhn discloses, “wherein the control objective comprises at least one of tracking error, magnitude of input signal, or a change in input signal.” (Evaluation, pp. 529; “To this end, the controllers are evaluated on the following criteria: Success/failure, whether the controller is successful in controlling the state to within some bound of the setpoint. The state must remain within the bounds for at least 100 consecutive time steps to be counted as a success. The bound was chosen to be
±
5°for the roll and pitch angles, and
±
2m/s for the airspeed. … When comparing controllers, the success criterion is the most important, as it is indicative of stability as well as achieving the control objective.” In this system the control objective is based on the correct movements of the system. The control objected is based on the results of the system and stable flight.)
Regarding claim 6, Bøhn discloses, “wherein the multidimensional vector is a user defined parameter that is less than or equal to 5 dimensions.” (Method, pp. 527; "The policy network takes the state, s, as its input, i.e. the observation vector, and outputs an action, a, consisting of the elevator, aileron and throttle set points." The input data here consists of different flight data which is equal to 5 dimensions being the state, observation vector, and outputs of 3 different systems.)
Regarding claim 8, Bøhn discloses, “wherein the training further comprises training the process controller using a distribution of different processes or control objective models to determine a latent representation of each process.” (Training of Controller, pp. 528; "The PPO RL controller was initialized with the default hyperparameters in the OpenAI Baselines implementation [52], and ran with 6 parallel actors. The controller policy is an extended version of the default two hidden layer, 64 nodes multi-layer perceptron (MLP) policy: The observation vector is first processed in a convolutional layer with three filters spanning the time dimension for each component, before being fed to the default policy. This allows the policy to construct functions on the time evolution of the observation vector, while scaling more favorably in parameter count with increasing observation vector size compared to a fully connected input layer." The controller is trained using many difference processes and models. This is used to help learn the process of the controller.)
Regarding claim 9, Bøhn discloses, “wherein the control objective comprises at least one of tracking error, magnitude of input signal, or a change in input signal.” (Evaluation, pp. 529; “To this end, the controllers are evaluated on the following criteria: Success/failure, whether the controller is successful in controlling the state to within some bound of the setpoint. The state must remain within the bounds for at least 100 consecutive time steps to be counted as a success. The bound was chosen to be
±
5°for the roll and pitch angles, and
±
2m/s for the airspeed. … When comparing controllers, the success criterion is the most important, as it is indicative of stability as well as achieving the control objective.” In this system the control objective is based on the correct movements of the system. The control objected is based on the results of the system and stable flight.)
Regarding claim 10, Bøhn discloses, “wherein the DRL algorithm comprises a policy network, wherein the policy network is configured for taking a multidimensional vector variable.§. and a current state of the new industrial process as inputs, and outputting a control action configured for actuators to control processing equipment.” (The Proximal Policy Optimization Algorithm, pp. 527; “PPO is a model-free, on-policy, actor-critic, policy-gradient method. It aims to retain the reliable performance of TRPO algorithms, which guarantee monotonic improvements by considering the Kullback–Leibler (KL) divergence of policy updates, while only using first-order optimization. In this section, π is the policy network (that is, the control law) which is optimized wrt.” The system uses the policy network to help improve the current state of the policy.)
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, Rakelly and Richards in view of Xiao et al, (Xiao et al, “Learning Locomotion Skills via Model-based Proximal Meta-Reinforcement Learning”, 2019, hereinafter “Xiao”).
Regarding claim 11, Xiao discloses, “wherein the meta-RL agent is trained to find a suitable set of parameters using a meta-RL algorithm.” (MB-ProMe Algorithm, pp. 1548; “The model-based RL is converted to a meta-learning problem by meta-learning on an ensemble of dynamics models. The learned dynamics models
f
^
ω
1
:
m
s
t
,
a
t
is embedded in a set of MDPs, i.e.,
ρ
M
=
p
0
,
r
,
S
,
A
,
f
^
ω
1
:
m
s
t
,
a
t
. They share the same state space, action space and a reward function, but hold different beliefs about the true dynamics of an environment.” This model draws different parameters from different dynamic models to help train the meta-agent.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly, Richards and Xiao. Liu teaches a system which use multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that is able to take dynamic data and train models to output proper polices. Richards teaches a system using Meta-Reinforcement Learning to train actuators and control systems for stability in arial vehicles. Xiao teaches the use of met-reinforcement learning to train locomotion skills to a system containing sensors and actuators. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment with a system that is able to use meta-reinforcement learning to train actuators, “The aim of this work is to reduce the sample-complexity and computational cost of reinforcement learning in robot locomotion tasks. Simulation experiments show that the proposed algorithm achieves an asymptotic performance compared with the state-of-the-art model-free reinforcement learning methods with significantly" (Xiao, Introduction, pp. 1545).
Regarding claim 12, Xiao discloses, “wherein the meta-RL agent finds the set of parameters to enable the meta-RL agent to control the new process.” (Dynamics Ensemble, pp. 1546; “The performance of model-based RL largely relies on an accurate learned dynamics model. We use an ensemble of bootstraps
f
^
ω
1
:
m
s
t
,
a
t
to capture the uncertainties of robot dynamics. These models share the same state space S but differ in initialization parameters and training subsets.” This system teaches the use of many different dynamics models to help train the Meta-RL in the MB-ProMe Algorithm.)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL MICHAEL GALVIN-SIEBENALER whose telephone number is (571)272-1257. The examiner can normally be reached Monday - Friday 8AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAUL M GALVIN-SIEBENALER/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights