Prosecution Insights
Last updated: November 04, 2026
Application No. 17/653,175

PROCESS CONTROLLER WITH META-REINFORCEMENT LEARNING

Final Rejection §103
Filed
Mar 02, 2022
Priority
Mar 15, 2021 — provisional 63/161,003
Examiner
GALVIN-SIEBENALER, PAUL MICHAEL
Art Unit
2147
Tech Center
2100 — Computer Architecture & Software
Assignee
Honeywell Limited
OA Round
4 (Final)
27%
Grant Probability
At Risk
5-6
OA Rounds
0m
Est. Remaining
55%
With Interview

Examiner Intelligence

Grants only 27% of cases
27%
Career Allowance Rate
3 granted / 11 resolved
-27.7% vs TC avg
Strong +28% interview lift
Without
With
+27.8%
Interview Lift
resolved cases with interview
Typical timeline
3y 10m
Avg Prosecution
24 currently pending
Career history
61
Total Applications
across all art units

Statute-Specific Performance

§101
30.6%
-9.4% vs TC avg
§103
44.0%
+4.0% vs TC avg
§102
12.7%
-27.3% vs TC avg
§112
12.7%
-27.3% vs TC avg
Black line = Tech Center average estimate • Based on career data from 11 resolved cases

Office Action

§103
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is in response to the amendment filed on Jul. 20th, 2026. The amendments are linked to the original application filed on Mar. 2nd, 2022. Response to Amendment Regarding Claim Rejections – 35 U.S.C. 103 Rejection Applicant Remarks The Applicant asserts the combination of Liu, Rakelly and Richards fail to teach or disclose the independent claims. Specifically, the Applicant asserts that Richards fail to teach the limitation: “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.”. The Applicant provides an argument against Richards in provided remarks. Examiner’s Response The Examiner has reviewed the remarks, prior office actions, as well as the prior art on record and does not find the Applicant’s argument persuasive. The Examiner has reviewed the claims using the Broadest Reasonable Interpretation, BRI, per MPEP 2111. The Examiner has taken the BRI of the limitation, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process.” to recite a process of determining, using a trained model, a meta-RL agent model to tune or train a model with previously collected parameters for a new unique process for a controller in an industrial process. The Examiner points to Richards: (Model Ensembling as a Proxy for Feedback Offline, pp. 4) "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data T j j = 1 M to roughly capture the distribution of f ( ∙ , ∙ , w ) over possible values of the disturbance w. Specifically, we fit a model f ^ j ( x , u ; ψ j ) with parameters ψ j each trajectory T j , and use this as a proxy for f ( x , u , w j ) in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model f ^ j is dependent on the trajectory data T j , while { r i } i - 1 N can be chosen freely. The collection { f ^ j } i = 1 N is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]. To train the parameters ψ j of model f ^ j on the trajectory data T j , we do gradient descent on the one-step prediction problem [See Equation (12)] where μ e n s e m > 0 regularizes ψ j . Since we meta-train θ in (12) to be adaptable to every model in the ensemble, we only need to roughly characterize how the dynamics f ( ∙ , ∙ , w ) vary with the disturbance w, rather than do exact model fitting of f j to T j .” (emphasis added). The section recites using previously collected parameters, ψ j , to be applied to models and these models are separately trained using parameters ψ j as well the dynamics, f ( ∙ , ∙ , w ) , containing disturbance value w. This training process will train the models parameters to generate unique models for process control. Even stated above, the disturbance is added to the models to ensure they do not all overfit the data to the models not generate identical models. Finally, these models in Richards are trained for process control of controllers on a Planar Fully-Artuacted Rotorcraft, which can be applied in industrial settings for surveillance or aerial imagery. The Examiner has reviewed the proposed claims, remarks, prior office actions and prior art. Further, a complete and thorough search was completed, and the Examiner noted no other prior art was located, at this time, to teach the claims as currently recited. However, as stated above, the Examiner asserts that the combination of Liu, Rakelly and Richards teaches the claimed subject matter and a person of ordinary skill in the art would be able to combine the teachings or concepts of these arts to discover the claimed subject matter. As a result, the Examiner has upheld the rejection under 35 USC 103, see 103 rejection below. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1, 7, and 13-20 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al, (Liu et al, "SFC Embedding Meets Machine Learning: Deep Reinforcement Learning Approaches", 2021, hereinafter "Liu") and Rakelly et al, (Rakelly et al, "Efficient Off-Policy Meta- Reinforcement Learning via Probabilistic Context Variables", 2019, hereinafter, "Rakelly") in view of Richards et al, (Richards et al, "Adaptive-Control-Oriented Meta-Learning for Nonlinear Systems", 2019, hereinafter "Richards"). Regarding claim 1, Liu discloses, "training, using an embedding neural network, an actor-critic deep reinforcement learning (DRL) algorithm wherein the embedding neural network processing context data including input-output process data and historical process data from the industrial process to generate an embedding input for the DRL algorithm;" (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927) “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models. "generating, by the embedding neural network, a multidimensional vector based on the training of the DRL, wherein the multidimensional vector is lower in dimensions as compared to the processing context data, summarizing the processing context data to represent dynamics of the industrial process and a control objective," (Proposed A3C-Based SFC Embedding Approach, pp. 1928) “Algorithm 2 describes the proposed A3C based SFC dynamic embedding approach, which mainly consists of two main procedures, i.e., the algorithm initialization step and A3C for embedding SFC. The corresponding parameters of the actor-critic is initialized in line 2, the learning agent performs online learning by actor-critic and asynchronous learner in lines 4-8. Lines 10-12 show that the outputs of the gradient updating are separately added to update the parameters of the global network, which can be copied to each thread. Finally, the actions can be obtained when reaching the stopping condition.” Algorithm 2 line 2 shows the initialization of multidimensional vector. Further, (Cloud-Edge-End-Based SFC Embedding Framework, 1927) “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data. "providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;" (Figure 1, pp. 1927) “As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network. "training a meta-reinforcement learning agent (meta-RL agent) using a meta-RL training algorithm, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, and the meta-RL agent is trained to collect a set of parameters from at least one process controllers, wherein the set of parameters are associated with the new industrial process; and" (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929) This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters. Liu fails to explicitly disclose: "A method of meta-reinforcement learning (MRL) for process control of an industrial process run by a process control system (PCS) including at least one process controller coupled to actuators that is configured for controlling processing equipment, comprising:" "adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process;" "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Rakelly discloses, "A method of meta-reinforcement learning (MRL) for process control of an industrial process run by a process control system (PCS) including at least one process controller coupled to actuators that is configured for controlling processing equipment, comprising:" (Meta-RL Problem Statement, pp. 3) “Similar to previous meta-RL formulations, we assume a distribution of tasks p(T ), where each task is a Markov decision process (MDP), consisting of a set of states, actions, a transition function, and a bounded reward function. We assume that the transition and reward functions are unknown, but can be sampled by taking actions in the environment. Formally, a task T = p s 0 , p s t + 1 s t , a t , r ( s t , a t ) consists of an initial state distribution p s 0 , transition distribution p s t + 1 s t , a t , and reward function r ( s t , a t ) . This problem definition encompasses task distributions with varying transition functions (e.g., robots with different dynamics) and varying reward functions (e.g., navigating to different locations).” This article discloses a meta-reinforcement learning system that can take in environment states and produce an action or policy by using an Actor-Critic model. "adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process;" (Algorithm 3 PEARL Meta-Testing, pp. 5) This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model. Liu and Rakelly fail to explicitly disclose: "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Richards discloses "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." (Model Ensembling as a Proxy for Feedback Offline, pp. 4) "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data T j j = 1 M to roughly capture the distribution of f ( ∙ , ∙ , w ) over possible values of the disturbance w. Specifically, we fit a model f ^ j ( x , u ; ψ j ) with parameters ψ j each trajectory T j , and use this as a proxy for f ( x , u , w j ) in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model f ^ j is dependent on the trajectory data T j , while { r i } i - 1 N can be chosen freely. The collection { f ^ j } i = 1 N is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]. To train the parameters ψ j of model f ^ j on the trajectory data T j , we do gradient descent on the one-step prediction problem [See Equation (12)] where μ e n s e m > 0 regularizes ψ j . Since we meta-train θ in (12) to be adaptable to every model in the ensemble, we only need to roughly characterize how the dynamics f ( ∙ , ∙ , w ) vary with the disturbance w, rather than do exact model fitting of f j to T j .” This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy. As stated above in the remarks, this teaches a process of updating model parameters to produce an ensemble of process control models which are unique from one another using collected parameters. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly and Richards. Liu teaches a system which use multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that can take dynamic data and train models to output proper polices. Richards teaches a system using Meta- Reinforcement Learning to train actuators and control systems for stability in arial vehicles. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment, "We further observe in Fig. 4 that, regardless of the control gains, using our features y ( q , q ˙ , θ y ) in the adaptive controller yields the lowest tracking error. Moreover, using our features with our meta-learned gains yields the lowest tracking error in all but one case." (Richards, Results and Discussion, pp. 8). Regarding claim 7, Liu discloses, "training, using the embedding neural network, the DRL algorithm by processing context data comprising input-output process data and historical process data from an industrial process run by a process control system (PCS) to generate a multidimensional vector by the embedding neural network wherein the multidimensional vector is lower in dimensions as compared to the processing context data to represent dynamics of the industrial process and a control objective;" (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927) “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models. Further, (Proposed A3C-Based SFC Embedding Approach, pp. 1928) “Algorithm 2 describes the proposed A3C based SFC dynamic embedding approach, which mainly consists of two main procedures, i.e., the algorithm initialization step and A3C for embedding SFC. The corresponding parameters of the actor-critic is initialized in line 2, the learning agent performs online learning by actor-critic and asynchronous learner in lines 4-8. Lines 10-12 show that the outputs of the gradient updating are separately added to update the parameters of the global network, which can be copied to each thread. Finally, the actions can be obtained when reaching the stopping condition.” Algorithm 2 line 2 shows the initialization of multidimensional vector. Further, (Cloud-Edge-End-Based SFC Embedding Framework, 1927) “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data. "providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;" (Figure 1, pp. 1927) “As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network. "training a meta-reinforcement learning agent (meta-RL agent) to collect a set of parameters from the at least one process controller, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, and wherein the set of parameters are associated with the new industrial process; and" (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929) This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters. Liu fails to explicitly disclose: "A process controller, comprising: a data processing system comprising at least one processor and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm and an embedding neural network configured for:" "using the multidimensional vector as an input to the actor-critic DRL policy to adapt, the at least one process controller to a new industrial process," "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Rakelly discloses, "A process controller, comprising: a data processing system comprising at least one processor and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm and an embedding neural network configured for:" (Experimental setup, pp. 6) “We evaluate PEARL on six continuous control environments focused around robotic locomotion, simulated via the MuJoCo simulator (Todorov et al., 2012). These locomotion task families require adaptation across reward functions (walking direction for Half-Cheetah- Fwd-Back, Ant-Fwd-Back, Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal location for Ant-Goal- 2D) or across dynamics (random system parameters for Walker-2D-Params).” The system in this article was used to perform and evaluate robotic movements. This teaches a system which would be connected to a generic computing device to store and execute functions. "using the multidimensional vector as an input to the actor-critic DRL policy to adapt, the at least one process controller to a new industrial process," (Algorithm 3 PEARL Meta-Testing, pp. 5) This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model. Liu and Rakelly fail to explicitly disclose: "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Richards discloses, "determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." (Model Ensembling as a Proxy for Feedback Offline, pp. 4) "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data T j j = 1 M to roughly capture the distribution of f ( ∙ , ∙ , w ) over possible values of the disturbance w. Specifically, we fit a model f ^ j ( x , u ; ψ j ) with parameters ψ j each trajectory T j , and use this as a proxy for f ( x , u , w j ) in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model f ^ j is dependent on the trajectory data T j , while { r i } i - 1 N can be chosen freely. The collection { f ^ j } i = 1 N is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]. To train the parameters ψ j of model f ^ j on the trajectory data T j , we do gradient descent on the one-step prediction problem [See Equation (12)] where μ e n s e m > 0 regularizes ψ j . Since we meta-train θ in (12) to be adaptable to every model in the ensemble, we only need to roughly characterize how the dynamics f ( ∙ , ∙ , w ) vary with the disturbance w, rather than do exact model fitting of f j to T j .” This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy. As stated above in the remarks, this teaches a process of updating model parameters to produce an ensemble of process control models which are unique from one another using collected parameters. Regarding claim 13, Richards discloses, "wherein the meta-RL agent is used to tune a proportional integral derivative controller." (Figure 1, pp. 1) "Our meta-learning is control oriented rather than regression-oriented; specifically, we: 1) collect input-output data on the true system, 2) train a parametric adaptive controller in closed-loop simulation to adapt well to each model of an ensemble constructed from past input-output data, and 3) test our adaptive controller on the real system." This system uses Meta-reinforcement learning to train flight information for nonlinear planar rotorcraft in adverse conditions. Regarding claim 14, Richards discloses, "wherein proportional integral tuning is performed in a closed-loop without system identification." (Figure 1, pp. 1) "Our meta-learning is control oriented rather than regression-oriented; specifically, we: 1) collect input-output data on the true system, 2) train a parametric adaptive controller in closed-loop simulation to adapt well to each model of an ensemble constructed from past input-output data, and 3) test our adaptive controller on the real system." The adaptive controllers can be trained in closed loop simulations. Regarding claim 15, Liu discloses, "train, using the embedding neural network, the actor-critic DRL algorithm wherein the embedding neural network processing context data including input-output process data and historical process data from an industrial process to generate via the embedding neural network, a multidimensional vector which is lower in dimensions as compared to the processing context data, and the multidimensional latent vector summarizing the context data to represent dynamics of the industrial process and a control objective," (Cloud-Edge-End-Based SFC Embedding Network, pp. 1927) “The network state can be input to train the network model, and we use the DRL-based schemes to train the network model. The DRL agent action including the VNF embedding and the flow scheduling can be output to NFV-Orchestration (NFVO) and SDN-Controller (SDNC) in the edge layer for the SFC resource allocation.” The model in this article used multiple different models to embed information from an environment and further process the embeddings using an actor-critic model. The embedding process is handed as a service request and is denoted as input to train the actor critic models. Further, (Proposed A3C-Based SFC Embedding Approach, pp. 1928) “Algorithm 2 describes the proposed A3C based SFC dynamic embedding approach, which mainly consists of two main procedures, i.e., the algorithm initialization step and A3C for embedding SFC. The corresponding parameters of the actor-critic is initialized in line 2, the learning agent performs online learning by actor-critic and asynchronous learner in lines 4-8. Lines 10-12 show that the outputs of the gradient updating are separately added to update the parameters of the global network, which can be copied to each thread. Finally, the actions can be obtained when reaching the stopping condition.” Algorithm 2 line 2 shows the initialization of multidimensional vector. Further, (Cloud-Edge-End-Based SFC Embedding Framework, 1927) “For the deployment of the proposed DRL schemes, the DRL-driven agent can be run in the core cloud datacenter. The key of the cloud layer is the DRL-driven agent and therefore it requires adequate data for training. In the cloud layer, the agent can dynamically train the network model and calculate the SFC embedding strategy. … In the SFC embedding process, an agent allocates one VNF at each time step, and the embedding process ends after the execution of the final VNF of the requested service chain. In each valid decision, one VNF is embedded in a server node; then the current state is transferred to the next.” This model will embed data from the network into a data structure that is smaller than the sum of all the collected data. This would teach the use of a vector which is compressed to fit the important historical data and network data. "providing the multidimensional vector as an input to an actor network and a critic network of the DRL algorithm, wherein the actor network is configured to output one or more control actions for the at least one process controller and the critic network is configured to evaluate a value function associated with the control actions;" (Figure 1, pp. 1927) As seen in the figure the embeddings are fed into the Actor-Critic modes to produce an action. These actions are considered the output and are applied to the environment. This change is then evaluated by the critic network to produce a reward for the actor network. "train a meta-reinforcement learning agent (meta-RL agent) using a meta-RL training algorithm, wherein the meta-RL agent comprises the embedding neural network, the actor network, and the critic network, wherein the meta-RL agent is trained to collect a set of parameters from the at least one process controller, wherein the set of parameters are associated with the new industrial process; and" (Algorithm 2 A3C Based Approach for SFC-DMP, pp. 1929) This algorithm discloses the use of multiple networks to train the model. This model contains an embedding, actor and critic models. These models are trained using the generated embeddings. This process is used to update policies of the network they are connected to. Lines 9-11 disclose the training process of updating parameters. Liu fails to explicitly disclose: "A system comprising: one or more processors and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm, and an embedding neural network configured to:" "adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process, and" “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Rakelly discloses, "A system comprising: one or more processors and a memory that stores an actor-critic deep reinforcement learning (DRL) algorithm, and an embedding neural network configured to:" (Experimental setup, pp. 6; “We evaluate PEARL on six continuous control environments focused around robotic locomotion, simulated via the MuJoCo simulator (Todorov et al., 2012). These locomotion task families require adaptation across reward functions (walking direction for Half-Cheetah- Fwd-Back, Ant-Fwd-Back, Humanoid-Direc-2D, target velocity for Half-Cheetah-Vel, and goal location for Ant-Goal- 2D) or across dynamics (random system parameters for Walker-2D-Params).” The system in this article was used to perform and evaluate robotic movements. This teaches a system which would be connected to a generic computing device to store and execute functions. "adapting, using the multidimensional vector and the actor network the at least one process controller to a new industrial process, and" (Algorithm 3 PEARL Meta-Testing, pp. 5) This algorithm discloses a testing process for the model. After training the model, it will take in a set of tasks and evaluate them. This will then produce a new policy based on the output of the model. Liu and Rakelly fail to explicitly disclose: “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." However, Richards discloses, “determining, using the trained meta-RL agent, a meta-RL process model to tune the collected set of parameters for a new process controller that is different from the at least one process controller from which the set of parameters are collected, wherein the new process controller is configured to control the new industrial process." (Model Ensembling as a Proxy for Feedback Offline, pp. 4) "Inspired by Clavera et al. [20], since we cannot simulate the true dynamics f offline, we propose to first train a model ensemble from the trajectory data T j j = 1 M to roughly capture the distribution of f ( ∙ , ∙ , w ) over possible values of the disturbance w. Specifically, we fit a model f ^ j ( x , u ; ψ j ) with parameters ψ j each trajectory T j , and use this as a proxy for f ( x , u , w j ) in (9). The meta-problem (9) is now [see Equation 11] This form is still semi-supervised, since each model f ^ j is dependent on the trajectory data T j , while { r i } i - 1 N can be chosen freely. The collection { f ^ j } i = 1 N is termed a model ensemble. Empirically, the use of model ensembles has been shown to improve robustness to model bias and train-test data shift of deep predictive models [40] and policies in reinforcement learning [58, 39, 20]. To train the parameters ψ j of model f ^ j on the trajectory data T j , we do gradient descent on the one-step prediction problem [See Equation (12)] where μ e n s e m > 0 regularizes ψ j . Since we meta-train θ in (12) to be adaptable to every model in the ensemble, we only need to roughly characterize how the dynamics f ( ∙ , ∙ , w ) vary with the disturbance w, rather than do exact model fitting of f j to T j .” This model uses a form of supervised learning. This process is designed to help fine tune a model's output. This teaches a form of structured training to improve the model's accuracy. As stated above in the remarks, this teaches a process of updating model parameters to produce an ensemble of process control models which are unique from one another using collected parameters. Regarding claim 16, Richards discloses, "wherein the meta-RL agent is trained offline across a distribution of simulated processes." (Introduction, pp. 2) "In this work, we acknowledge this distinction between regression-oriented and control-oriented learning, and propose a control-oriented method to learn a parametric adaptive controller that performs well in closed-loop at test time. Critically, our method (outlined in Fig. 1) focuses on offline learning from past trajectory data. We formalize training the adaptive controller as a semi-supervised, bi-level meta learning problem, with the average integrated tracking error across chosen reference trajectories as the meta-objective." This system utilizes offline data which is from past trajectories to help train the meta learning agent. Regarding claim 17, Richards discloses, "wherein the meta-RL agent is configured to produced closed-loop behavior on one or more systems." (Related work, pp. 2) "In this paper, we focus on offline meta-learning targeting a downstream closed-loop control objective, to train adaptive controllers for nonlinear systems." This system teaches closed-loop training of adaptive controllers for nonlinear systems. Regarding claim 18, Richards discloses, "wherein the meta-RL agent is configured to be deployed on novel systems, wherein novel systems are systems without any online training." (Experiments, pp. 5) "We evaluate our method in simulation on a Planar Fully-Actuated Rotorcraft (PFAR) with degrees of freedom q ≔ x , y , ϕ governed by the nonlinear equations of motion where (x, y) is the position of the center of mass in the inertial frame,¢ is the roll angle, g = (O, 9:81, O) m/s2 is the gravitational acceleration in vector form, R( ϕ ) is a rotation matrix, f e x t is some unknown external force, and u = ( u 1 , u 2 , u 3 ) are the normalized thrust along the body x-axis, thrust along the body y-axis, and torque about the center of mass, respectively." This system was run in simulation with the above variables defined. This system would be designed to run and execute on PFAR aircraft. Further, (Model Ensembling as a Proxy for Feedback Offline, pp. 4; “In practice, we cannot simulate the true dynamics f or sample an actual disturbance trajectory w(t) offline. Instead, we can more reasonably assume we have past data collected with some other, possibly poorly tuned controller. In particular, we make the following assumptions: We have access to trajectory data T j j = 1 M , such that [see Equation 10] where x k ( j ) ∈ R n and u k ( j ) ∈ R m were the state and control input, respectively, at time t k ( j ) . Moreover, u k ( j ) was applied in a zero-order hold over [ t k j , t k + 1 j ) ,   i . e . ,   u t = u t k j f o r   a l l   t ∈ [ t k ( j ) , t k + 1 ( j ) ) along each trajectory T j . During the collection of trajectory data T j , the disturbance w(t) took on a fixed, unknown value w j . The second point is inspired by both meta-learning literature, where it is usually assumed the training data can be segmented according to the latent task, and adaptive control literature, where it is usually assumed that any unknown parameters are constant or slowly time-varying. These assumptions can be generalized to any collection of measured time-state-control transition tuples that can be segmented according to some latent task; in (10) we consider when such tuples can be grouped into trajectories, since this is a natural manner in which data is collected from dynamical systems.” This system recognizes that not all elements are able to be taught offline. To remedy this, they are able to simulate true dynamics in the specified way above. This teaches that some aspects are designed to be taught offline. Regarding claim 19, Richards discloses, "wherein in a control policy using the meta-reinforcement learning agent is performed online." (Related Work, pp. 2) "Overall, these works concern learning a fixed model-controller pair. Instead, with offline meta-learning, we train an adaptive controller that can update its internal representation of the dynamics on line." This system is designed to adjust controls online while in flight. Some training can be done offline, but this would need to be executed online for real time flight adjustments. Regarding claim 20, Richards discloses, "wherein for each task, a trajectory is collected using a meta-policy." (Bi-Level Meta-Learning, pp. 3) In meta-learning, we instead have a collection of loss functions l i i = 1 M , training datasets D i t r a i n i = 1 M , and evaluation data sets D i e v a l i = 1 M , where each i corresponds to a task. Moreover, during each task i, we can apply an adaptation mechanism Adapt: θ , D i t r a i n ↦ φ i to map so-called meta-parameters θ and the task-specific training data D i t r a i n to task-specific parameters φ i .” This system teaches that each task i is used to help train the meta-RL algorithm. Claims 2-6 and 8-10 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, Rakelly, and Richards in view of Bøhn et al, (Bøhn et al, "Deep Reinforcement Learning Attitude Control of Fixed-Wing UAVs Using Proximal Policy Optimization", 2019, hereinafter, "Bøhn"). Regarding claim 2, Bøhn discloses, "wherein the DRL algorithm comprises a policy network, wherein the policy network is configured for taking a multidimensional vector variables and a current state of the new industrial process as inputs, and outputting a control action configured for the actuators to control the processing equipment." (Method, pp. 527) “The objective is to control the UAV’s attitude, so a natural choice of controlled variables are the roll, pitch and yaw angles. However, the yaw angle of the aircraft is typically not controlled directly, but through the yaw-rate that depends on the roll angle. In addition, it is desirable to stay close to some nominal airspeed to ensure energy efficient flight, to avoid stall, and to maintain control surface effectiveness which is proportional to airspeed squared. The RL controller is therefore tasked with controlling the roll and pitch angles, φ and θ, and the airspeed Va to desired reference values.” This system uses the DRL network to train different actuators on a UAV. The input information consists of the observation vector, which is disclosed above, which contains information related to feedback and different measurements. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly, Richards and Bøhn. Liu teaches a system which use multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that is able to take dynamic data and train models to output proper polices. Richards teaches a system using Meta- Reinforcement Learning to train actuators and control systems for stability in arial vehicles. Bøhn teaches the use of Deep reinforcement learning to help train actuators and control systems on UAVs. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment with a system that applies reinforcement learning to real world UAV flight controls, "A comparison of the two controllers is shown in Figure 3 on a scenario involving fairly aggressive maneuvers, which both are able to execute. Figure 2 and 3 illustrate an interesting result, the RL controller is able to eliminate steady state errors. While the PID controller has integral action to mitigate steady state errors, the control law of the RL controller is only a function of the last few states and references. This might suggest that the RL controller has learned some feed-forward action, including nominal inputs in each equilibrium state, thus removing steady-state errors in most cases. Another possibility is that steady-state errors are greatly reduced through the use of high-gain feedback, but the low control variation shown for severe turbulence in Table Ill indicates that the gain is not excessive. Future work should include integral error states in the observations and evaluate the implications on training and flight performance." (Bøhn, Evaluation of Controller, pp. 530) Regarding claim 3, Bøhn discloses, "wherein a policy neural network comprises an actor-neural network, and wherein the training further comprises training the policy neural network using a distribution of different processes or control objective models to determine a latent representation of each process." (The Proximal Policy Optimization Algorithm, pp. 527) “PPO is a model-free, on-policy, actor-critic, policy-gradient method. It aims to retain the reliable performance of TRPO algorithms, which guarantee monotonic improvements by considering the Kullback-Leibler (KL) divergence of policy updates, while only using first-order optimization. In this section, π is the policy network (that is, the control law) which is optimized wrt.” The system uses the policy network to help improve the current state of the policy. Regarding claim 4, Bøhn discloses, "wherein the processing context data further comprises online output data obtained from the PCS, wherein the PCS comprises a physical PCS or a simulated PCS." (Method, pp. 527) "The policy network takes the state, s, as its input, i.e. the observation vector, and outputs an action, a, consisting of the elevator, aileron and throttle setpoints." The context data is used to train the system and while in use obtain data from the physical components of the UAV to help stabilize the flight. Regarding claim 5, Bøhn discloses, "wherein the control objective comprises at least one of tracking error, magnitude of input signal, or a change in input signal." (Evaluation, pp. 529) “To this end, the controllers are evaluated on the following criteria: Success/failure, whether the controller is successful in controlling the state to within some bound of the setpoint. The state must remain within the bounds for at least 100 consecutive time steps to be counted as a success. The bound was chosen to be ± 5°for the roll and pitch angles, and ± 2m/s for the airspeed. … When comparing controllers, the success criterion is the most important, as it is indicative of stability as well as achieving the control objective.” In this system the control objective is based on the correct movements of the system. The control objected is based on the results of the system and stable flight. Regarding claim 6, Bøhn discloses, "wherein the multidimensional vector is a user defined parameter that is less than or equal to 5 dimensions." (Method, pp. 527) "The policy network takes the state, s, as its input, i.e. the observation vector, and outputs an action, a, consisting of the elevator, aileron and throttle set points." The input data here consists of different flight data which is equal to 5 dimensions being the state, observation vector, and outputs of 3 different systems. Regarding claim 8, Bøhn discloses, "wherein the training further comprises training the process controller using a distribution of different processes or control objective models to determine a latent representation of each process." (Training of Controller, pp. 528) "The PPO RL controller was initialized with the default hyperparameters in the OpenAI Baselines implementation [52], and ran with 6 parallel actors. The controller policy is an extended version of the default two hidden layer, 64 nodes multi-layer perceptron (MLP) policy: The observation vector is first processed in a convolutional layer with three filters spanning the time dimension for each component, before being fed to the default policy. This allows the policy to construct functions on the time evolution of the observation vector, while scaling more favorably in parameter count with increasing observation vector size compared to a fully connected input layer." The controller is trained using many different processes and models. This is used to help learn the process of the controller. Regarding claim 9, Bøhn discloses, "wherein the control objective comprises at least one of tracking error, magnitude of input signal, or a change in input signal." (Evaluation, pp. 529) “To this end, the controllers are evaluated on the following criteria: Success/failure, whether the controller is successful in controlling the state to within some bound of the setpoint. The state must remain within the bounds for at least 100 consecutive time steps to be counted as a success. The bound was chosen to be ± 5°for the roll and pitch angles, and ± 2m/s for the airspeed. … When comparing controllers, the success criterion is the most important, as it is indicative of stability as well as achieving the control objective.” In this system the control objective is based on the correct movements of the system. The control objected is based on the results of the system and stable flight. Regarding claim 10, Bøhn discloses, "wherein the DRL algorithm comprises a policy network, wherein the policy network is configured for taking a multidimensional vector variables and a current state of the new industrial process as inputs, and outputting a control action configured for actuators to control processing equipment." (The Proximal Policy Optimization Algorithm, pp. 527) “PPO is a model-free, on-policy, actor-critic, policy-gradient method. It aims to retain the reliable performance of TRPO algorithms, which guarantee monotonic improvements by considering the Kullback–Leibler (KL) divergence of policy updates, while only using first-order optimization. In this section, π is the policy network (that is, the control law) which is optimized wrt.” The system uses the policy network to help improve the current state of the policy. Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Liu, Rakelly and Richards in view of Xiao et al, (Xiao et al, "Learning Locomotion Skills via Model-based Proximal Meta-Reinforcement Learning", 2019, hereinafter "Xiao"). Regarding claim 11, Xiao discloses, "wherein the meta-RL agent is trained to find a suitable set of parameters using a meta-RL algorithm." (MB-ProMe Algorithm, pp. 1548) “The model-based RL is converted to a meta-learning problem by meta-learning on an ensemble of dynamics models. The learned dynamics models f ^ ω 1 : m s t , a t is embedded in a set of MDPs, i.e., ρ M = p 0 , r ,   S ,   A , f ^ ω 1 : m s t , a t . They share the same state space, action space and a reward function, but hold different beliefs about the true dynamics of an environment.” This model draws different parameters from different dynamic models to help train the meta-agent. It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Liu, Rakelly, Richards and Xiao. Liu teaches a system which uses multiple networks to embed data and use an Actor-Critic models to evaluate network services. Rakelly teaches a meta-reinforcement learning system that can take dynamic data and train models to output proper polices. Richards teaches a system using Meta- Reinforcement Learning to train actuators and control systems for stability in arial vehicles. Xiao teaches the use of meta-reinforcement learning to train locomotion skills to a system containing sensors and actuators. One of ordinary skill would have motivation to combine a system which uses an embedding network with a system that uses actor-critic models with a system that uses meta learning and multiple models to execute actions in an environment with a system that is able to use meta-reinforcement learning to train actuators, "The aim of this work is to reduce the sample-complexity and computational cost of reinforcement learning in robot locomotion tasks. Simulation experiments show that the proposed algorithm achieves an asymptotic performance compared with the state-of-the-art model-free reinforcement learning methods with significantly" (Xiao, Introduction, pp. 1545). Regarding claim 12, Xiao discloses, "wherein the meta-RL agent finds the set of parameters to enable the meta-RL agent to control the new process." (Dynamics Ensemble, pp. 1546) “The performance of model-based RL largely relies on an accurate learned dynamics model. We use an ensemble of bootstraps f ^ ω 1 : m s t ,   a t to capture the uncertainties of robot dynamics. These models share the same state space S but differ in initialization parameters and training subsets.” This system teaches the use of many different dynamics models to help train the Meta-RL in the MB-ProMe Algorithm. Conclusion THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL MICHAEL GALVIN-SIEBENALER whose telephone number is (571)272-1257. The examiner can normally be reached Monday - Friday 8AM to 5PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /PAUL M GALVIN-SIEBENALER/Examiner, Art Unit 2147 /ERIC NILSSON/Primary Examiner, Art Unit 2151
Read full office action

Prosecution Timeline

Show 1 earlier event
May 20, 2025
Non-Final Rejection mailed — §103
Aug 13, 2025
Response Filed
Oct 31, 2025
Final Rejection mailed — §103
Feb 02, 2026
Request for Continued Examination
Feb 09, 2026
Response after Non-Final Action
May 05, 2026
Non-Final Rejection mailed — §103
Jul 20, 2026
Response Filed
Sep 25, 2026
Final Rejection mailed — §103 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12743624
METHOD AND APPARATUS FOR TRAINING NEURAL NETWORK MODEL
4y 6m to grant Granted Sep 22, 2026
Patent 12694262
ALMOST-INDIRECT CONVOLUTION IN QUANTIZED NEURAL NETWORKS
4y 9m to grant Granted Jul 28, 2026
Study what changed to get past this examiner. Based on 2 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

5-6
Expected OA Rounds
27%
Grant Probability
55%
With Interview (+27.8%)
3y 10m (~0m remaining)
Median Time to Grant
High
PTA Risk
Based on 11 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month