Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 27 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because claim 27 is directed to “a computer storage medium”. The BRI of this term encompasses non-statutory subject matter (e.g., signals per se). While the specification states this term may be non-transitory it does not limit the scope of the term to only non-transitory embodiments. Examiner recommends the term “non-transitory” included in the claim language.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-18 and 26-27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ionescu et al. (from IDS: US Pub. No. 2020/0104645, published April 2020, hereinafter “Ionescu”) in view of Dalli et al. (from IDS: US Pub. No. 2022/0147876, published 12 May 2022, hereinafter “Dalli”).
Regarding claim 1, Ionescu teaches a computer-implemented method of training a machine learning system configured to monitor the control of an agent in an environment performing a task (Ionescu, [0002] and [0005] – teaches training a machine learning system configured to monitor control of an agent in an environment performing a task (agent interacting with an environment performing actions selected by reinforcement learning system, reinforcement learning system can learn to perform complex tasks)), wherein the machine learning system is configured to, for each of a succession of time steps:
obtain an observation characterizing a state of the environment for a current time step (Ionescu, [0055] – “The reinforcement learning neural network system 100 selects actions 104 to be performed by an agent 102 interacting with an environment 110 at each of multiple time steps. In order for the agent 102 to interact with the environment 110, the system 100 receives observation data 106 characterizing an observation of a current state of the environment and provides action data for selecting an action 104 to be performed by the agent 102 in response to the received observation data 106. ”);
process an internal state of the machine learning system at the current time step using an action selection subsystem of the machine learning system to generate an action selection policy output, wherein the internal state of the machine learning system at the current time step depends on the observation for the current time step (Ionescu, [0055] – teaches processing an internal state of the machine learning system (observation data characterizing an observation of current state of the environment) at the current time step (actions performed by agents at each of multiple time steps, thus teaching a current time step for current state) using an action selection subsystem to generate an action selection policy output (system receives observation data and provides action data, thus an action selection policy, for selecting an action 104), wherein the internal state of the machine learning system at the current time step depends on the observation for the current time step (observation data characterizing an observation of current state of the environment, thus the internal state, or current state, depends on the observation for the current time step));
process the internal state of the machine learning system at the current time step using a decoder neural network to generate a decoder output for the current time step, wherein the decoder output for the current time step describes the internal state of the machine learning system at the current time step (Ionescu, Fig. 1, [0045] – “Some implementations of the reinforcement learning neural network system can also provide human-interpretable representations of the internal basis for action selection policies used by the system. These may be presented to a user separately or in combination with representations of observations and/or actions of the system, to facilitate the user understanding the basis for the actions. For example a spatial map or mask dependent upon a collection of concrete and more abstract features may be presented as a visual map.”, and in [0059] – “The reinforcement learning neural network system 100 comprises a visual entity identifying neural network subsystem 120 coupled to a reinforcement learning neural network subsystem 130. The visual entity identifying neural network subsystem 120 is configured to process an images of the environment to observation data determine a set of spatial maps 122… A visual entity corresponds to a spatial and/or temporal grouping of image pixels. These tend to correlate with human-interpretable entities in the environment such as walls, floor, manipulable objects, background, and the like. They also tend to carry semantic meaning for the reinforcement learning task.” – teaches processing the internal state of the machine learning system at the current time step using a decoder neural network (as shown in Fig. 1, observation data is provided to visual entity identifying neural network subsystem) to generate a decoder output for the current time step, wherein the decoder output for the current time step describes the internal state of the machine learning system at the current time step (visual entity identifying network determines spatial maps which are human-interpretable representations of internal states of the system, the maps carry semantic meaning for the system, and facilitate user understanding, thus the visual entity identifying network is a decoder neural network that generates a decoder output that describes the internal state of the machine learning system));
select an action to be performed by the agent at the current time step in response to the observation using the action selection policy output (Ionescu, [0055] – teaches select an action to be performed by the agent at the current time step in response to the observation using the action selection policy output (system receives observation data and provides action data, thus an action selection policy, for selecting an action 104 to be performed by an agent interacting within an environment at each of multiple time steps));
cause the agent to perform the selected action (Ionescu, [0055] - teaches cause the agent to performed the selected action (system provides action data for selecting an action to be performed by an agent, thus causing the agent to perform the selected action)); and
the method comprising training the machine learning system (Ionescu, [0061] – “A training engine 140 controls training of the reinforcement learning neural network system 100 according to objective functions described later.” – teaches training the machine learning system) by:
determining the internal state of the machine learning system at a next time step by processing, using an internal state updating subsystem, a combination of the decoder output for the current time step, the observation for the next time step, and a counterfactual internal state, wherein the counterfactual internal state is different to the internal state of the machine learning system at the current time step (Ionescu, Fig. 6 and in [0083] – “FIG. 6 shows a flow diagram of a process for training the reinforcement learning neural network system 100 of FIGS. 1-4. Observation data comprising an image of the environment is received by the system (step 600). Based on this, the Q-value neural network subsystem 330 determines a value for Q.sup.meta and the meta-controller 310 uses Q.sup.meta, e.g. with an epsilon-greedy policy, to determine a meta-action to select either an option or a series of individual tasks (step 602).” – teaches determining the internal state of the machine learning system at a next time step (as in Fig. 6, determines data characterizing transition to update parameters of system, thus determining an internal state at a next time step), by processing, using an internal state updating subsystem (training engine 140, as stated in Ionescu at [0061]), a combination of the decoder output for the current time step (as in Fig. 6 step 604, determines spatial maps from current observation, thus the decoder output for the current time step), the observation for the next time step (as in Fig. 6 step 610, receives observation data for a next state), and a counterfactual internal state wherein the counterfactual internal state is different to the internal state of the machine learning system at the current time step (the counterfactual internal state being the options of Qmeta, wherein the options of Qmeta are different from the internal state at the current time step)).
Ionescu fails to explicitly teach provide a signal derived from the decoder output for the current time step as a monitoring signal for monitoring the control of the agent in the environment;
However, analogous to the field of the claimed invention, Dalli teaches:
provide a signal derived from the [[a]] decoder output for the current time step as a monitoring signal for monitoring the control of the agent in the environment (Dalli, [0042], [0076], [0102] – “Explanations in XRL agents may be seen as performing two main functions: (i.) explaining the actions and decisions of the agent, in an audit-like manner; (ii.) using the explanation model itself as a form of predictive function that acts using the explanatory model space together with the observed space… This use may be enhanced further with the XRL agent use of causal models and simulator constraint models, such as in our XRL XGAIL model, where explanatory models may be used to implement plausibility based deconfounding. The addition of plausibility-based objectives in XRL agents may make XRL agents more suitable for use in exploration, observation, and interventional interactions with realistic environments such as the real physical world.”, and in [0200] – “In another exemplary application used for illustrative purposes, a typical XRL agent such as the one illustrated in FIG. 3 may utilize control engineering models, such as Nyquist analysis, Bode plots, and other forms of analyses to process a wide variety of processing signals.” – teaches providing a signal (agents may utilize analyses to process wide variety of processing signals) derived from a decoder output (explanations in agents) for the current time step as a monitoring signal for monitoring control of the agent in the environment (agents utilize analyses to process signals and explanations that make agents more suitable for use in exploration, observation, and interventional actions, thus providing a signal derived from the decoder output as a monitoring signal for monitoring control of the agent in the environment));
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the monitoring signal of Dalli to the decoder output, training methods, and machine learning system of Ionescu. Doing so would incorporate explanations as part of state/action spaces of an RL agent and thus provide richer exploratory space for the agent (Dalli, [0042]), and provide fully traceable explanations at all steps taken by the RL agent (Dalli, [0102]).
Claims 26 and 27 incorporate substantively all the limitations of claim 1 in a system and a computer storage medium, and are rejected on similar grounds as above. Ionescu teaches the computers, storage devices, and mediums of these claims at [0086]-[0087].
Regarding claim 2, the combination of Ionescu and Dalli teaches the method of claim 1, wherein training the machine learning system further comprises, for each of the succession of time steps:
obtaining the observation characterizing the state of the environment for the current time step (Ionescu, [0055] – “The reinforcement learning neural network system 100 selects actions 104 to be performed by an agent 102 interacting with an environment 110 at each of multiple time steps. In order for the agent 102 to interact with the environment 110, the system 100 receives observation data 106 characterizing an observation of a current state of the environment and provides action data for selecting an action 104 to be performed by the agent 102 in response to the received observation data 106. ” – teaches obtaining the observation characterizing the state of the environment for the current time step);
processing the internal state of the machine learning system at the current time step using the action selection subsystem of the machine learning system to generate the action selection policy output (Ionescu, [0055] – teaches processing the internal state of the machine learning system (observation data characterizing an observation of current state of the environment) at the current time step (actions performed by agents at each of multiple time steps, thus teaching a current time step for current state) using the action selection subsystem to generate an action selection policy output (system receives observation data and provides action data, thus an action selection policy, for selecting an action 104));
processing the internal state of the machine learning system at the current time step using the decoder neural network to generate the decoder output for the current time step (Ionescu, Fig. 1, [0045], and in [0059] – “The reinforcement learning neural network system 100 comprises a visual entity identifying neural network subsystem 120 coupled to a reinforcement learning neural network subsystem 130. The visual entity identifying neural network subsystem 120 is configured to process an images of the environment to observation data determine a set of spatial maps 122… A visual entity corresponds to a spatial and/or temporal grouping of image pixels. These tend to correlate with human-interpretable entities in the environment such as walls, floor, manipulable objects, background, and the like. They also tend to carry semantic meaning for the reinforcement learning task.” – teaches processing the internal state of the machine learning system at the current time step using a decoder neural network (as shown in Fig. 1, observation data is provided to visual entity identifying neural network subsystem) to generate a decoder output for the current time step (visual entity identifying network determines spatial maps which are human-interpretable representations of internal states of the system, the maps carry semantic meaning for the system and facilitate user understanding, thus the visual entity identifying network is a decoder neural network that generates a decoder output));
selecting the action to be performed by the agent at the current time step in response to the observation using the action selection policy output (Ionescu, [0055] – teaches selecting the action to be performed by the agent at the current time step in response to the observation using the action selection policy output (system receives observation data and provides action data, thus an action selection policy, for selecting an action 104 to be performed by an agent interacting within an environment at each of multiple time steps));
causing the agent to perform the selected action wherein, in response to causing the agent to perform the selected action, the machine learning system receives a reward (Ionescu, [0055] and in [0056] – “At each time step, the system 100 also receives environmental reward data 108 characterizing an external reward based on the current state of the environment 110 and the action 104 of the agent 102 at the time step.” - teaches causing the agent to perform the selected action (system provides action data for selecting an action to be performed by an agent, thus causing the agent to perform the selected action) wherein, in response to causing the agent to perform the selected action, the machine learning system receives a reward (system receives environmental reward data in response to the action of the agent at the time step)); and
training the machine learning system using the observations, the actions selected using the action selection policy output, and the rewards, by backpropagating gradients of a reinforcement learning objective function into the machine learning system (Ionescu, Fig. 6, [0083], [0084] – “The process then outputs action data for the selected action to control the agent to perform the action (step 608) and receives observation data, i.e. another image, and reward data, for a next state of the environment (step 610).”, and in [0085] – “ The process may then backpropagate gradients of the observation loss L.sub.obs to update neural network parameters of the visual entity identifying neural network subsystem 120 and may backpropagate gradients of the reinforcement learning loss L.sub.RL to update neural network parameters of the reinforcement learning neural network subsystem 130 (step 614).” – teaches training the machine learning system using the observations, the actions selected using the action selection policy output, and the rewards, by backpropagating gradients of a reinforcement learning objective function into the machine learning system); and
wherein the training includes:
determining the internal state of the machine learning system at the next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and in [0083] – “FIG. 6 shows a flow diagram of a process for training the reinforcement learning neural network system 100 of FIGS. 1-4. Observation data comprising an image of the environment is received by the system (step 600). Based on this, the Q-value neural network subsystem 330 determines a value for Q.sup.meta and the meta-controller 310 uses Q.sup.meta, e.g. with an epsilon-greedy policy, to determine a meta-action to select either an option or a series of individual tasks (step 602).” – teaches determining the internal state of the machine learning system at a next time step (as in Fig. 6, determines data characterizing transition to update parameters of system, thus determining an internal state at a next time step), by processing, using an internal state updating subsystem (training engine 140, as stated in Ionescu at [0061]), the combination of the decoder output for the current time step (as in Fig. 6 step 604, determines spatial maps from current observation, thus the decoder output for the current time step), the observation for the next time step (as in Fig. 6 step 610, receives observation data for a next state), and the counterfactual internal state (the counterfactual internal state being the options of Qmeta, wherein the options of Qmeta are different from the internal state at the current time step)).
Regarding claim 3, the combination of Ionescu and Dalli teaches the method of claim 2, wherein determining the internal state of the machine learning system at the next time step by processing the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and [0083] as explained above) defines an intervention performed according to an intervention schedule, wherein the intervention schedule comprises selecting randomly selecting time steps for the intervention (Ionescu, [0077] – “A selection made by the option controller 304, meta-controller 310, or task controller 320 may be made according to an epsilon-greedy policy, which makes a random selection with a probability ϵ and a selection based on a determined score or Q-value with a probability 1−ϵ.”, [0083], and in [0084] – “The process then executes either the option or the individual tasks for the next N time steps: Thus the image is processed to determine a set of spatial maps for the visual entities in the image, which are in turn used to determine a set of intrinsic rewards as previously described (step 604). The process then selects an action either using the action selection policy for the selected option, i.e. based upon the corresponding option Q-value, or using the Q-value for an individual task, Q.sup.task, if a series of individual tasks was selected as the meta-action.” – teaches defining an intervention performed according to an intervention schedule, wherein the intervention schedule comprises selecting randomly selecting time steps for the intervention (selection made by controllers is random according to epsilon-greedy policy, thus randomly selecting time steps for the intervention)).
Regarding claim 4, the combination of Ionescu and Dalli teaches the method of claim 2, wherein the training comprises, for each of a plurality of intervention time steps after a first of the time steps (Ionescu, [0084] – “The process then executes either the option or the individual tasks for the next N time steps: …” – teaches a plurality of intervention time steps after a first of the time steps (next N time steps)), determining the internal state of the machine learning system at the next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and [0083] as explained above); and for other time steps,
determining the internal state of the machine learning system at the next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the internal state at the current time step (Ionescu, Fig. 6 - teaches determining the internal state of the machine learning system at the next time step (as in Fig. 6, determines data characterizing transition to update parameters of system, thus determining an internal state at a next time step), by processing, using the internal state updating subsystem (training engine 140, as stated in Ionescu at [0061]), the combination of the decoder output for the current time step (as in Fig. 6 step 604, determines spatial maps from current observation, thus the decoder output for the current time step), the observation for the next time step (as in Fig. 6 step 610, receives observation data for a next state), and the internal state at the current time step (as in Fig. 6 step 600, receives observation data of current state at current time step)).
Regarding claim 5, the combination of Ionescu and Dalli teaches the method of claim 1, further comprising:
obtaining a dataset of training data, the training data comprising data defining an observation and an action for each of a succession of time steps (Ionescu, [0078] – “The reinforcement learning neural network subsystem 130 is trained end-to-end by reinforcement learning training engine 340, using off-policy learning based on data stored in a shared repl[a]y buffer 342, which stores historical observation data, action data, and external reward data from the environment.” – teaches obtaining a dataset of training data (replay buffer), the training data comprising data defining an observation and an action for each of a succession of time steps (replay buffer stores historical observation data and action data)), the training data having been obtained by, for each of the succession of time steps:
obtaining an observation characterizing the state of the environment for a current time step (Ionescu, [0055] – “The reinforcement learning neural network system 100 selects actions 104 to be performed by an agent 102 interacting with an environment 110 at each of multiple time steps. In order for the agent 102 to interact with the environment 110, the system 100 receives observation data 106 characterizing an observation of a current state of the environment and provides action data for selecting an action 104 to be performed by the agent 102 in response to the received observation data 106. ”); and
processing the internal state of the machine learning system at the current time step using the action selection subsystem of the machine learning system to generate the action selection policy output (Ionescu, [0055] – teaches processing the internal state of the machine learning system (observation data characterizing an observation of current state of the environment) at the current time step (actions performed by agents at each of multiple time steps, thus teaching a current time step for current state) using the action selection subsystem to generate the action selection policy output (system receives observation data and provides action data, thus an action selection policy, for selecting an action 104)); and
processing the dataset of training data by, for each of a succession of time steps:
processing the internal state of the machine learning system at the current time step using the decoder neural network to generate the decoder output for the current time step (Ionescu, Fig. 1, [0045], and in [0059] – “The reinforcement learning neural network system 100 comprises a visual entity identifying neural network subsystem 120 coupled to a reinforcement learning neural network subsystem 130. The visual entity identifying neural network subsystem 120 is configured to process an images of the environment to observation data determine a set of spatial maps 122… A visual entity corresponds to a spatial and/or temporal grouping of image pixels. These tend to correlate with human-interpretable entities in the environment such as walls, floor, manipulable objects, background, and the like. They also tend to carry semantic meaning for the reinforcement learning task.” – teaches processing the internal state of the machine learning system at the current time step using a decoder neural network (as shown in Fig. 1, observation data is provided to visual entity identifying neural network subsystem) to generate a decoder output for the current time step (visual entity identifying network determines spatial maps which are human-interpretable representations of internal states of the system, the maps carry semantic meaning for the system and facilitate user understanding, thus the visual entity identifying network is a decoder neural network that generates a decoder output)); and
wherein processing the dataset of training data includes:
determining the internal state of the machine learning system at a next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and in [0083] – “FIG. 6 shows a flow diagram of a process for training the reinforcement learning neural network system 100 of FIGS. 1-4. Observation data comprising an image of the environment is received by the system (step 600). Based on this, the Q-value neural network subsystem 330 determines a value for Q.sup.meta and the meta-controller 310 uses Q.sup.meta, e.g. with an epsilon-greedy policy, to determine a meta-action to select either an option or a series of individual tasks (step 602).” – teaches determining the internal state of the machine learning system at a next time step (as in Fig. 6, determines data characterizing transition to update parameters of system, thus determining an internal state at a next time step), by processing, using an internal state updating subsystem (training engine 140, as stated in Ionescu at [0061]), the combination of the decoder output for the current time step (as in Fig. 6 step 604, determines spatial maps from current observation, thus the decoder output for the current time step), the observation for the next time step (as in Fig. 6 step 610, receives observation data for a next state), and the counterfactual internal state (the counterfactual internal state being the options of Qmeta, wherein the options of Qmeta are different from the internal state at the current time step)); and further comprising
training the machine learning system using an objective function (Ionescu, [0061] – “A training engine 140 controls training of the reinforcement learning neural network system 100 according to objective functions described later.”) that depends on either:
i) a difference between a) the internal state of the agent control system at the next time step as determined by processing the counterfactual internal state, and b) an internal state of the agent control system at a next time step determined by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the internal state of the agent control system at the current time step (Ionescu, [0067] – “A second classification loss l((G.sub.t, G.sub.t+1), A.sub.t) may be used to train the visual entity identifying neural network subsystem 120, more particularly the vector quantizer 200, to distinguish between an action A.sub.t taken at time step t, transitioning between frames represented by (G.sub.t, G.sub.t+1), and actions not selected. In this way the visual entity identifying neural network subsystem 120 may learn to characterize controllable information in image observations of the environment.”, Fig. 6, and in [0083] – teaches training the machine learning system using an objective function that depends on a difference between the internal state of the agent control system at the next time step as determined by processing the counterfactual internal state and an internal state of the agent control system at a next time step determined by processing the combination of the decoder output for the current time step, the observation for the next time step, and the internal state of the agent control system at the current time step (trains to distinguish between action taken at time step t, transitioning between frames, and actions not selected, wherein processing the actions that are not selected are the counterfactual internal state, and as in Fig. 6, teaches determining an internal state of the agent control system at a next time step by processing using the decoder output for the current time step as in step 604, the observation for the next time step as in step 610, and the internal state at the current time step as in step 600)); or
ii) a difference between a) an action selection policy of the action selection subsystem determined from the internal state of the agent control system at the current time step, and b) an action selection policy of the action selection subsystem determined from the counterfactual internal state (Ionescu, [0079] – “Each Q-value has a corresponding action selection policy π.sup.meta, π.sup.task, π.sup.e×m; Q.sup.meta is used every N steps whilst Q.sup.task and Q.sup.e,m are used at every step.”, [0080] – “Each Q-value output from the Q-value neural network subsystem 330 may be trained by minimizing a corresponding loss function, L.sub.meta, L.sub.task, and L.sub.bank.sup.e,m…”, and in [0083] – teaches training the machine learning system using an objective function that depends on a difference between an action selection policy of the action selection subsystem from the internal state of the agent control system at the current time step (Q.sup.task) and an action selection policy of the action selection subsystem determined from the counterfactual internal state (Q.sup.meta)).
Regarding claim 6, the combination of Ionescu and Dalli teaches the method of claim 5, wherein the objective function depends on the difference between the internal states of the agent control system (Ionescu, [0067], Fig. 6, and [0083] as explained above), and wherein determining the internal state of the agent control system at the next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and [0083] as explained above) defines an intervention performed according to an intervention schedule, and wherein the intervention schedule comprises performing the intervention at each of the time steps (Ionescu, [0084] – “The process then executes either the option or the individual tasks for the next N time steps:” – teaches an intervention schedule wherein the intervention schedule comprises performing the intervention at each of the time steps (for the next N time steps)).
Regarding claim 7, the combination of Ionescu and Dalli teaches the method of claim 5, wherein the objective function depends on the difference between the action selection policies (Ionescu, [0079]-[0080] and [0083] as explained above), wherein determining the internal state of the agent control system at the next time step by processing, using the internal state updating subsystem, the combination of the decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and [0083] as explained above) defines an intervention performed according to an intervention schedule, and wherein the intervention schedule is defined by dividing the time steps into a sequence of blocks of time steps of variable duration and performing the intervention at each block of time steps (Ionescu, [0018] – “The system may be trained using a classification loss based on a taken/not-taken action within the time window; the time window may be a single frame, i.e. consecutive frames may be processed.”,[0077] – “A selection made by the option controller 304, meta-controller 310, or task controller 320 may be made according to an epsilon-greedy policy, which makes a random selection with a probability ϵ and a selection based on a determined score or Q-value with a probability 1−ϵ.”, [0083], and in [0084] – “The process then executes either the option or the individual tasks for the next N time steps: Thus the image is processed to determine a set of spatial maps for the visual entities in the image, which are in turn used to determine a set of intrinsic rewards as previously described (step 604). The process then selects an action either using the action selection policy for the selected option, i.e. based upon the corresponding option Q-value, or using the Q-value for an individual task, Q.sup.task, if a series of individual tasks was selected as the meta-action.” – teaches defining an intervention performed according to an intervention schedule, wherein the intervention schedule is defined by dividing the time steps into a sequence of blocks of time steps of variable duration and performing the intervention at each block of time steps (system may be trained based on actions taken within a time window, process executes option or individual tasks for next N time windows, thus dividing the time steps into a sequence of N time windows of time steps of variable duration and performing the intervention at each N window)).
Regarding claim 8, the combination of Ionescu and Dalli teaches the method of claim 5, wherein the difference between the action selection policies comprises a metric of a difference in distributions defined by the action selection policy outputs (Ionescu, [0077] – “] A score or Q-value may be provided as an output from a neural network or a neural network may provide an output defining a probability distribution from which a score or Q-value may be selected.”, [0079] – “Each Q-value has a corresponding action selection policy π.sup.meta, π.sup.task, π.sup.e×m;”, [0080] – “Each Q-value output from the Q-value neural network subsystem 330 may be trained by minimizing a corresponding loss function, L.sub.meta, L.sub.task, and L.sub.bank.sup.e,m…” – teaches wherein the difference (corresponding loss function) between the action selection policies (each Q-value has a corresponding action selection policy) comprises a metric of a difference (result of loss) in distributions defined by the action selection policy outputs (output from neural network defining a probability distribution, thus the corresponding loss function comprises a metric difference in distributions defined by the action selection policy outputs)).
Regarding claim 9, the combination of Ionescu and Dalli teaches the method of claim 5, wherein the difference between the action selection policies comprises a sum of differences between the action selection policies over a plurality of the time steps (Ionescu, [0079] – “In some implementations the Q-value neural network subsystem 330 determines Q-values for Q.sup.meta, Q.sup.task and E×M Q-values for Q.sup.e,m. Each Q-value has a corresponding action selection policy π.sup.meta, π.sup.task, π.sup.e×m;”, [0080] – “Each Q-value output from the Q-value neural network subsystem 330 may be trained by minimizing a corresponding loss function, L.sub.meta, L.sub.task, and L.sub.bank.sup.e,m… These losses may be combined into a total reinforcement learning loss, L.sub.RL, for the reinforcement learning neural network subsystem 130…” – teaches wherein the difference between action selection policies comprises a sum of differences between the action selection policies (each Q-value has a corresponding action selection policy, each Q-value output from the Q-value neural network subsystem may be trained by minimizing a corresponding loss function, thus a difference between action selection policies, these losses may be combined into a total reinforcement learning loss, thus a sum of differences between action selection policies)).
Regarding claim 10, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the counterfactual internal state comprises the internal state of the machine learning system at an earlier time step than the current time step (Ionescu, Fig. 6 and in [0080] – “The training may be performed using data previously stored in the repl[a]y buffer 342. The stored data may comprise data characterizing tuples (s.sub.t, (e.sub.t, m.sub.t, a.sub.t), r.sub.t, s.sub.t+1) where s.sub.t comprises data e.g. an image, characterizing an observation at time t, (e.sub.t, m.sub.t) characterize an option selected at time t (where e.sub.t=E+1 may represent a series of atomic actions), a.sub.t characterizes an action taken at time t which results in the state transitioning to s.sub.t+1 and a reward from the environment characterized by r.sub.t.” – teaches wherein the counterfactual internal states comprises the internal state of the machine learning system at an earlier time step (training may be performed using data previously stored in a replay buffer, thus the counterfactual internal state comprises the internal state of the machine learning system at an earlier time step)).
Regarding claim 11, the combination of Ionescu and Dalli teaches the method of claim 10, wherein the counterfactual internal state comprises the internal state of the machine learning system at a first time step (Ionescu, Fig. 6 and in [0080] – “The training may be performed using data previously stored in the repl[a]y buffer 342. The stored data may comprise data characterizing tuples (s.sub.t, (e.sub.t, m.sub.t, a.sub.t), r.sub.t, s.sub.t+1) where s.sub.t comprises data e.g. an image, characterizing an observation at time t, (e.sub.t, m.sub.t) characterize an option selected at time t (where e.sub.t=E+1 may represent a series of atomic actions), a.sub.t characterizes an action taken at time t which results in the state transitioning to s.sub.t+1 and a reward from the environment characterized by r.sub.t.” – teaches wherein the counterfactual internal state comprises the internal state of the machine learning system at a first time step (training may be performed using data previously stored in a replay buffer where t=1 is the first time step, thus the counterfactual internal state comprises the internal state at a first time step)).
Regarding claim 12, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the decoder output for the current time step comprises a vector having elements that each represent a respective one of a plurality of discrete internal states of the machine learning system (Ionescu, [0059] – “The reinforcement learning neural network system 100 comprises a visual entity identifying neural network subsystem 120 coupled to a reinforcement learning neural network subsystem 130. The visual entity identifying neural network subsystem 120 is configured to process an images of the environment to observation data determine a set of spatial maps 122.”, [0064], and in [0066] – “In implementations the visual entity identifying neural network subsystem 120 includes a frame encoding neural network 210, e.g. a convolutional neural network, to process an output of the vector quantizer 200, e.g. the set of spatial maps, to generate frame embedding data 212 comprising a frame level embedding vector G.sub.t.” – teaches wherein the decoder output for the current time step comprises a vector having elements that each represent a respective one of a plurality of discrete internal states of the machine learning system (visual entity identifying neural network subsystem determines a set of spatial maps, which are the outputs of a vector quantizer, the spatial maps carry semantic meaning for the reinforcement learning task, thus teaching wherein the decoder output comprises a vector having elements that represent discrete internal states of the machine learning system)).
Regarding claim 13, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the decoder output for the current time step comprises a natural language description of the internal state of the machine learning system (Dalli, [0197] – “ It is further contemplated that such an exemplary AutoXAI system may utilize a formal language or some other suitable means to represent a computational graph for the state knowledge stored within the XRL agent, together with more dynamic changes like the XRL loss function itself or alter the definition of Q.sub.x in Q-learning type of XRL agents… The concept of a meta-learning curiosity system in non-explainable RL agents in (Ferran et al., 2020) can thus be incorporated within XRL, capitalizing on the fact that the explainable model used within an XRL agent may also be represented by a formal language or computational graph, unlike non-explainable RL agents that utilize black-boxes.” – teaches wherein the output for the current time step comprises a natural language description of the internal state of the machine learning system (utilizes formal language or other suitable means to represent a computational graph for the state knowledge stored within the explainable RL agent, the explainable model used within an RL agent may be represented by a formal language, thus the output for the current time step comprises a natural language description of the internal state of the system)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the output comprising natural language description of Dalli to the decoder output and internal states of Ionescu. Doing so would provide means to represent the knowledge of the RL agent in a formal language (Dalli, [0197]), and enable RL agents to have human knowledge embedded and combined with machine generated knowledge (Dalli, [0052]).
Regarding claim 14, the combination of Ionescu and Dalli teaches the method of claim 1, further comprising:
modifying the decoder output for the current time step (Ionescu, Fig. 1, [0045] – “In this way concepts used by the RL system to guide its actions can be expressed visually to the human user. Such information can be particularly important for safety and diagnostic purposes. For example it may be used to determine whether or not the RL system is behaving, or is likely to behave, in a safe way and/or it may be used to determine whether or not the system is behaving correctly/as expected, for example for fault-finding.”, and in [0059] – “The reinforcement learning neural network system 100 comprises a visual entity identifying neural network subsystem 120 coupled to a reinforcement learning neural network subsystem 130. The visual entity identifying neural network subsystem 120 is configured to process an images of the environment to observation data determine a set of spatial maps 122… These tend to correlate with human-interpretable entities in the environment such as walls, floor, manipulable objects, background, and the like. They also tend to carry semantic meaning for the reinforcement learning task.” – teaches modifying the decoder output for the current time step (provides spatial maps to human user for diagnostic purposes and fault finding, thus modifying the decoder output for the current time step according to diagnostics and faults found by human user)), and
determining the internal state of the machine learning system at a next time step by processing, using the internal state updating subsystem, a combination of the modified decoder output for the current time step, the observation for the next time step, and the counterfactual internal state (Ionescu, Fig. 6 and in [0083]).
Regarding claim 15, the combination of Ionescu and Dalli teaches the method of claim 14, further comprising controlling the behavior of the agent in the environment by:
outputting the decoder output for the current time step via a user interface (Ionescu, Fig. 1, [0045] – “In this way concepts used by the RL system to guide its actions can be expressed visually to the human user. Such information can be particularly important for safety and diagnostic purposes. For example it may be used to determine whether or not the RL system is behaving, or is likely to behave, in a safe way and/or it may be used to determine whether or not the system is behaving correctly/as expected, for example for fault-finding.”, [0059], and [0095] – teaches outputting the decoder output for the current time step via a user interface (spatial maps, which are the decoder output for the current time step, are presented visually to the human, for example by a user interface as in Ionescu at [0095]));
receiving an instruction from the user interface, wherein the instruction relates to performance of the task in the environment (Ionescu, Fig. 1, [0045] – “In this way concepts used by the RL system to guide its actions can be expressed visually to the human user. Such information can be particularly important for safety and diagnostic purposes. For example it may be used to determine whether or not the RL system is behaving, or is likely to behave, in a safe way and/or it may be used to determine whether or not the system is behaving correctly/as expected, for example for fault-finding.” – teaches receiving an instruction from the user interface, wherein the instruction relates to performance of the task in the environment (concepts used by RL system to guide actions are expressed visually to human user for diagnostics and fault-finding, thus receiving an instruction related to performance of the task in the environment)); and
generating the modified decoder output from the instruction(Ionescu, Fig. 1, [0045] – “In this way concepts used by the RL system to guide its actions can be expressed visually to the human user. Such information can be particularly important for safety and diagnostic purposes. For example it may be used to determine whether or not the RL system is behaving, or is likely to behave, in a safe way and/or it may be used to determine whether or not the system is behaving correctly/as expected, for example for fault-finding.”, and in [0059] – teaches generating the modified decoder output from the instruction (provides spatial maps to human user for diagnostic purposes and fault finding, thus modifying the decoder output for the current time step according to diagnostics and faults found by human user)).
Ionescu fails to explicitly teach outputting the signal derived from the decoder output.
However, analogous to the field of the claimed invention, Dalli teaches:
outputting the signal derived from the decoder output (Dalli, [0042], [0076], [0102] – “Explanations in XRL agents may be seen as performing two main functions: (i.) explaining the actions and decisions of the agent, in an audit-like manner; (ii.) using the explanation model itself as a form of predictive function that acts using the explanatory model space together with the observed space… The addition of plausibility-based objectives in XRL agents may make XRL agents more suitable for use in exploration, observation, and interventional interactions with realistic environments such as the real physical world.”, and in [0200] – “In another exemplary application used for illustrative purposes, a typical XRL agent such as the one illustrated in FIG. 3 may utilize control engineering models, such as Nyquist analysis, Bode plots, and other forms of analyses to process a wide variety of processing signals.” – teaches providing a signal (agents may utilize analyses to process wide variety of processing signals) derived from a decoder output (explanations in agents) (agents utilize analyses to process signals and explanations that make agents more suitable for use in exploration, observation, and interventional actions, thus providing a signal derived from the decoder output as a monitoring signal for monitoring control of the agent in the environment)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the signal output of Dalli to the decoder output, training methods, and machine learning system of Ionescu. Doing so would incorporate explanations as part of state/action spaces of an RL agent and thus provide richer exploratory space for the agent (Dalli, [0042]), and provide fully traceable explanations at all steps taken by the RL agent (Dalli, [0102]).
Regarding claim 16, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the action selection subsystem comprises an action selection policy neural network of the machine learning system (Ionescu, [0022] – “The reinforcement learning neural network subsystem may be configured to process the internal reward data for the option to determine a corresponding set of state-action Q-values for the option, later “option Q-values”. The option Q-values may be used for an action selection policy for the option when performing a series of actions associated with the option.” and in [0055] – teaches wherein the action selection subsystem comprises an action selection policy neural network of the machine learning system (option Q-values may be used for an action selection policy, thus the action selection subsystem comprises an action selection policy neural network)).
Regarding claim 17, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the internal state updating subsystem comprises a recurrent neural network (Ionescu, [0025] – “In some implementations the reinforcement learning (RL) neural network subsystem (agent) may have an RL agent structure comprising one or more convolutional neural network layers followed by one or more recurrent neural network layers, for example comprising one or more LSTM (Long Short-Term Memory) layers.” – teaches wherein the internal state updating system comprises a recurrent neural network (RL neural network subsystem may have one or more recurrent neural network layers, for example one or more LSTM layers)).
Regarding claim 18, the combination of Ionescu and Dalli teaches the method of claim 1, wherein the internal state updating subsystem comprises a transformer neural network (Dalli, [0045] – “A second alternative embodiment may use an XRL induction method which makes use of an experience dataset to learn a supervised white-box model, using an explainable architecture x, where x an XAI model (XAI), interpretable neural network (INN), explainable transducer transformer (XTT), explainable GAN (XGAN), explainable auto-encoder/decoder (XAED)...” – teaches wherein the internal state updating subsystem comprises a transformer neural network (learns a supervised white-box model such as a transformer or an auto-encoder/decoder, thus the system comprises a transformer neural network)).
Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the transformer neural network of Dalli to the internal state updating subsystem of Ionescu. Doing so would provide architectures that match or mimic the behavior of an RL system (Dalli, [0045]) and provide explanations as part of the RL agent model, giving it a richer exploratory space (Dalli, [0042]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Foerster et al. (NPL: Counterfactual Multi-Agent Policy Gradients, published 2018) teaches a multi-agent actor-critic method called counterfactual multi-agent policy gradients. Teaches obtaining an observation, an action selection policy network for selecting an action each time step, and determining counterfactual internal states. Teaches training the system by a difference between the counterfactual internal state and the internal state at the current time step.
Madumal et al. (Distal Explanations for Explainable Reinforcement Learning Agents, published Jan. 2020) teaches casual explanations for reinforcement learning agents, where casual explanations provide explanations of events through casual chains. Teaches a distal explanation model that analyses counterfactuals and generate counterfactual explanations. Teaches embedding counterfactuals in explanations to make explanations more meaningful.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LOUIS C NYE whose telephone number is 571-272-0636. The examiner can normally be reached Monday - Friday 9:00AM - 5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATT ELL can be reached at 571-270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LOUIS CHRISTOPHER NYE/Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights