LANGUAGE-GUIDED DISTRIBUTIONAL TREE SEARCH

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Response to argument Applicant's arguments filed 08/04/2026 (Remarks/Arguments: (pgs. 10 – 13)) have been fully considered but they are not persuasive. Applicant’s arguments with respect to amended claim(s) have been considered but are moot, because arguments/remarks are directed to amended claim limitations that were not previously examined by the examiner. The rejections are noted in the current office action to address amended claim limitations. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claim(s) 1 – 3, 6 – 7, 18 – 25, 27 and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan, et al., "Learning to win by reading manuals in a monte-carlo framework." in view of Al, et al., "Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images" and Karaletsos et al., Pub. No.: US20200372410A1. Regarding claim 1, Branavan teaches: A method comprising: performing a language-guided tree search of a tree structure using one or more neural networks (Branavan, page: 663, “Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the Monte-Carlo Search [of a tree structure] framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward from these simulations to update the parameters of our neural network [using one or more neural networks] via backpropagation. This focuses learning on the current game state, allowing our method to learn language analysis [performing a language-guided tree search] and game-play appropriate to the observed game context.”) expand a search tree to select a selected action of a plurality of actions associated with a goal (Branavan, page: 664, “Assume that every state s is represented by a set of n features [s1, s2, . . . , sn]. Given a state s, our goal is to select the best possible action aj from a fixed set A [expand a search tree to select a selected action of a plurality of actions associated with a goal]. We can model this task as multiclass classification, where each choice aj is represented by a feature vector [(s1, aj ),(s2, aj ), . . . ,(sn, aj )]. Here, (si , aj ), i PNG media_image1.png 28 24 media_image1.png Greyscale [1, n] represents a feature created by taking the Cartesian product between [s1, s2, . . . , sn] and aj . To learn this classifier effectively, we need a training set that sufficiently covers the possible combinations of state features and actions. However, in do1mains with complex state spaces and a large number of possible actions, many instances of state-action feature values will be unobserved in training.”) causing the one or more autonomous agents to perform the selected action, (Branavan, page: 670, “The game playing agent [causing the one or more autonomous agents] selects actions [to perform the selected action] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a).”) based at least in part, on the expansion of the search tree. (Branavan, Fig. 3) PNG media_image2.png 321 203 media_image2.png Greyscale PNG media_image3.png 139 575 media_image3.png Greyscale [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] Branavan does not teach: to predict a set of future world states based, at least in part, on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the plurality of actions, a particular one of the set of distributions of rewards generated as a result of using the one or more neural networks to simulate an execution of a particular action of the plurality of actions in an environment, causing the one or more autonomous agents to move in response to performing the selected action. wherein the set of future world states indicate a position of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D) Karaletsos teaches: to predict a set of future world states (Karaletsos, "[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [to predict a set of future world states] induced by proposed action sequences at, at+1, . . . , at+h. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence.”) based, at least in part, on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the plurality of actions, (Karaletsos, “[0076] In equation 6, the term p(zd) represents the distribution of the latent variable zd, the term p(zr) represents the distribution of the latent variable zr, and the term p(s0) represents the distribution of the state s0, the term [p(rt+1|st, at, st+1, zr) represents the distribution of the reward rt+1 [based, at least in part, on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the plurality of actions], the term p(st+1|st, at, zd) represents the distribution of the state st+1, and the term p(at|st, zr, zd) represents the distribution of actions at. This structure of the model facilitates solving tasks where both of these aspects (dynamics and reward) can vary independently.”) a particular one of the set of distributions of rewards generated as a result of using the one or more neural networks to simulate an execution of a particular action of the plurality of actions in an environment (Karaletsos, “[0077] In general, the model may have c arbitrary plated contexts, such as agent, dynamics, reward variation, etc [a particular one of the set of distributions of rewards]. Accordingly, the model uses a set of latent variables {z1, . . . , zc}, each latent variable representing a different factor of variation in the system, implying p(z)=Πp(zc). This allows the model to have separate degrees of freedom in latent space for distinct effects [generated as a result of using the one or more neural networks]. The use of plated variables implies that tasks will have known factors of variation (but unknown values and effects) at training time only. In practice, this is the case when training on a simulator [to simulate an execution of a particular action of the plurality of actions in an environment].”) causing the one or more autonomous agents to move in response to performing the selected action. (Karaletsos, “[0045] An action represents a move that the agent can make. An agent selects from a set of possible actions [causing the one or more autonomous agents to move in response to performing the selected action]. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives. If the system is part of a drone, the set of actions includes increasing speed, decreasing speed, changing direction, and so on. If the system is part of a robot, the set of actions includes walking forward, turning left or right, climbing, and so on. If the system is part of a self-driving vehicle, the set of actions includes driving the vehicle, stopping the vehicle, accelerating the vehicle, turning left/right, changing gears of the vehicle, changing lanes, and so on.”) Karaletsos and Branavan are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Karaletsos with teaching of Branavan to add latent variable based model to improve transferability and adaptability to new and unseen conditions. (Karaletsos, Abstract). Branavan in view of Karaletsos do not teach: wherein the set of future world states indicate a position of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D) AI teaches: wherein the set of future world states indicate a position (AI, page: 1247, “A. State: We represent the state as a function of the agent position [wherein the set of future world states indicate a position]. For any position q in the discrete voxel-space, the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position.”) of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D) (AI, page: 1248, “C. Reward The agent at any position inside a 3D volume targets at choosing an action that maximizes the discounted cumulative reward [of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D)]. Therefore, we should encourage the agent to come closer to the target by giving an appropriate reward. We propose to use a simple binary reward function, where a positive reward is given if an action leads the agent closer to target landmark, and a negative reward is given otherwise.”) AI, Branavan and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of AI with teaching of Branavan and Karaletsos to improve learning speed and robustness, while the reconstructed global policy ensures coordinated behavior across the entire task space (AI, Abstract). Regarding claim 2, Branavan in view of Karaletsos and AI teach the method of claim 1. Branavan further teaches: wherein the set of action is provided by an action proposal network. (Branavan, page: 664, “2. Learning Game Play from Text: In this section, we provide an intuitive explanation of how textual information can help improve action selection in a complex game. For clarity, we first discuss the benefits of textual information in the supervised scenario, thereby decoupling questions concerning modeling and representation from those related to parameter estimation. Assume that every state s is represented by a set of n features [s1, s2, . . . , sn]. Given a state s, our goal is to select the best possible action aj [wherein the set of action is provided by an action proposal network] from a fixed set A. We can model this task as multiclass classification, where each choice aj is represented by a feature vector [(s1, aj ),(s2, aj ), . . . ,(sn, aj )]. Here, (si , aj ), i PNG media_image1.png 28 24 media_image1.png Greyscale [1, n] represents a feature created by taking the Cartesian product between [s1, s2, . . . , sn] and aj . To learn this classifier effectively, we need a training set that sufficiently covers the possible combinations of state features and actions. However, in domains with complex state spaces and a large number of possible actions, many instances of state-action feature values will be unobserved in training.”) Regarding claim 3, Branavan in view of Karaletsos and AI teach the method of claim 1. Branavan further teaches: wherein the search tree further comprises a first edge representing the selected action and a node including a representation of a world state conditioned on performing the selected action. (Branavan, Fig. 2) [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: ] PNG media_image6.png 234 502 media_image6.png Greyscale [AltContent: textbox ([wherein the search tree further comprises a first edge representing the selected action])][AltContent: textbox ([and a node including a representation of a world state conditioned on performing the selected action])] (“Figure 2: Markov Decision Process. Actions are selected according to policy function π(s, a) given the current state s. The execution of the selected action ai (e.g., a1), causes the MDP to transition to a new state s 0 according to the stochastic state transition distribution T(s 0 | s, a).”) Regarding claim 6, Branavan in view of Karaletsos and AI teach the method of claim 1. Branavan further teaches: wherein the goal is determined based at least in part on natural language. (Branavan, page: 667, “While our method is also driven by control feedback, our language interpretation task itself is fundamentally different. We assume that the given text document provides high level advice without directly describing the correct actions for every potential game state. Furthermore, the textual advice does not necessarily translate to a single strategy – in fact, the text may describe several strategies, each contingent on specific game states. For this reason, the strategy text cannot simply be interpreted directly into a policy. Therefore, our goal [wherein the goal] is to bias a learned policy using information extracted from text [is determined based at least in part on natural language] (i.e.: using natural language (textual information) to bias or influence a learned policy, that the goal or objective in decision-making is influenced by insights derived from the textual advice). To this end, we do not aim to achieve a complete semantic interpretation, but rather use a partial text analysis to compute features relevant for the control application.”) Regarding claim 7, Branavan in view of Karaletsos and AI teach the method of claim 6. Branavan further teaches: wherein the goal is a long horizon goal comprising one or more tasks determined based at least in part on the natural language. (Branavan, page: 672, “5. Adding Linguistic Knowledge to the Monte-Carlo Framework: The goal [wherein the goal is a long horizon goal] of our work is to improve the performance of the Monte- Carlo Search framework described above, using information automatically extracted from text [based at least in part on the natural language]. In this section, we describe how we achieve this in terms of model structure and parameter estimation. 5.1 Model Structure: To achieve our aim of leveraging textual information to improve game-play, our method needs to perform three tasks [comprising one or more tasks determined]: (1) identify sentences relevant to the current game state, (2) label sentences with a predicate structure, and (3) predict good game actions by combining game features with text features extracted via the language analysis steps. We first describe how each of these tasks can be modeled separately before showing how we integrate them into a single coherent model.”) Regarding claim 18, Branavan teaches: An automated agent comprising: (Branavan, page: 672, “In fact, in our approach, we use standard gradient descent updates from RL to estimate the parameters of Q(s, a). There is, however, one crucial difference between these two techniques: In general, the goal in RL is to find a Q(s, a) applicable to any state the agent [An automated agent] may observe during its existence. In the Monte-Carlo Search framework, the aim is to learn a Q(s, a) specialized to the current state s.”) one or more circuits (Branavan, page: 683, “The experiments were run on typical desktop PCs [one or more circuits] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU).”) to perform natural language goal-directed tasks using one or more neural networks (Branavan, page: 663, “1.1 Summary of the Approach: We address the above challenges in a unified framework based on Markov Decision Processes (MDP), a formulation commonly used for game playing algorithms. This setup consists of a game in a stochastic environment, where the goal of the player [goal-directed tasks] is to maximize a given utility function R(s) at state s. The player’s behavior is determined by an action-value function Q(s, a) [at least in part on performance of an action of a set of actions given a particular state] that assesses the goodness of action a at state s based on the attributes of s and a. To incorporate linguistic information into the MDP formulation, we expand the action value function to include linguistic features. While state and action features are known at each point of computation, relevant words and their semantic roles are not observed. Therefore, we model text relevance as a hidden variable. Similarly, we use hidden variables to discriminate the words that describe actions and those that describe state attributes from the rest of the sentence. To incorporate these hidden variables in our action-value function, we model Q(s, a) as a non-linear function approximation using a multi-layer neural network. Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the MonteCarlo Search framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward [to determine a future reward] from these simulations to update the parameters of our neural network [using one or more neural networks] via backpropagation. This focuses learning on the current game state, allowing our method to learn language analysis [to perform natural language] and game-play appropriate to the observed game context.”) to execute a language-guided tree search. (Branavan, page: 669, “3.3.2 Estimating the Value of Untried Actions Previous approaches to estimating the value of untried actions have relied on two techniques. The first, Upper Confidence bounds for Tree (UCT) is a heuristic used in concert with the Monte-Carlo Tree Search [tree search] variant of MCS. It augments an action’s value with an exploration bonus for rarely visited stateaction pairs, resulting in better action selection and better overall game performance (Gelly et al., 2006; Sturtevant, 2008; Balla & Fern, 2009). The second technique is to learn a linear function approximation of action values for the current state s, based on game feedback (Tesauro & Galperin, 1996; Silver, Sutton, & Muller, 2008). Even though our method follows the latter approach, we model action-value Q(s, a) via a non-linear function approximation. Given the complexity of our application domain, this non-linear approximation generalizes better than a linear one, and as shown by our results significantly improves performance. More importantly, the non-linear model enables our method to represent text analysis [to execute a language-guided] as latent variables, allowing it to use textual information to estimate the value of untried actions.”) a tree structure to define relationships between the goal-directed tasks and the future state; (Branavan, Fig. 3) [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] PNG media_image2.png 321 203 media_image2.png Greyscale [AltContent: textbox ([a tree structure to define relationships between the goal-directed tasks and the future state])] Branavan does not teach: based, at least in part, on a future state predicted by the one or more neural networks reward distributions generated based, at least in part, on the future state and at least one of the goal-directed tasks the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks, cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks. the future state indicating a position of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed, Karaletsos teaches: based, at least in part, on a future state predicted by the one or more neural networks reward distributions generated based, at least in part, on the future state and at least one of the goal-directed tasks (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h induced by proposed action sequences at, at+1, . . . , at+h [based, at least in part, on a future state predicted by the one or more neural networks]. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence [reward distributions generated based, at least in part, on the future state and at least one of the goal-directed tasks]. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”) the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks (Karaletsos, “[0075] In some embodiments, the machine learning model uses multiple plated variables which constitute the structured latent space of the GHP-MDP. Separate latent spaces for dynamics and reward allow agents to pursue different goals across environments with different dynamics [the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks]. The joint model p(s0:T+1, a0:T, R, zd, zr), including the action distribution implied by control, is described using equation (6).”) cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks (Karaletsos, “[0045] An action represents a move that the agent can make [cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks]. An agent selects from a set of possible actions. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives…”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan for the same reasons disclosed for claim 1. Branavan in view of Karaletsos do not teach: the future state indicating a position of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed, AI teaches: the future state indicating a position (AI, page: 1247, “A. State: We represent the state as a function of the agent position [the future state indicating a position]. For any position q in the discrete voxel-space, the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position.”) of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed, (AI, page: 1248, “C. Reward The agent at any position inside a 3D volume targets at choosing an action that maximizes the discounted cumulative reward [of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed]. Therefore, we should encourage the agent to come closer to the target by giving an appropriate reward. We propose to use a simple binary reward function, where a positive reward is given if an action leads the agent closer to target landmark, and a negative reward is given otherwise.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of AI with teaching of Branavan and Karaletsos for the same reasons disclosed for claim 1. Regarding claim 19, Branavan in view of Karaletsos and AI teach the method of claim 18. Karaletsos further teaches: wherein the automated agent further comprises a robot. (Karaletsos, “[0019] A hidden parameter may represent an attribute of the agent, for example, a condition of robot indicating whether a part of the robot is rusted [wherein the automated agent further comprises a robot], whether a part of the robot (e.g., a leg) is not functioning, whether a camera is malfunctioning, and so on. A hidden parameter may represent conditions of environment, for example, icy condition, windy conditions, rainy condition, and so on.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teachings of Branavan and Al for the same reasons disclosed for claim 1. Regarding claim 20, Branavan in view Karaletsos and AI teach the method of claim 19. Karaletsos further teaches: performing the at least one of the natural language goal-directed tasks causes the robot to perform an action comprising moving. (Karaletsos, “[0045] An action represents a move that the agent can make [performing the at least one of the natural language goal-directed tasks causes the robot to perform an action comprising moving]. An agent selects from a set of possible actions. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives…”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan and AI for the same reasons disclosed for claim 1. Regarding claim 21, Branavan in view of Karaletsos and AI teach the method of claim 20. Branavan further teaches: wherein the action is proposed by a policy model of the one or more neural networks. (Branavan, page: 670, “The game playing agent selects actions [wherein the action is proposed] according to a stochastic policy [is proposed by a policy model of the one or more neural networks] π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image8.png 19 18 media_image8.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ (s, a) is known, the optimal game-playing behavior would be to select the action a with the highest Qπ PNG media_image9.png 19 18 media_image9.png Greyscale (s, a).”) Regarding claim 22, Branavan in view of Karaletsos and AI teach the method of claim 20. Branavan further teaches: wherein the action is selected based at least in part on a value associated with the action determined by a value model of the one or more neural networks. (Branavan, page: 670, “The game playing agent selects actions [wherein the action is selected] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a [based at least in part on a value associated with the action] (i.e.: the action is chosen based on the expected total reward (action-value function), which is a value associated with the action) in state s, and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image10.png 19 18 media_image10.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ PNG media_image11.png 18 17 media_image11.png Greyscale (s, a) is known, the optimal game-playing behavior would be to select the action a with the highest Qπ PNG media_image12.png 19 18 media_image12.png Greyscale (s, a) [determined by a value model of the one or more neural networks].”) Regarding claim 23, Branavan teaches: A processor comprising one or more circuits to: A processor comprising: one or more circuits to: obtain a goal based, at least in part, on natural language; (Branavan, page: 670, “In contrast to such approaches, our goal is to automatically extract and use domain knowledge from relevant natural language documents, thus bypassing the need for manual specification. Our method learns both text interpretation and game action selection based on the outcomes of simulated games in MCS [A processor comprising: one or more circuits to: obtain a goal based, at least in part, on natural language].”) an edge of the tree PNG media_image2.png 321 203 media_image2.png Greyscale [AltContent: textbox ([an edge of the tree])][AltContent: arrow](Branavan, Fig. 3) corresponds to a distribution of rewards generated based, at least in part, on the action and the future state, the set of distributions of rewards comprising a different reward (Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [corresponds to a distribution of rewards generated based, at least in part, on the action and the future state] and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image13.png 19 18 media_image13.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game [the set of distributions of rewards comprising a different reward].”) distribution associated with each of two or more nodes edges of the tree, the two or more edges representing two or more actions; (Branavan, Fig. 3) PNG media_image2.png 321 203 media_image2.png Greyscale [AltContent: textbox ([distribution associated with each of two or more nodes edges of the tree, the two or more edges representing two or more actions])][AltContent: rect][AltContent: arrow] The rest of the limitations are analogous to claim 1, so are rejected under similar rationale. Regarding claim 24, Branavan in view of Karaletsos and AI teach the method of claim 23. Branavan further teaches: the distribution of the set of distributions of rewards represents a utility as a result of being in the future state. (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [represents a utility as a result of being in the future state] induced by proposed action sequences at, at+1, . . . , at+h. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence [the distribution of the set of distributions of rewards]. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teachings of Branavan and Al for the same reasons disclosed for claim 1. Regarding claim 25, Branavan in view of Karaletsos and AI teach the method of claim 24. Branavan further teaches: wherein the node includes a value independent of the distribution of the set of distributions of rewards and represents the utility of being in the future state. (Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [wherein the node includes a value independent of the distribution of the set of distributions of rewards and represents the utility of being in the future state] and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image10.png 19 18 media_image10.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”) Regarding claim 27, Branavan in view of Karaletsos and AI teach the method of claim 23. Karaletsos further teaches: wherein the automated agent further comprises a robot. (Karaletsos, “[0019] A hidden parameter may represent an attribute of the agent, for example, a condition of robot indicating whether a part of the robot is rusted [wherein the automated agent further comprises a robot], whether a part of the robot (e.g., a leg) is not functioning, whether a camera is malfunctioning, and so on. A hidden parameter may represent conditions of environment, for example, icy condition, windy conditions, rainy condition, and so on.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teachings of Branavan and Al for the same reasons disclosed for claim 1. Regarding claim 31, Branavan in view of Karaletsos and AI teach the method of claim 23. Branavan further teaches: wherein the goal is determined based, at least in part, on unstructured natural language instructions. (Branavan, page: 672, “5. Adding Linguistic Knowledge to the Monte-Carlo Framework: The goal [wherein the goal] of our work is to improve the performance of the Monte-Carlo Search framework described above, using information automatically extracted from text [is determined based, at least in part, on unstructured natural language instructions] (i.e.: the extracted text can come from various sources, such as game manuals, dialogues, user instructions, or any relevant documentation (unstructured natural language instruction)). In this section, we describe how we achieve this in terms of model structure and parameter estimation. 5.1 Model Structure To achieve our aim of leveraging textual information to improve game-play, our method needs to perform three tasks: (1) identify sentences relevant to the current game state, (2) label sentences with a predicate structure (i.e.: step of structuring extracted text), and (3) predict good game actions by combining game features with text features extracted via the language analysis steps. We first describe how each of these tasks can be modeled separately before showing how we integrate them into a single coherent model.”) Claim(s) 4 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI and in further view of Venkatraman et al. "Improved learning of dynamics models for control." Regarding claim 4, Branavan in view of Karaletsos and AI teach the method of claim 3. Branavan in view of Karaletsos and AI do not teach: wherein the representation is generated by a dynamics model. Venkatraman teaches: wherein the representation is generated by a dynamics model. (Venkatraman, page: 3, “This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state-action pairs; (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [wherein the representation is generated by a dynamics model] (i.e.: the dynamics model is learned from the aggregated dataset, which represents how the system transitions from one state to the next given an action) ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated.”) Venkatraman, Branavan, Karaletsos and AI are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Venkatraman with teachings of Branavan, Karaletsos and AI to enhance a reinforcement learning agent by allowing it to simulate and adapt to complex dynamics. (Venkatraman, page: 1). Regarding claim 8, Branavan in view of Karaletsos and AI teach the method of claim 1. Branavan further teaches: wherein expanding the search tree further comprises, for a first node of the search tree: obtaining the set of actions from an action proposal model; creating an edge from the first node to a second node representing the selected action; annotating the second node to indicate a value associated with the world state and the edge with a reward associated with the selected action. obtaining a state representation from a dynamics model representing a world state upon completion of the selected action PNG media_image14.png 387 642 media_image14.png Greyscale [AltContent: textbox ([creating an edge from the first node to a second node representing the selected action])][AltContent: textbox ([annotating the second node to indicate a value associated with the world state and the edge with a reward associated with the selected action])][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect](Branavan, Fig. 3) (Branavan, page: 671, “Figure 3: Overview of Monte-Carlo Search algorithm. For each game state st , an independent set of simulated games or roll-outs are done to find the best possible game action at . Each roll-out starts at state st , with actions [for a first node of the search tree: obtaining the set of actions from an action proposal model] selected according to a simulation policy π(s, a). This policy is learned from the roll-outs themselves – with the roll-outs improving the policy, which in turn improves roll-out action selection. The process is repeated for every actual game state, with the simulation policy being relearned from scratch each time.”) Branavan in view of Karaletsos and AI do not teach: obtaining a state representation from a dynamics model representing a world state upon completion of the selected action; Venkatraman teaches: obtaining a state representation from a dynamics model representing a world state upon completion of the selected action; (Venkatraman, page: 3, “We specifically refer to the left loop in Fig. 1 as the DAgger (Data-set Aggregation) system identification learning framework [18]. A key difference lies in the aggregation step of the procedure in order to provide model agnostic guarantees. At the beginning of the algorithm, DAgger initializes an empty training data-set and an exploration policy πexplore(xt) that generates an action (control) ut given a state xt. [obtaining a state representation] This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state [representing a world state] -action pairs; [upon completion of the selected action] (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [from a dynamics model] ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated. During the execution of the first DAgger loop, the state distribution induced by π can greatly differ from the initial π explore; the first generated policies may perform poorly due to inaccuracies in ˆf. The iterative procedure refines the dynamics model by aggregating data from states induced by running the system with π1, . . . , πN .”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Venkatraman with teaching of Branavan, Karaletsos and Al for the same reasons disclosed for claim 4. Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI and in further view of Remez et al., "Improved learning of dynamics models for control." Regarding claim 5, Branavan in view of Karaletsos and AI teach the method of claim 4. Branavan in view of Karaletsos, AI do not teach: wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics. Remez teaches: wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics. (Remez, page: 6, “3 Architecture There are three modules in our model [wherein the dynamics model]: (1) the generator, which predicts a mask, (2) the cut-and-paste module [further comprises a cut and paste model that implements observed space dynamics] (i.e.: the model predicts certain features (masks or patches), manipulates them (cut-and-paste), and possibly uses them in subsequent steps (like discrimination between real and fake patches). This reflects a system where dynamics are modeled not just through prediction but also through manipulation and simulation of observed data dynamics), which produces a “fake patch” given the predicted mask, and (3), the discriminator, which distinguishes between real and fake patches, see Figure 3. In the following, we describe the architecture for each of these modules that we have used in our experiments.”) Remez, Branavan, Karaletsos and AI are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Remez with teachings of Branavan, Karaletsos and AI to enhance the agent’s ability to accurately segment objects in new environments, by leveraging adversarial learning to refine its predictions, ultimately achieving performance close to fully supervised methods (Remez, Abstract). Claim(s) 9 – 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI and in further view of Simonyan et al. Pub. No.: US20200143239A1. Regarding claim 9, Branavan teaches: A system comprising: one or more processors; one or more memories to store parameters associated with one or more neural networks; (Branavan, page: 683, “The experiments were run on typical desktop PCs [one or more processors] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [one or more memories to store parameters associated with one or more neural networks].”) use an action proposal model to generate a plurality of actions; (Branavan, page: 663, “1.1 Summary of the Approach: We address the above challenges in a unified framework based on Markov Decision Processes (MDP), a formulation commonly used for game playing algorithms. This setup consists of a game in a stochastic environment, where the goal of the player is to maximize a given utility function R(s) at state s. The player’s behavior is determined by an action-value function Q(s, a) that assesses the goodness of action a at state s based on the attributes of s and a [use an action proposal model to generate a plurality of actions]. To incorporate linguistic information into the MDP formulation, we expand the action value function to include linguistic features. While state and action features are known at each point of computation, relevant words and their semantic roles are not observed.”) the one or more autonomous agents to perform a selected action of the plurality of actions (Branavan, page: 670, “The game playing agent [the one or more autonomous agents] selects actions [to perform a selected action of the plurality of actions] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a).”) based, at least in part, on a performance of a language-guided tree search of the tree structure by the one or more neural networks (Branavan, Fig. 3) PNG media_image2.png 321 203 media_image2.png Greyscale PNG media_image16.png 138 575 media_image16.png Greyscale [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] (Branavan, page: 663, “Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the Monte-Carlo Search framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward from these simulations to update the parameters of our neural network via backpropagation [of the tree structure by the one or more neural networks]. This focuses learning on the current game state, allowing our method to learn language analysis and game-play appropriate to the observed game context.”) wherein the performance of the language-guided tree search causes the selected action to be selected from the plurality of actions based, at least in part, on the set of values and the set of reward distributions. (Branavan, page: 670, “All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a [wherein the performance of the language-guided tree search causes the selected action to be selected from the set of actions based, at least in part, on the set of values]. When the agent executes action a, the game state changes according to the state transition distribution [and the set of reward distributions]. While T(s 0 | s, a) is not known a priori, state transitions can be sampled from this distribution by invoking the game code as a black-box simulator – i.e., by playing the game. After each action, the agent receives a reward according to the reward function R(s).”) Branavan does not teach: use a dynamics model to predict a set of future world states that would result from simulating an execution of the plurality of actions in an environment use a value model to generate a set of values associated with the set of future world states; use a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the plurality of actions; the selected action when performed causing the one or more autonomous agents to move. wherein the set of future world states indicates a position of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to performing the actions, a tree structure to define relationships between the plurality of actions and the set of future world states; Karaletsos teaches: use a dynamics model to predict a set of future world states that would result (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [to predict a set of future world states that would result] induced by proposed action sequences at, at+1, . . . , at+h. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model [use a dynamics model].”) … from simulating an execution of the plurality of actions in an environment (Karaletsos, “[0048] … If the agent is being trained using a simulator, the state representation may include information that can be extracted from the simulator that may not be available in real-world [… from simulating an execution of the plurality of actions in an environment], for example, the position of the robot even if the position may not be available to a robot in real world. The availability of additional information that may not be available in real world is utilized by the explore phase to efficiently find solutions to the task.”) use a value model to generate a set of values associated with the set of future world states; (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan [use a value model to generate a set of values] into the future by recursively predicting future states st+1, . . . , st+h [associated with the set of future world states] induced by proposed action sequences at, at+1, . . . , at+h. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model [use a dynamics model].”) use a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the plurality of actions; (Karaletsos, “[0074] In this model, a global latent variable z conditions both the dynamics and the reward distributions [use a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the plurality of actions]. The priors for auxiliary latent variable are set to simple normal distributions, p(z)=n(0, I), and initial state distribution p(s0) to the environment simulator.”) the selected action when performed causing the one or more autonomous agents to move. (Karaletsos, “[0045] An action represents a move that the agent can make. An agent selects from a set of possible actions [the selected action when performed causing the one or more autonomous agents to move]. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives. If the system is part of a drone, the set of actions includes increasing speed, decreasing speed, changing direction, and so on. If the system is part of a robot, the set of actions includes walking forward, turning left or right, climbing, and so on. If the system is part of a self-driving vehicle, the set of actions includes driving the vehicle, stopping the vehicle, accelerating the vehicle, turning left/right, changing gears of the vehicle, changing lanes, and so on.”) Karaletsos and Branavan are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Karaletsos with teaching of Branavan to add latent variable based model to improve transferability and adaptability to new and unseen conditions. (Karaletsos, Abstract). Branavan in view of Karaletsos do not teach: wherein the set of future world states indicates a position of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to performing the actions, a tree structure to define relationships between the plurality of actions and the set of future world states; AI teaches: wherein the set of future world states indicates a position (AI, page: 1247, “A. State: We represent the state as a function of the agent position [wherein the set of future world states indicates a position]. For any position q in the discrete voxel-space, the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position.”) of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to performing the actions, (AI, page: 1248, “C. Reward The agent at any position inside a 3D volume targets at choosing an action that maximizes the discounted cumulative reward [of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to performing the actions]. Therefore, we should encourage the agent to come closer to the target by giving an appropriate reward. We propose to use a simple binary reward function, where a positive reward is given if an action leads the agent closer to target landmark, and a negative reward is given otherwise.”) AI, Branavan and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of AI with teaching of Branavan and Karaletsos to improve learning speed and robustness, while the reconstructed global policy ensures coordinated behavior across the entire task space (AI, Abstract). Branavan in view of Karaletsos and Al do not teach: a tree structure to define relationships between the plurality of actions and the set of future world states; Simonyan teaches: a tree structure to define relationships between the plurality of actions and (Simonyan, “[0095] To expand the leaf node, the system may add a respective new edge to the state tree for each action that is a valid action to be performed by the agent [a tree structure to define relationships between the plurality of actions] in response to a leaf observation characterizing the state represented by the leaf node. The system also initializes the edge data for each new edge by setting the visit count and action scores for the new edge to zero.”) … the set of future world states; (Simonyan, “[0081] The system generates a target action selection output for the observation by performing a look ahead search using the neural network and in accordance with current values of the neural network (step 304). In particular, the system performs the look ahead search to traverse possible future states of the environment starting from the current state characterized by the current observation [… the set of future world states]. The system continues the look ahead search until a possible future state that satisfies termination criteria is encountered. For example, the look ahead search may be a tree search and the criteria may be that the future state is represented by a leaf node in the state tree. Performing the look ahead search will be described in more detail below with reference to FIG. 4.”) Simonyan, Branavan, Karaletsos and AI are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Simonyan with teaching of Branavan, Karaletsos and AI to add selection process, using relevant text associated with the current state to bias the selection toward promising actions to improve action-selection efficiency and performance by narrowing or guiding the search toward actions that are more consistent with the contextual information conveyed by the text. (Simonyan, Abstract). Regarding claim 10, Branavan in view of Karaletsos, Al and Simonyan teach the method of claim 9. Branavan further teaches: wherein a reward distribution of the set of reward distributions represents (Branavan, page: 670, “Reward function [wherein a reward distribution of the set of reward distributions represents], R(s) PNG media_image18.png 28 24 media_image18.png Greyscale R, is the immediate reward received when transitioning to state s. The value of the reward correlates with the goodness of actions executed up to now, with higher reward indicating better actions. All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a.”) Karaletsos further teaches: a predicted reward distribution and at least a relative distance to a goal as a result of being in a particular future world state based at least in part on performing an action of the plurality of actions. (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [and at least a relative distance to a goal as a result of being in a particular future world state] induced by proposed action sequences at, at+1, . . . , at+h [based at least in part on performing an action of the plurality of actions]. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence [a predicted reward distribution]. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan, Al and Simonyan for the same reasons disclosed for claim 9. Regarding claim 11, Branavan in view of Karaletsos, Al and Simonyan teach the method of claim 10. Branavan further teaches: wherein the goal is determined based at least in part on a natural language task. (Branavan, page: 667, “While our method is also driven by control feedback, our language interpretation task itself is fundamentally different. We assume that the given text document provides high level advice without directly describing the correct actions for every potential game state. Furthermore, the textual advice does not necessarily translate to a single strategy – in fact, the text may describe several strategies, each contingent on specific game states. For this reason, the strategy text cannot simply be interpreted directly into a policy. Therefore, our goal [wherein the goal] is to bias a learned policy using information extracted from text [is determined based at least in part on a natural language task] (i.e.: using natural language (textual information) to bias or influence a learned policy, that the goal or objective in decision-making is influenced by insights derived from the textual advice). To this end, we do not aim to achieve a complete semantic interpretation, but rather use a partial text analysis to compute features relevant for the control application.”) Regarding Claim 12, Branavan in view of Karaletsos, AI and Simonyan teach the method of claim 9. Karaletsos further teaches: wherein the one or more processors are to use the dynamics model to predict one or more effects of the plurality of actions on the set of future world states. (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [on the set of future world states] induced by proposed action sequences at, at+1, . . . , at+h [wherein the one or more processors are to use the dynamics model to predict one or more effects of the plurality of actions]. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan, Al and Simonyan for the same reasons disclosed for claim 9. Regarding claim 13, Branavan in view of Karaletsos, Al, and Simonyan teach the method of claim 9. Branavan further teaches: wherein the action proposal model further comprises a policy model. (Branavan, page: 670, “The game playing agent selects actions [wherein the action proposal model]according to a stochastic policy π(s, a), [further comprises a policy model] which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image19.png 18 18 media_image19.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ (s, a) is known, the optimal gameplaying behavior would be to select the action a with the highest Qπ PNG media_image20.png 18 18 media_image20.png Greyscale (s, a). While it may be computationally hard to find an optimal policy π PNG media_image8.png 19 18 media_image8.png Greyscale (s, a) or Qπ PNG media_image12.png 19 18 media_image12.png Greyscale (s, a), many well studied algorithms are available for estimating an effective approximation (Sutton & Barto, 1998)”). Regarding claim 14, Branavan in view of Karaletsos, Al, and Simonyan teach the method of claim 9. Branavan further teaches: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to: (Branavan, page: 683, “The experiments were run on typical desktop PCs [as a result of being executed by the one or more processors, cause the one or more processors to] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [wherein one or more memories further store instructions that].”) create a set of edges from a root node to a set of leaf nodes representing a plurality of actions of the set of actions; [AltContent: rect][AltContent: rect][AltContent: rect] PNG media_image21.png 234 746 media_image21.png Greyscale [AltContent: textbox ([to a set of leaf nodes representing a plurality of actions of the set of actions])][AltContent: textbox ([create a set of edges])][AltContent: textbox ([from a root node])](Branavan, Fig. 2) obtain the set of actions from the action proposal model; (Branavan, page: 671, “Figure 3: Overview of Monte-Carlo Search algorithm. For each game state st , an independent set of simulated games or roll-outs are done to find the best possible game action at . Each roll-out starts at state st , with actions [obtain the set of actions from the action proposal model] selected according to a simulation policy π(s, a). This policy is learned from the roll-outs themselves – with the roll-outs improving the policy, which in turn improves roll-out action selection. The process is repeated for every actual game state, with the simulation policy being relearned from scratch each time.”) [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] PNG media_image23.png 87 1000 media_image23.png Greyscale [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect] PNG media_image24.png 387 381 media_image24.png Greyscale PNG media_image25.png 188 458 media_image25.png Greyscale [AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]annotate the set of leaf nodes to indicate values of the set of values and edges of the set of edges with reward distributions of the set of reward distributions. Karaletsos further teaches: obtain the set of future world states from the dynamics model representing one or more future world states upon completion of the plurality of actions; (Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h [obtain the set of future world states from the dynamics model] induced by proposed action sequences at, at+1, . . . , at+h [representing one or more future world states upon completion of the plurality of actions]. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan, Al and Simonyan for the same reasons disclosed for claim 9. Regarding claim 16, Branavan in view of Karaletsos, Al and Simonyan teach the method of claim 14. Branavan further teaches: wherein the values of the set of values further comprise a first distribution of the set of values. (Branavan, page: 670, “Reward function [set of rewards], R(s) PNG media_image29.png 28 24 media_image29.png Greyscale R, is the immediate reward received when transitioning to state s [a first distribution of the set of values] (i.e.: refers to the action-value function, which represents the expected reward and can be considered as having a distribution because it is based on expected outcomes over potentially many trials and different state-action pairs). The value of the reward correlates with the goodness of actions [wherein the values of the set of values further comprise] executed up to now, with higher reward indicating better actions. All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a.”) Claim(s) 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI, Simonyan and in further view of XIAO et al., Pub. No.: US20200234167. Regarding claim 15, Branavan in view of Karaletsos, AI and Simonyan teach the method of claim 14. Branavan further teaches: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors (Branavan, page: 683, “The experiments were run on typical desktop PCs [as a result of being executed by the one or more processors, cause the one or more processors] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [wherein one or more memories further store instructions that].”) Branavan in view of Karaletsos, AI and Simonyan do not teach: to perform a backup operation to update the set of leaf nodes based at least in part on the set of values and the set of rewards. XIAO teaches: to perform a backup operation to update the set of leaf nodes based at least in part on the set of values and the set of rewards. (XIAO, “[0078] Further, in the present embodiment, the method may include selecting actions to take until a node s is identified, where ξ(s) PNG media_image30.png 28 24 media_image30.png Greyscale L. The method may include utilizing the stochastic value function at a node s and a result of the stochastic value function may be used to update the search tree [update the set of leaf nodes based at least in part on the set of values and the set of rewards] as in the above-described in the backup operation [to perform a backup operation]. In the present example, because each iteration may add a separate path of nodes from the root to the leaf [leaf nodes] of T to the current search tree, the present example may be called a path-REWT.”) XIAO, Branavan, Karaletsos, AI and Simonyan are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of XIAO with teachings of Branavan, Karaletsos, AI and Simonyan to enhance the reinforcement learning system by updating a multi-level data structure to improve agent control (XIAO, ¶[0004]). Regarding claim 17 Branavan in view of Karaletsos, AI and Simonyan teach the method of claim 16. Branavan further teaches: based at least in part on the first distribution and reward distributions of the set of reward distributions. (Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [based at least in part on the first distribution and reward distributions of the set of reward distributions] and then following policy π is termed the actionvalue function Qπ (s, a). Our goal is to find the optimal policy π PNG media_image31.png 19 18 media_image31.png Greyscale (s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”) Branavan in view of Karaletsos, AI and Simonyan do not teach: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to perform a backup operation to update the set of leaf nodes XIAO teaches: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to perform a backup operation to update the set of leaf nodes (XIAO, “[0078] Further, in the present embodiment, the method may include selecting actions to take until a node s is identified, where ξ(s) PNG media_image30.png 28 24 media_image30.png Greyscale L. The method may include utilizing the stochastic value function at a node s and a result of the stochastic value function may be used to update the search tree [to update the set of leaf nodes] as in the above-described in the backup operation [to perform a backup operation]. In the present example, because each iteration may add a separate path of nodes from the root to the leaf of T to the current search tree, the present example may be called a path-REWT. [0079] To illustrate a theoretical analysis of convergence property for operations associated with MENTS, for any node in a search tree, after its subtree has been explored, the estimated softmax value may converge to an optimal value at an exponential rate. Recall that in Theorem 1, an optimal sampling algorithm for the softmax stochastic bandit problem may guarantee limt→∞Nt(a)/t=πsft*(a) for any action a. This may be shown in E2 W with high probability and may be based on the proof of Theorem 2.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of XIAO with teaching of Branavan, Karaletsos, AI and Simonyan for the same reasons disclosed for claim 15. Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI and in further view of Vernade et al., Pub. No.: US20210158196A1. Regarding claim 26, Branavan in view of Karaletsos and AI teach the method of claim 25. Branavan in view of Karaletsos and AI do not teach: selecting the action comprises computing a Q-distribution based, at least in part, on the value and the set of distribution of rewards Vernade teaches: selecting the action comprises computing a Q-distribution based, at least in part, on the value and the set of distribution of rewards (Vernade, “[0044] In particular, an action selection engine 110 [selecting the action] maintains count data 150 and uses the maintained count data 150 to select actions 122 that optimize expected rewards, i.e., that optimize the expected delayed reward 124 to be received in response to performing an action given the current transition probability distribution [comprises computing a Q distribution] and the stationary reward distribution [based, at least in part, on the value and the set of distribution of rewards].”) Vernade, Branavan, Karaletsos and AI are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Vernade with teachings of Branavan, Karaletsos and AI to improve the ability to choose actions that align with high-level goals or situational understanding. (Vernade, Abstract). Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI and in further view of XIAO et al., Pub. No.: US20200234167. Regarding claim 28, Branavan in view of Karaletsos and AI teach the method of claim 23. Branavan in view of Karaletsos and AI do not teach: wherein generating the tree further comprises performing a backup operation to surface a highest predicted value associated with the set of proposed actions. XIAO teaches: wherein generating the tree further comprises performing a backup operation to surface a highest predicted value associated with the set of proposed actions. (XIAO, “[0068] A node s that ξ(s)∈ T/L may contain edges (s, a) for all actions a ∈ A(s) and may store a set of statistics, including a softmax state value estimation Vsft(s). In some embodiments, for each edge {N(s, a), {circumflex over (Q)}sft(s, a)}, N(s,a) may be the visit count and Qsft*(s, a) may be the softmax state-action value estimation [to surface a highest predicted value associated with the set of proposed actions] (i.e.: softmax function is primarily used for probabilistic action selection, the underlying Q(s, a) values can be directly used to identify the highest predicted value associated with a set of proposed actions. Thus, the action with the highest Q -value can be surfaced as the best predicted action). [0069]In some embodiments, operations of REWT may include iterations of 4 operations, including a select operation, an evaluate operation, an expand operation, and a backup operation [wherein generating the tree further comprises performing a backup operation]. [0070] At the select operation, the method may include generating a trajectory of nodes based on the REW method. Generating the trajectory of nodes may include a series of nodes including a root node of the search tree, so, and ending with a leaf node, sL, of the search tree.”) XIAO, Branavan, Karaletsos and AI are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of XIAO with teachings of Branavan, Karaletsos and AI to enhance the reinforcement learning system by updating a multi-level data structure to improve agent control (XIAO, ¶[0004]). Claim(s) 29 – 30 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI, XIAO and in further view of Ren et al., “Policy Optimization for Spoken Dialog Management Using Genetic Algorithm.” Regarding claim 29, Branavan in view of Karaletsos, AI and XIAO teach the method of claim 28. Branavan in view of Karaletsos, AI and XIAO do not teach: wherein the backup operation further comprises a Bellman backup operation. Ren teaches: wherein the backup operation further comprises a Bellman backup operation. (Ren, page: 5, “3.4 DM Policy Evaluation Using Dialog Corpora We describe a DM policy performance evaluation method on dialog corpus to avoid the need for deploying the DM on-line. We estimate a policy by learning its value function using a similar iterative routine to FQI on a held-out testing dialog corpus. The estimated cumulative reward when following the policy is used as a metric for performance. A similar approach has been taken in evaluating the effect of dialog state tracker on end-to-end performance of SDS [23]. To estimate the Q-function of a DM policy, the Bellman backup operation [wherein the backup operation further comprises a Bellman backup operation] (Line 10) in Algorithm 2 is changed to: Qi,t ← ri,t + γ ˆQ(si,t+1, π(si,t+1)), (4) where π is the DM policy to evaluate. After the value function converges, the average reward for all the training samples of initial dialog turn 1 N i Qi,0 is used as a metric for performance.”) Ren, Branavan, Karaletsos, AI and XIAO are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Ren with teachings of Branavan, Karaletsos, AI and XIAO to ensure effective dialog management in noisy environments (Ren, Abstract). Regarding claim 30, Branavan in view of Karaletsos, AI and XIAO teach the method of claim 28. Branavan in view of Karaletsos, AI and XIAO do not teach: wherein the backup operation further comprises backing up through the tree the distribution of rewards Ren teaches: wherein the backup operation further comprises backing up through the tree the distribution of rewards (Ren, page: 5, “3.4 DM Policy Evaluation Using Dialog Corpora We describe a DM policy performance evaluation method on dialog corpus to avoid the need for deploying the DM on-line. We estimate a policy by learning its value function using a similar iterative routine to FQI on a held-out testing dialog corpus. The estimated cumulative reward when following the policy is used as a metric for performance. A similar approach has been taken in evaluating the effect of dialog state tracker on end-to-end performance of SDS [23]. To estimate the Q-function of a DM policy, the Bellman backup operation [wherein the backup operation further comprises backing up through the tree] (Line 10) in Algorithm 2 is changed to: PNG media_image32.png 28 426 media_image32.png Greyscale [the distribution of rewards] (i.e.: r ti is the immediate reward received after taking action). (i.e.: represents the discounted future value estimate. This is the second distribution of values because it considers the expected future rewards (values) from state \( s_{i,t+1} \) onward, influenced by the policy π). where π is the DM policy to evaluate. After the value function converges, the average reward for all the training samples of initial dialog turn 1 N i Qi,0 is used as a metric for performance.”) It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Ren with teachings of Branavan, Karaletsos, AI and XIAO for the same reasons disclosed for claim 29. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Zhu, et al. "Target-driven visual navigation in indoor scenes using deep reinforcement learning" , 2017. An actor-critic model that uses both the goal and the current state to improve generalization. Introduce the AI2-THOR framework, which offers a rich environment with high-quality 3D scenes and a physics engine. This framework lets agents perform actions and interact with objects, allowing us to gather a large number of training samples effectively.. Kempka, et al., "Vizdoom: A doom-based ai research platform for visual reinforcement learning.", (2016). A platform for reinforcement learning research that uses raw visual information in a semirealistic 3D world. This software, called ViZDoom, is based on the classic video game Doom and enables the development of bots that play the game using the screen visuals. ViZDoom is lightweight, fast, and easily customizable through user-defined scenarios. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATIYAS T MARU whose telephone number is (571)270-0902 or via email: matiyas.maru@uspto.gov. The examiner can normally be reached Monday - Friday (8:00am - 4:00pm) EST. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization were this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /M.T.M./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148