DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/04/2026 has been entered.
Examiner’s Note
In regards to the 35 USC § 101 rejection, has been withdrawn in light of the instant amendments to the claim. The amended claims recite limitations which integrate the judicial exception into practical application, causing the autonomous agents to move in response to performing the selected action. Consistent with MPEP 2106.05(b), the claim relies on a specific technological implementation in which the autonomous agents are used to carried out the recited functionality and the claimed step.
Response to argument
Applicant's arguments filed 03/04/2026 (Remarks/Arguments: (pgs. 13 – 15)) have been fully considered but they are not persuasive.
Applicant’s arguments with respect to amended claim(s) have been considered but are moot, because arguments/remarks are directed to amended claim limitations that were not previously examined by the examiner. The rejections are noted in the current office action to address amended claim limitations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 3, 6 – 7, 18 – 25, 27 and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan, et al., "Learning to win by reading manuals in a monte-carlo framework." in view of Al, et al., "Partial policy-based reinforcement learning for anatomical landmark localization in 3d medical images.", Bennice et al., Pub. No.: US20220288782A1 and Karaletsos et al., Pub. No.: US20200372410A1.
Regarding claim 1, Branavan teaches: A method comprising: performing a language guided tree search using one or more neural networks to expand a search tree
(Branavan, page: 663, “Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the Monte-Carlo Search [tree search] framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward from these simulations to update the parameters of our neural network [using one or more neural networks to expand a search tree] via backpropagation. This focuses learning on the current game state, allowing our method to learn language analysis [performing a language-guided] and game-play appropriate to the observed game context.”)
to select a selected action of a set of actions associated with a goal
(Branavan, page: 664, “Assume that every state s is represented by a set of n features [s1, s2, . . . , sn]. Given a state s, our goal is to select the best possible action aj from a fixed set A [to select a selected action of a set of actions associated with a goal]. We can model this task as multiclass classification, where each choice aj is represented by a feature vector [(s1, aj ),(s2, aj ), . . . ,(sn, aj )]. Here, (si , aj ), i
PNG
media_image1.png
28
24
media_image1.png
Greyscale
[1, n] represents a feature created by taking the Cartesian product between [s1, s2, . . . , sn] and aj . To learn this classifier effectively, we need a training set that sufficiently covers the possible combinations of state features and actions. However, in do1mains with complex state spaces and a large number of possible actions, many instances of state-action feature values will be unobserved in training.”)
causing the one or more autonomous agents to perform the selected action,
(Branavan, page: 670, “The game playing agent [causing the one or more autonomous agents] selects actions [to perform the selected action] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a).”)
based at least in part, on the expansion of the search tree.
(Branavan, Fig. 3)
PNG
media_image2.png
321
203
media_image2.png
Greyscale
PNG
media_image3.png
139
575
media_image3.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
Branavan does not teach:
based, at least in part, on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the set of actions, a particular one of the set of distributions of rewards generated as a result of a performance of a particular action of the set of actions
causing the one or more autonomous agents to move in response to performing the selected action.
given a particular state represented by one or more voxels that indicate a position of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D)
Karaletsos teaches:
based at least in part, on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the set of actions, a particular one of the set of distributions of rewards generated as a result of a performance of a particular action of the set of actions
(Karaletsos, “[0076] In equation 6, the term p(zd) represents the distribution of the latent variable zd, the term p(zr) represents the distribution of the latent variable zr, and the term p(s0) represents the distribution of the state s0, the term [p(rt+1|st, at, st+1, zr) represents the distribution of the reward rt+1 [on a distribution of rewards from a set of distributions of rewards comprising a different distribution of rewards associated with each action of the set of actions], the term p(st+1|st, at, zd) represents the distribution of the state st+1, and the term p(at|st, zr, zd) represents the distribution of actions at [a particular one of the set of distributions of rewards generated as a result of a performance of a particular action of the set of actions]. This structure of the model facilitates solving tasks where both of these aspects (dynamics and reward) can vary independently.”)
causing the one or more autonomous agents to move in response to performing the selected action.
(Karaletsos, “[0045] An action represents a move that the agent can make. An agent selects from a set of possible actions [causing the one or more autonomous agents to move in response to performing the selected action]. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives. If the system is part of a drone, the set of actions includes increasing speed, decreasing speed, changing direction, and so on. If the system is part of a robot, the set of actions includes walking forward, turning left or right, climbing, and so on. If the system is part of a self-driving vehicle, the set of actions includes driving the vehicle, stopping the vehicle, accelerating the vehicle, turning left/right, changing gears of the vehicle, changing lanes, and so on.”)
Karaletsos and Branavan are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Karaletsos with teaching of Branavan to add latent variable based model to improve transferability and adaptability to new and unseen conditions. (Karaletsos, Abstract).
Branavan in view of Karaletsos do not teach:
given a particular state represented by one or more voxels that indicate a position of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D)
AI teaches:
a particular state represented by one or more voxels
(AI, page: 1247, “A. State: We represent the state [a particular state] as a function of the agent position. For any position q in the discrete voxel-space, [represented by one or more voxels] the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position.”)
that indicate a position
(AI, page: 1247, “B. Action Similar to Ghesu et al.’s approach [4], we use a discrete action space where the agent can take a unit step along either of the axes to update its position to a neighboring voxel. Therefore, the agent holds three degrees of freedom to move, enabling six actions i.e., right, le f t, up, down, slice_ forward, slice_backward. The first four moves are along the axial slice (X and Y axes), whereas, the last two actions allow the agent to jump across the slices moving along Z-axis. [that indicate a position] ”).
AI, Branavan and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of AI with teaching of Branavan and Karaletsos to improve learning speed and robustness, while the reconstructed global policy ensures coordinated behavior across the entire task space (AI, Abstract).
Branavan in view of Karaletsos and Al do not teach:
of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D)
Bennice teaches:
of a set of objects to interact with one or more autonomous agents during the performance of the action in three dimensions (3D)
(Bennice, “[0010] In some implementations, a computer implemented method may be provided that includes: simulating a three-dimensional (3D) environment, wherein the simulated 3D environment includes a plurality of simulated robots controlled by a single robot controller; rendering multiple instances of an interactive object in the simulated 3D environment, wherein each instance of the interactive object has a simulated physical characteristic that is unique among the multiple instances of the interactive object; and receiving, from the robot controller, a common set of joint commands to be issued to each of the plurality of simulated robots, wherein for each simulated robot of the plurality of simulated robots [to interact with one or more autonomous agents], the common command causes actuation of one or more joints of the simulated robot to interact [during the performance of the action] with a respective instance of the interactive object [of a set of objects] in the simulated 3D environment [in three dimensions (3D)].”)
Bennice, Branavan, Karaletsos and Al are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to substitute Bennice’s objects that the agent interacting with, are denoted in 3D environment (voxel-space) with AI’s teachings of the agents’ location represented by voxel-space. Accordingly, this falls within the substitution rationale outlined in KSR Int’I Co. V. Teleflex Inc. 550 U.S 398 (2007).
Regarding claim 2, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 1.
Branavan further teaches: wherein the set of actions are provided by an action proposal network. (Branavan, page: 664, “2. Learning Game Play from Text: In this section, we provide an intuitive explanation of how textual information can help improve action selection in a complex game. For clarity, we first discuss the benefits of textual information in the supervised scenario, thereby decoupling questions concerning modeling and representation from those related to parameter estimation. Assume that every state s is represented by a set of n features [s1, s2, . . . , sn]. Given a state s, our goal is to select the best possible action aj [wherein the set of actions are provided by an action proposal network] from a fixed set A. We can model this task as multiclass classification, where each choice aj is represented by a feature vector [(s1, aj ),(s2, aj ), . . . ,(sn, aj )]. Here, (si , aj ), i
PNG
media_image1.png
28
24
media_image1.png
Greyscale
[1, n] represents a feature created by taking the Cartesian product between [s1, s2, . . . , sn] and aj . To learn this classifier effectively, we need a training set that sufficiently covers the possible combinations of state features and actions. However, in domains with complex state spaces and a large number of possible actions, many instances of state-action feature values will be unobserved in training.”)
Regarding claim 3, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 1.
Branavan further teaches: wherein the search tree further comprises a first edge representing the selected action and a node including a representation of a world state conditioned on performing the selected action.
(Branavan, Fig. 2)
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: ]
PNG
media_image6.png
234
502
media_image6.png
Greyscale
[AltContent: textbox ([wherein the search tree further comprises a first edge representing the selected action])][AltContent: textbox ([and a node including a representation of a world state conditioned on performing the selected action])]
(“Figure 2: Markov Decision Process. Actions are selected according to policy function π(s, a) given the current state s. The execution of the selected action ai (e.g., a1), causes the MDP to transition to a new state s 0 according to the stochastic state transition distribution T(s 0 | s, a).”)
Regarding claim 6, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 1.
Branavan further teaches: wherein the goal is determined based at least in part on natural language.
(Branavan, page: 667, “While our method is also driven by control feedback, our language interpretation task itself is fundamentally different. We assume that the given text document provides high level advice without directly describing the correct actions for every potential game state. Furthermore, the textual advice does not necessarily translate to a single strategy – in fact, the text may describe several strategies, each contingent on specific game states. For this reason, the strategy text cannot simply be interpreted directly into a policy. Therefore, our goal [wherein the goal] is to bias a learned policy using information extracted from text [is determined based at least in part on natural language] (i.e.: using natural language (textual information) to bias or influence a learned policy, that the goal or objective in decision-making is influenced by insights derived from the textual advice). To this end, we do not aim to achieve a complete semantic interpretation, but rather use a partial text analysis to compute features relevant for the control application.”)
Regarding claim 7, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 6.
Branavan further teaches: wherein the goal is a long horizon goal comprising one or more tasks determined based at least in part on the natural language.
(Branavan, page: 672, “5. Adding Linguistic Knowledge to the Monte-Carlo Framework: The goal
[wherein the goal is a long horizon goal] of our work is to improve the performance of the Monte-
Carlo Search framework described above, using information automatically extracted from text
[based at least in part on the natural language]. In this section, we describe how we achieve this in terms of model structure and parameter estimation. 5.1 Model Structure: To achieve our aim of leveraging textual information to improve game-play, our method needs to perform three tasks [comprising one or more tasks determined]: (1) identify sentences relevant to the current game state, (2) label sentences with a predicate structure, and (3) predict good game actions by combining game features with text features extracted via the language analysis steps. We first describe how each of these tasks can be modeled separately before showing how we integrate them into a single coherent model.”)
Regarding claim 18, Branavan teaches: An automated agent comprising:
(Branavan, page: 672, “In fact, in our approach, we use standard gradient descent updates from RL to estimate the parameters of Q(s, a). There is, however, one crucial difference between these two techniques: In general, the goal in RL is to find a Q(s, a) applicable to any state the agent [An automated agent] may observe during its existence. In the Monte-Carlo Search framework, the aim is to learn a Q(s, a) specialized to the current state s.”)
one or more circuits
(Branavan, page: 683, “The experiments were run on typical desktop PCs [one or more circuits] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU).”)
to perform natural language goal-directed tasks using one or more neural networks
(Branavan, page: 663, “1.1 Summary of the Approach: We address the above challenges in a unified framework based on Markov Decision Processes (MDP), a formulation commonly used for game playing algorithms. This setup consists of a game in a stochastic environment, where the goal of the player [goal-directed tasks] is to maximize a given utility function R(s) at state s. The player’s behavior is determined by an action-value function Q(s, a) [at least in part on performance of an action of a set of actions given a particular state] that assesses the goodness of action a at state s based on the attributes of s and a. To incorporate linguistic information into the MDP formulation, we expand the action value function to include linguistic features. While state and action features are known at each point of computation, relevant words and their semantic roles are not observed. Therefore, we model text relevance as a hidden variable. Similarly, we use hidden variables to discriminate the words that describe actions and those that describe state attributes from the rest of the sentence. To incorporate these hidden variables in our action-value function, we model Q(s, a) as a non-linear function approximation using a multi-layer neural network. Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the MonteCarlo Search framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward [to determine a future reward] from these simulations to update the parameters of our neural network [using one or more neural networks] via backpropagation. This focuses learning on the current game state, allowing our method to learn language analysis [to perform natural language] and game-play appropriate to the observed game context.”)
to execute a language-guided tree search.
(Branavan, page: 669, “3.3.2 Estimating the Value of Untried Actions Previous approaches to estimating the value of untried actions have relied on two techniques. The first, Upper Confidence bounds for Tree (UCT) is a heuristic used in concert with the Monte-Carlo Tree Search [tree search] variant of MCS. It augments an action’s value with an exploration bonus for rarely visited stateaction pairs, resulting in better action selection and better overall game performance (Gelly et al., 2006; Sturtevant, 2008; Balla & Fern, 2009). The second technique is to learn a linear function approximation of action values for the current state s, based on game feedback (Tesauro & Galperin, 1996; Silver, Sutton, & Muller, 2008). Even though our method follows the latter approach, we model action-value Q(s, a) via a non-linear function approximation. Given the complexity of our application domain, this non-linear approximation generalizes better than a linear one, and as shown by our results significantly improves performance. More importantly, the non-linear model enables our method to represent text analysis [to execute a language-guided] as latent variables, allowing it to use textual information to estimate the value of untried actions.”)
and reward distributions generated based, at least in part, on the state and at least one of the goal-directed tasks
(Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s,
a), which specifies the probability of selecting action a in state s. The expected total reward [and at least one of the goal-directed tasks] after executing action a in state s, [reward distributions generated based, at least in part, on the state] and then following policy π is termed the actionvalue function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image8.png
19
18
media_image8.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”)
Branavan does not teach:
the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks,
cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks.
at least in part, on a state represented by one or more voxels, the one or more voxels indicating a position.
the one or more voxels indicating a position of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed;
Karaletsos teaches:
the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks
(Karaletsos, “[0075] In some embodiments, the machine learning model uses multiple plated variables which constitute the structured latent space of the GHP-MDP. Separate latent spaces for dynamics and reward allow agents to pursue different goals across environments with different dynamics [the reward distributions comprising a different reward distribution associated with each of the goal-directed tasks]. The joint model p(s0:T+1, a0:T, R, zd, zr), including the action distribution implied by control, is described using equation (6).”)
cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks
(Karaletsos, “[0045] An action represents a move that the agent can make [cause the automated agent to move as a result of performing at least one of the natural language goal-directed tasks]. An agent selects from a set of possible actions. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives…”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan for the same reasons disclosed for claim 1.
Branavan in view of Karaletsos do not teach:
at least in part, on a state represented by one or more voxels, the one or more voxels indicating a position.
the one or more voxels indicating a position of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed;
AI teaches:
at least in part, on a state represented by one or more voxels
(AI, page: 1247, “A. State: We represent the state [at least in part, on a state] as a function of the agent-position. For any position q in the discrete voxel-space, [represented by one or more voxels] the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position.”)
the one or more voxels indicating a position
(AI, page: 1247, “B. Action Similar to Ghesu et al.’s approach [4], we use a discrete action space where the agent can take a unit step along either of the axes to update its position to a neighboring voxel [the one or more voxels indicating a position]. Therefore, the agent holds three degrees of freedom to move, enabling six actions i.e., right, le f t, up, down, slice_ forward, slice_backward. The first four moves are along the axial slice (X and Y axes), whereas, the last two actions allow the agent to jump across the slices moving along Z-axis.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of AI with teaching of Branavan and Karaletsos for the same reasons disclosed for claim 1.
Branavan in view of Karaletsos and Al do not teach:
of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed.
Bennice teaches:
of an object to interact with the automated agent in three dimensions (3D) once the natural language goal-directed tasks are performed.
(Bennice, “[0010] In some implementations, a computer implemented method may be provided that includes: simulating a three-dimensional (3D) environment, wherein the simulated 3D environment includes a plurality of simulated robots controlled by a single robot controller; rendering multiple instances of an interactive object in the simulated 3D environment [in three dimensions (3D)], wherein each instance of the interactive object [of an object] has a simulated physical characteristic that is unique among the multiple instances of the interactive object; and receiving, from the robot controller, a common set of joint commands to be issued [once the natural language goal-directed tasks are performed] to each of the plurality of simulated robots [to interact with one or more autonomous agents], wherein for each simulated robot of the plurality of simulated robots, the common command causes actuation of one or more joints of the simulated robot to interact with a respective instance of the interactive object in the simulated 3D environment.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Bennice with teaching of Branavan, Karaletsos and Al for the same reasons disclosed for claim 1.
Regarding claim 19, Branavan in view of Karaletsos, AI and Bennice teaches the method of claim 18.
Bennice teaches: wherein the automated agent further comprises a robot.
(Bennice, “[0008] In some implementations, the outcomes may be analyzed to ascertain inherent tolerances of component(s) of the robot controller and/or the real-world robot it represents. For example, it may be observed that the robot [wherein the automated agent further comprises a robot] is able to successfully interact with instances of the interactive object with poses that are within some translational and/or rotational tolerance of the baseline. Outside of those tolerances, the simulated robot may fail.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Bennice with teachings of Branavan, Karaletsos and Al for the same reasons disclosed for claim 1.
Regarding claim 20, Branavan in view Karaletsos, AI and Bennice teach the method of claim 19.
Karaletsos further teaches: performing the at least one of the natural language goal-directed tasks causes the robot to perform an action comprising moving.
(Karaletsos, “[0045] An action represents a move that the agent can make [performing the at least one of the natural language goal-directed tasks causes the robot to perform an action comprising moving]. An agent selects from a set of possible actions. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives…”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan, AI and Bennice for the same reasons disclosed for claim 1.
Regarding claim 21, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 20. Branavan further teaches: wherein the action is proposed by a policy model of the one or more neural networks.
(Branavan, page: 670, “The game playing agent selects actions [wherein the action is proposed] according to a stochastic policy [is proposed by a policy model of the one or more neural networks]
π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ
(s, a). Our goal is to find the optimal policy π
PNG
media_image8.png
19
18
media_image8.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ (s, a) is known, the optimal game-playing behavior would be to select the action a with the highest Qπ
PNG
media_image9.png
19
18
media_image9.png
Greyscale
(s, a).”)
Regarding claim 22, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 20.
Branavan further teaches: wherein the action is selected based at least in part on a value associated with the action determined by a value model of the one or more neural networks.
(Branavan, page: 670, “The game playing agent selects actions [wherein the action is selected] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a [based at least in part on a value associated with the action] (i.e.: the action is chosen based on the expected total reward (action-value function), which is a value associated with the action) in state s, and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image10.png
19
18
media_image10.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ
PNG
media_image11.png
18
17
media_image11.png
Greyscale
(s, a) is known, the optimal game-playing behavior would be to select the action a with the highest Qπ
PNG
media_image12.png
19
18
media_image12.png
Greyscale
(s, a) [determined by a value model of the one or more neural networks].”)
Regarding claim 23, Branavan teaches: A processor comprising one or more circuits to:
corresponds to a distribution of rewards generated based, at least in part, on the action and the future state;
(Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after
executing action a in state s, [corresponds to a distribution of rewards generated based, at least in part, on the action and the future state] and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image13.png
19
18
media_image13.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”)
The rest of the limitations are analogous to claim 18, so are rejected under similar rationale.
Regarding claim 24, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 23.
Branavan further teaches: wherein the action represents an edge between two nodes of the tree (Branavan, Fig. 3)
PNG
media_image14.png
267
429
media_image14.png
Greyscale
PNG
media_image15.png
205
446
media_image15.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
and is annotated with a the distribution of rewards representing a utility as a result of being in the future state.
(Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s,
a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [and is annotated with a the distribution of rewards representing a utility as a result of being in the future state] and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image13.png
19
18
media_image13.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”)
Regarding claim 25, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 24.
Branavan further teaches: wherein the nodes include a value independent of the distribution of rewards and representing the value of being in the future state.
(Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s,
a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [wherein the nodes include a value independent of the distribution of rewards and representing the value of being in the future state] and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image10.png
19
18
media_image10.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”)
Regarding claim 27, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 23.
Bennice further teaches: wherein the one or more autonomous agents comprise a robot
(Bennice, “[0008] In some implementations, the outcomes may be analyzed to ascertain inherent tolerances of component(s) of the robot controller and/or the real-world robot it represents [wherein the one or more autonomous agents comprise a robot]. For example, it may be observed that the robot is able to successfully interact with instances of the interactive object with poses that are within some translational and/or rotational tolerance of the baseline. Outside of those tolerances, the simulated robot may fail.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Bennice with teaching of Branavan, Karaletsos and Al for the same reasons disclosed for claim 1.
Regarding claim 31, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 23.
Branavan further teaches: wherein the goal is determined based, at least in part, on unstructured natural language instructions.
(Branavan, page: 672, “5. Adding Linguistic Knowledge to the Monte-Carlo Framework: The goal [wherein the goal] of our work is to improve the performance of the Monte-Carlo Search framework described above, using information automatically extracted from text [is determined based, at least in part, on unstructured natural language instructions] (i.e.: the extracted text can come from various sources, such as game manuals, dialogues, user instructions, or any relevant documentation (unstructured natural language instruction)). In this section, we describe how we achieve this in terms of model structure and parameter estimation. 5.1 Model Structure To achieve our aim of leveraging textual information to improve game-play, our method needs to perform three tasks: (1) identify sentences relevant to the current game state, (2) label sentences with a predicate structure (i.e.: step of structuring extracted text), and (3) predict good game actions by combining game features with text features extracted via the language analysis steps. We first describe how each of these tasks can be modeled separately before showing how we integrate them into a single coherent model.”)
Claim(s) 4, 8 – 11, 13 – 14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI, Bennice and in further view of Venkatraman et al. "Improved learning of dynamics models for control."
Regarding claim 4, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 3.
Branavan in view of Karaletsos, AI and Bennice do not teach:
wherein the representation is generated by a dynamics model.
Venkatraman teaches:
wherein the representation is generated by a dynamics model.
(Venkatraman, page: 3, “We specifically refer to the left loop in Fig. 1 as the DAgger (Data-set Aggregation) system identification learning framework [18]. A key difference lies in the aggregation step of the procedure in order to provide model agnostic guarantees. At the beginning of the algorithm, DAgger initializes an empty training data-set and an exploration policy πexplore(xt) that generates an action (control) ut given a state xt. This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state-action pairs; (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [wherein the representation is generated by a dynamics model] (i.e.: the dynamics model is learned from the aggregated dataset, which represents how the system transitions from one state to the next given an action) ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated. During the execution of the first DAgger loop, the state distribution induced by π can greatly differ from the initial π explore; the first generated policies may perform poorly due to inaccuracies in ˆf. The iterative procedure refines the dynamics model by aggregating data from states induced by running the system with π1, . . . , πN .”)
Venkatraman, Branavan, Karaletsos, AI and Bennice are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Venkatraman with teachings of Branavan, Karaletsos, AI and Bennice to enhance a reinforcement learning agent by allowing it to simulate and adapt to complex dynamics. (Venkatraman, page: 1).
Regarding claim 8, Branavan in view of Karaletsos, AI and Bennice teach the method of claim 1.
Branavan further teaches:
wherein expanding the search tree further comprises, for a first node of the search tree:
obtaining the set of actions from an action proposal model;
creating an edge from the first node to a second node representing the selected action;
annotating the second node to indicate a value associated with the world state and the edge with a reward associated with the selected action.
obtaining a state representation from a dynamics model representing a world state upon completion of the selected action
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
PNG
media_image18.png
387
642
media_image18.png
Greyscale
[AltContent: textbox ([creating an edge from the first node to a second node representing the selected action])][AltContent: textbox ([annotating the second node to indicate a value associated with the world state and the edge with a reward associated with the selected action])]
(Branavan, page: 671, “Figure 3: Overview of Monte-Carlo Search algorithm. For each game state st , an independent set of simulated games or roll-outs are done to find the best possible game action
at . Each roll-out starts at state st , with actions [for a first node of the search tree: obtaining the set of actions from an action proposal model] selected according to a simulation policy π(s, a). This policy is learned from the roll-outs themselves – with the roll-outs improving the policy, which in turn improves roll-out action selection. The process is repeated for every actual game state, with the simulation policy being relearned from scratch each time.”)
Branavan in view of Karaletsos, AI and Bennice do not teach:
obtaining a state representation from a dynamics model representing a world state upon completion of the selected action;
Venkatraman teaches:
obtaining a state representation from a dynamics model representing a world state upon completion of the selected action;
(Venkatraman, page: 3, “We specifically refer to the left loop in Fig. 1 as the DAgger (Data-set Aggregation) system identification learning framework [18]. A key difference lies in the aggregation step of the procedure in order to provide model agnostic guarantees. At the beginning of the algorithm, DAgger initializes an empty training data-set and an exploration policy πexplore(xt) that generates an action (control) ut given a state xt. [obtaining a state representation] This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state [representing a world state] -action pairs; [upon completion of the selected action] (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [from a dynamics model] ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated. During the execution of the first DAgger loop, the state distribution induced by π can greatly differ from the initial π explore; the first generated policies may perform poorly due to inaccuracies in ˆf. The iterative procedure refines the dynamics model by aggregating data from states induced by running the system with π1, . . . , πN .”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Venkatraman with teaching of Branavan, Karaletsos, Al and Bennice for the same reasons disclosed for claim 4.
Regarding claim 9, Branavan teaches: A system comprising:
one or more processors;
one or more memories to store parameters associated with the one or more neural networks;
(Branavan, page: 683, “The experiments were run on typical desktop PCs [one or more processors] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [one or more memories to store parameters associated with the one or more neural networks].”)
to perform a language-guided tree search using one or more neural networks including an action proposal model to generate a set of actions;
a value model to generate a set of values associated with the set of world states;
(Branavan, page: 663, “1.1 Summary of the Approach: We address the above challenges in a unified framework based on Markov Decision Processes (MDP), a formulation commonly used for game playing algorithms. This setup consists of a game in a stochastic environment, where the goal of the player [associated with a goal] is to maximize a given utility function R(s) at state s. The player’s behavior is determined by an action-value function Q(s, a) that assesses the goodness of action a at state s based on the attributes of s and a. To incorporate linguistic information into the MDP formulation, we expand the action value function to include linguistic features [to perform a language-guided]. While state and action features are known at each point of computation, relevant words and their semantic roles are not observed. Therefore, we model text relevance as a hidden variable. Similarly, we use hidden variables to discriminate the words that describe actions [including an action proposal model to generate a set of actions] and those that describe state [a value model to generate a set of values associated with the set of world states] attributes from the rest of the sentence. To incorporate these hidden variables in our action-value function, we model Q(s, a) as a non-linear function approximation using a multi-layer neural network. Despite the added complexity, all the parameters of our non-linear model can be effectively learned in the MonteCarlo Search [tree search] framework. In Monte-Carlo Search, the action value function is estimated by playing multiple simulated games starting at the current game state. We use the observed reward from these simulations to update the parameters of our neural network [using one or more neural networks] via backpropagation. This focuses learning on the current game state, allowing our method to learn language analysis [performing a language-guided] and game-play appropriate to the observed game context.”)
the one or more autonomous agents to perform a selected action of the set of actions
(Branavan, page: 670, “The game playing agent [the one or more autonomous agents] selects actions [to perform a selected action of the set of actions] according to a stochastic policy π(s, a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a).”)
based, at least in part, on a performance of the language-guided tree search
(Branavan, Fig. 3)
PNG
media_image2.png
321
203
media_image2.png
Greyscale
PNG
media_image20.png
138
575
media_image20.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
wherein the performance of the language-guided tree search causes the selected action to be selected from the set of actions based, at least in part, on the set of values and the set of reward distributions.
(Branavan, page: 670, “All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a [wherein the performance of the language-guided tree search causes the selected action to be selected from the set of actions based, at least in part, on the set of values]. When the agent executes action a, the game state changes according to the state transition distribution [and the set of reward distributions]. While T(s 0 | s, a) is not known a priori, state transitions can be sampled from this distribution by invoking the game code as a black-box simulator – i.e., by playing the game. After each action, the agent receives a reward according to the reward function R(s).”)
Branavan does not teach:
a dynamics model to generate a set of world states based at least in part on performance of actions of the set of actions;
wherein the set of world states is represented by one or more voxels that indicate a position
of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to the performance of the action
a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the set of actions;
the selected action when performed causing the one or more autonomous agents to move.
Venkatraman teaches:
a dynamics model to generate a set of world states based at least in part on performance of actions of the set of actions;
(Venkatraman, page: 3, “We specifically refer to the left loop in Fig. 1 as the DAgger (Data-set Aggregation) system identification learning framework [18]. A key difference lies in the aggregation step of the procedure in order to provide model agnostic guarantees. At the beginning of the algorithm, DAgger initializes an empty training data-set and an exploration policy πexplore(xt) that generates an action (control) ut given a state xt. [obtaining a state representation] This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state [to generate a set of world states] - action pairs; [based at least in part on performance of actions of the set of actions] (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [a dynamics model] ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated. During the execution of the first DAgger loop, the state distribution induced by π can greatly differ from the initial π explore; the first generated policies may perform poorly due to inaccuracies in ˆf. The iterative procedure refines the dynamics model by aggregating data from states induced by running the system with π1, . . . , πN .”)
Venkatraman and Branavan are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Venkatraman with teachings of Branavan to enhance a reinforcement learning agent by allowing it to simulate and adapt to complex dynamics to enable the agent experience diverse scenarios, improving its ability to understand and respond to real-world changes effectively (Venkatraman, page: 1).
Branavan in view of Venkatraman do not teach:
wherein the set of world states is represented by one or more voxels that indicate a position
of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to the performance of the action
a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the set of actions;
the selected action when performed causing the one or more autonomous agents to move.
AI teaches:
wherein the set of world states is represented by one or more voxels
(AI, page: 1247, “A. State: We represent the state [a particular state] as a function of the agentposition. For any position q in the discrete voxel-space, [represented by one or more voxels] the corresponding state s = S(q) refers to a stack of the axial, coronal, and sagittal sub-images observed through a squared window centered at the corresponding position. Thus, we allow the agent at any position to observe an m × m × 3 block of surrounding voxels. Here, m is the window size. In the previous RL-based localization approaches [4]–[6], the complete 3D ROI is used as the state, which can provide more useful context for the agent.”)
that indicate a position
(AI, page: 1247, “B. Action Similar to Ghesu et al.’s approach [4], we use a discrete action space where the agent can take a unit step along either of the axes to update its position to a neighboring voxel. Therefore, the agent holds three degrees of freedom to move, enabling six actions i.e., right, le f t, up, down, slice_ forward, slice_backward. The first four moves are along the axial slice (X and Y axes), whereas, the last two actions allow the agent to jump across the slices moving along Z-axis. [that indicate a position] ”).
AI, Branavan and Venkatraman are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of AI with teaching of Branavan and Venkatraman to improve learning speed and robustness, while the reconstructed global policy ensures coordinated behavior across the entire task space (AI, Abstract).
Branavan in view of Venkatraman and Al do not teach:
of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to the performance of the action;
a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the set of actions;
the selected action when performed causing the one or more autonomous agents to move
Bennice teaches:
of one or more objects to interact with one or more autonomous agents in three dimensions (3D) in response to the performance of the action;
(Bennice, “[0010] In some implementations, a computer implemented method may be provided that includes: simulating a three-dimensional (3D) environment, wherein the simulated 3D environment includes a plurality of simulated robots controlled by a single robot controller; rendering multiple instances of an interactive object in the simulated 3D environment, wherein each instance of the interactive object has a simulated physical characteristic that is unique among the multiple instances of the interactive object [of one or more objects to interact]; and receiving, from the robot controller, a common set of joint commands to be issued to each of the plurality of simulated robots [with one or more autonomous agents], wherein for each simulated robot of the plurality of simulated robots, the common command causes actuation of one or more joints of the simulated robot to interact [in response to the performance of the action] with a respective instance of the interactive object [of a set of objects] in the simulated 3D environment [in three dimensions (3D)].”)
Bennice, Branavan, Venkatraman and Al are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Bennice with teachings of Branavan, Venkatraman and Al to allow an agent to learn from diverse experiences simultaneously, improving its ability to handle varied scenarios and outcomes. (Bennice, Abstract).
Branavan in view of Venkatraman, Al and Bennice do not teach:
a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the set of actions;
the selected action when performed causing the one or more autonomous agents to move.
Karaletsos teaches:
a reward model to determine a set of reward distributions comprising a different reward distribution associated with each action of the set of actions
(Karaletsos, “[0076] In equation 6, the term p(zd) represents the distribution of the latent variable zd, the term p(zr) represents the distribution of the latent variable zr, and the term p(s0) represents the distribution of the state s0, the term [p(rt+1|st, at, st+1, zr) represents the distribution of the reward rt+1 [a reward model to determine a set of reward distributions comprising a different reward distribution], the term p(st+1|st, at, zd) represents the distribution of the state st+1, and the term p(at|st, zr, zd) represents the distribution of actions at [associated with each action of the set of actions]. This structure of the model facilitates solving tasks where both of these aspects (dynamics and reward) can vary independently.”)
the selected action when performed causing the one or more autonomous agents to move (Karaletsos, “[0045] An action represents a move that the agent can make [the selected action when performed causing the one or more autonomous agents to move]. An agent selects from a set of possible actions. For example, if the system is configured to play video games, the set of actions includes running right or left, jumping high or low, and so on. If the system is configured to trade stocks, the set of actions includes buying, selling or holding any one of an array of securities and their derivatives…”)
Karaletsos, Branavan, Venkatraman, Al and Bennice are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Karaletsos with teachings of Branavan, Venkatraman, Al and Bennice to add latent variable based model to improve transferability and adaptability to new and unseen conditions. (Karaletsos, Abstract).
Regarding claim 10, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 9.
Branavan further teaches: wherein a reward distribution of the set of reward distributions represents and at least a relative distance to a goal as a result of being in a particular state based at least in part on performing an action of the set of actions.
(Branavan, page: 670, “Reward function [wherein a reward distribution of the set of reward distributions], R(s)
PNG
media_image22.png
28
24
media_image22.png
Greyscale
R, is the immediate reward received when transitioning to state s [as a result of being in a particular state based]. The value of the reward correlates with the goodness of actions [at least in part on performing an action of the set of actions] executed up to now, with higher reward indicating better actions [represents at least a relative distance to a goal] (i.e.: the reward indicates how close or far the agent is from achieving its goal. Higher rewards might suggest the agent is closer to the goal, while lower rewards might suggest it is further away). All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a. When the agent executes action a, the game state changes according to the state transition distribution. While T(s 0 | s, a) is not known a priori, state transitions can be sampled from this distribution by invoking the game code as a black-box simulator – i.e., by playing the game. After each action, the agent receives a reward according to the reward function R(s). In a game playing setup, the value of this reward is an indication of the chances of winning the game from state s. Crucially, the reward signal may be delayed – i.e., R(s) may have a non-zero value only for game ending states such as a win, a loss, or a tie.”)
Karaletsos further teaches: a predicted reward distribution
(Karaletsos, “[0086] In some embodiments, the system uses learned dynamics model to allow agents to plan into the future by recursively predicting future states st+1, . . . , st+h induced by proposed action sequences at, at+1, . . . , at+h. If actions are conditioned on the previous state to describe a policy, then planning becomes learning a policy π* to maximize expected reward over the predicted state-action sequence [a predicted reward distribution]. In this approach, modeling errors are compounded at each time step, resulting in sub-optimal policies when the learning procedure overfits to the imperfect dynamics model.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Karaletsos with teaching of Branavan, Venkatraman, Al, Bennice for the same reasons disclosed for claim 9.
Regarding claim 11, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 10.
Branavan further teaches: wherein the goal is determined based at least in part on a natural language task.
(Branavan, page: 667, “While our method is also driven by control feedback, our language interpretation task itself is fundamentally different. We assume that the given text document provides high level advice without directly describing the correct actions for every potential game state. Furthermore, the textual advice does not necessarily translate to a single strategy – in fact, the text may describe several strategies, each contingent on specific game states. For this reason, the strategy text cannot simply be interpreted directly into a policy. Therefore, our goal [wherein the goal] is to bias a learned policy using information extracted from text [is determined based at least in part on a natural language task] (i.e.: using natural language (textual information) to bias or influence a learned policy, that the goal or objective in decision-making is influenced by insights derived from the textual advice). To this end, we do not aim to achieve a complete semantic interpretation, but rather use a partial text analysis to compute features relevant for the control application.”)
Regarding claim 13, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 9.
Branavan further teaches: wherein the action proposal model further comprises a policy model.
(Branavan, page: 670, “The game playing agent selects actions [wherein the action proposal model]according to a stochastic policy π(s, a), [further comprises a policy model] which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, and then following policy π is termed the action-value function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image23.png
18
18
media_image23.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game. If the optimal action-value function Qπ (s, a) is known, the optimal gameplaying behavior would be to select the action a with the highest Qπ
PNG
media_image24.png
18
18
media_image24.png
Greyscale
(s, a). While it may be computationally hard to find an optimal policy π
PNG
media_image8.png
19
18
media_image8.png
Greyscale
(s, a) or Qπ
PNG
media_image12.png
19
18
media_image12.png
Greyscale
(s, a), many well studied algorithms are available for estimating an effective approximation (Sutton & Barto, 1998)”).
Regarding claim 14, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 9.
Branavan further teaches: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to:
(Branavan, page: 683, “The experiments were run on typical desktop PCs [as a result of being executed by the one or more processors, cause the one or more processors to] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [wherein one or more memories further store instructions that].”)
create a set of edges from a root node to a set of leaf nodes representing a plurality of actions of the set of actions;
[AltContent: rect][AltContent: rect][AltContent: rect]
PNG
media_image25.png
234
746
media_image25.png
Greyscale
[AltContent: textbox ([to a set of leaf nodes representing a plurality of actions of the set of actions])][AltContent: textbox ([create a set of edges])][AltContent: textbox ([from a root node])](Branavan, Fig. 2)
obtain the set of actions from the action proposal model;
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
PNG
media_image27.png
87
1000
media_image27.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]
PNG
media_image28.png
387
381
media_image28.png
Greyscale
PNG
media_image29.png
188
458
media_image29.png
Greyscale
[AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect][AltContent: rect]annotate the set of leaf nodes to indicate values of the set of values and edges of the set of edges with reward distributions of the set of reward distributions.
(Branavan, page: 671, “Figure 3: Overview of Monte-Carlo Search algorithm. For each game state st , an independent set of simulated games or roll-outs are done to find the best possible game action at . Each roll-out starts at state st , with actions [obtain the set of actions from the action proposal model] selected according to a simulation policy π(s, a). This policy is learned from the roll-outs themselves – with the roll-outs improving the policy, which in turn improves roll-out action selection. The process is repeated for every actual game state, with the simulation policy being relearned from scratch each time.”)
Venkatraman further teaches: obtain a set of state representations from the dynamics model representing a world state upon completion of the plurality of actions of the set of actions; and
(Venkatraman, page: 3, “We specifically refer to the left loop in Fig. 1 as the DAgger (Data-set Aggregation) system identification learning framework [18]. A key difference lies in the aggregation step of the procedure in order to provide model agnostic guarantees. At the beginning of the algorithm, DAgger initializes an empty training data-set and an exploration policy πexplore(xt) that generates an action (control) ut given a state xt. [obtain a set of state representations] This initial policy can either consist of random controls (referred to as a random policy) or be an expert demonstration. Then, DAgger iteratively proceeds by: (1) executing the latest policy to collect a set of new trajectories {ξi} k−1 i=0 where ξi = {(xt, ut)...}i is a time series of state [representing a world state] - action pairs; [upon completion of the plurality of actions of the set of actions] (2) aggregating the trajectories {ξi} k−1 i=0 into the training data-set; (3) learning from the data-set a forward dynamics model [from the dynamics model] ˆf(xt, ut) → xt+1; (4) optimizing a new control policy π that minimizes a given cost function c(xt, ut) over the time horizon T of the control problem; (5) tracking the best policy from all those generated. During the execution of the first DAgger loop, the state distribution induced by π can greatly differ from the initial π explore; the first generated policies may perform poorly due to inaccuracies in ˆf. The iterative procedure refines the dynamics model by aggregating data from states induced by running the system with π1, . . . , πN .”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Venkatraman with teaching of Branavan, Al, Bennice and Karaletsos for the same reasons disclosed for claim 9.
Regarding claim 16, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 14.
Branavan further teaches: wherein the values of the set of values further comprise a first distribution of the set of values.
(Branavan, page: 670, “Reward function [set of rewards], R(s)
PNG
media_image33.png
28
24
media_image33.png
Greyscale
R, is the immediate reward received when transitioning to state s [a first distribution of the set of values] (i.e.: refers to the action-value function, which represents the expected reward and can be considered as having a distribution because it is based on expected outcomes over potentially many trials and different state-action pairs). The value of the reward correlates with the goodness of actions [wherein the values of the set of values further comprise] executed up to now, with higher reward indicating better actions. All the above aspects of the MDP representation of the game – i.e., S, A, T() and R() – are defined implicitly by the game rules. At each step of the game, the game-playing agent can observe the current game state s, and has to select the best possible action a.”)
Claim(s) 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of Karaletsos, AI, Venkatraman, Bennice and in further view of Remez et al., "Improved learning of dynamics models for control."
Regarding claim 5, Branavan in view of Karaletsos, AI, Venkatraman and Bennice teach the method of claim 4.
Branavan in view of Karaletsos, AI, Venkatraman and Bennice do not teach:
wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics.
Remez teaches:
wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics.
(Remez, page: 6, “3 Architecture There are three modules in our model [wherein the dynamics model]: (1) the generator, which predicts a mask, (2) the cut-and-paste module [further comprises a cut and paste model that implements observed space dynamics] (i.e.: the model predicts certain features (masks or patches), manipulates them (cut-and-paste), and possibly uses them in subsequent steps (like discrimination between real and fake patches). This reflects a system where dynamics are modeled not just through prediction but also through manipulation and simulation of observed data dynamics), which produces a “fake patch” given the predicted mask, and (3), the discriminator, which distinguishes between real and fake patches, see Figure 3. In the following, we describe the architecture for each of these modules that we have used in our experiments. Generator: Our generator is similar to that of Mask R-CNN [1]. A ResNet-50 backbone is used to extract ROI-aligned features and a mask prediction head is applied to these features. Our mask prediction head is described in Table 1, and is comprised of a series of convolutions, bilinear upsampling operations, and a Sigmoid nonlinearity resulting in a 28 × 28 mask output. We find that using corner-aligned bilinear upsampling generally provides better results than transposed convolutions and nearest neighbor upsampling layers.”)
Remez, Branavan, Karaletsos, AI, Venkatraman and Bennice are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Remez with teachings of Branavan, Karaletsos, AI, Venkatraman and Bennice to enhance the agent’s ability to accurately segment objects in new environments, by leveraging adversarial learning to refine its predictions, ultimately achieving performance close to fully supervised methods (Remez, Abstract).
Regarding Claim 12, Branavan in view of Venkatraman, Al, Bennice and Karaletsos teach the method of claim 9.
Branavan in view of Venkatraman, AI, Bennice and Karaletsos do not teach:
wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics.
Remez teaches:
wherein the dynamics model further comprises a cut and paste model that implements observed space dynamics.
(Remez, page: 6, “3 Architecture There are three modules in our model [wherein the dynamics model]: (1) the generator, which predicts a mask, (2) the cut-and-paste module [further comprises a cut and paste model that implements observed space dynamics] (i.e.: the model predicts certain features (masks or patches), manipulates them (cut-and-paste), and possibly uses them in subsequent steps (like discrimination between real and fake patches). This reflects a system where dynamics are modeled not just through prediction but also through manipulation and simulation of observed data dynamics), which produces a “fake patch” given the predicted mask, and (3), the discriminator, which distinguishes between real and fake patches, see Figure 3. In the following, we describe the architecture for each of these modules that we have used in our experiments. Generator: Our generator is similar to that of Mask R-CNN [1]. A ResNet-50 backbone is used to extract ROI-aligned features and a mask prediction head is applied to these features. Our mask prediction head is described in Table 1, and is comprised of a series of convolutions, bilinear upsampling operations, and a Sigmoid nonlinearity resulting in a 28 × 28 mask output. We find that using corner-aligned bilinear upsampling generally provides better results than transposed convolutions and nearest neighbor upsampling layers.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Remez with teaching of Branavan, Venkatraman, Al, Bennice and Karaletsos for the same reasons disclosed for claim 5.
Claim(s) 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of AI, Venkatraman, Bennice, Karaletsos and in further view of XIAO et al., Pub. No.: US20200234167.
Regarding claim 15, Branavan in view of AI, Venkatraman, Bennice and Karaletsos teach the method of claim 14.
Branavan further teaches: wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors (Branavan, page: 683, “The experiments were run on typical desktop PCs [as a result of being executed by the one or more processors, cause the one or more processors] with single Intel Core i7 CPUs (4 hyper-threaded cores per CPU) [wherein one or more memories further store instructions that].”)
Branavan in view of AI, Venkatraman, Bennice and Karaletsos do not teach:
to perform a backup operation to update the set of leaf nodes based at least in part on the set of values and the set of rewards.
XIAO teaches:
to perform a backup operation to update the set of leaf nodes based at least in part on the set of values and the set of rewards.
(XIAO, “[0078] Further, in the present embodiment, the method may include selecting actions to take until a node s is identified, where ξ(s)
PNG
media_image34.png
28
24
media_image34.png
Greyscale
L. The method may include utilizing the stochastic value function at a node s and a result of the stochastic value function may be used to update the search tree [update the set of leaf nodes based at least in part on the set of values and the set of rewards] as in the above-described in the backup operation [to perform a backup operation]. In the present example, because each iteration may add a separate path of nodes from the root to the leaf [leaf nodes] of T to the current search tree, the present example may be called a path-REWT.”)
XIAO, Branavan, AI, Venkatraman, Bennice and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of XIAO with teachings of Branavan, AI, Venkatraman, Bennice and Karaletsos to enhance the reinforcement learning system by updating a multi-level data structure to improve agent control (XIAO, ¶[0004]).
Regarding claim 17 Branavan in view of AI, Venkatraman, Bennice and Karaletsos teach the method of claim 16.
Branavan further teaches: based at least in part on the first distribution and reward distributions of the set of reward distributions.
(Branavan, page: 670, “The game playing agent selects actions according to a stochastic policy π(s,
a), which specifies the probability of selecting action a in state s. The expected total reward after executing action a in state s, [based at least in part on the first distribution and reward distributions of the set of reward distributions] and then following policy π is termed the actionvalue function Qπ (s, a). Our goal is to find the optimal policy π
PNG
media_image35.png
19
18
media_image35.png
Greyscale
(s, a) which maximizes the expected total reward – i.e., maximizes the chances of winning the game.”)
Branavan in view of AI, Venkatraman, Bennice and Karaletsos do not teach:
wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to perform a backup operation to update the set of leaf nodes
XIAO teaches:
wherein one or more memories further store instructions that, as a result of being executed by the one or more processors, cause the one or more processors to perform a backup operation to update the set of leaf nodes
(XIAO, “[0078] Further, in the present embodiment, the method may include selecting actions to take until a node s is identified, where ξ(s)
PNG
media_image34.png
28
24
media_image34.png
Greyscale
L. The method may include utilizing the stochastic value function at a node s and a result of the stochastic value function may be used to update the search tree [to update the set of leaf nodes] as in the above-described in the backup operation [to perform a backup operation]. In the present example, because each iteration may add a separate path of nodes from the root to the leaf of T to the current search tree, the present example may be called a path-REWT.
[0079] To illustrate a theoretical analysis of convergence property for operations associated with MENTS, for any node in a search tree, after its subtree has been explored, the estimated softmax value may converge to an optimal value at an exponential rate. Recall that in Theorem 1, an optimal sampling algorithm for the softmax stochastic bandit problem may guarantee limt→∞Nt(a)/t=πsft*(a) for any action a. This may be shown in E2 W with high probability and may be based on the proof of Theorem 2.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of XIAO with teaching of Branavan, Al, Venkatraman, Bennice and Karaletsos for the same reasons disclosed for claim 15.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of AI, Bennice, Karaletsos and in further view of Vernade et al., Pub. No.: US20210158196A1.
Regarding claim 26, Branavan in view of AI, Bennice and Karaletsos teach the method of claim 25.
Branavan in view of AI, Bennice and Karaletsos do not teach:
selecting the action comprises computing a Q-distribution based, at least in part, on the value and the distribution of rewards
Vernade teaches:
selecting the action comprises computing a Q-distribution based, at least in part, on the value and the distribution of rewards
(Vernade, “[0044] In particular, an action selection engine 110 [selecting the action] maintains count data 150 and uses the maintained count data 150 to select actions 122 that optimize expected rewards, i.e., that optimize the expected delayed reward 124 to be received in response to performing an action given the current transition probability distribution [comprises computing a Q distribution] and the stationary reward distribution [based, at least in part, on the value and the distribution of rewards].”)
Vernade, Branavan, AI, Bennice and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Vernade with teachings of Branavan, AI, Bennice and Karaletsos to improve the ability to choose actions that align with high-level goals or situational understanding. (Vernade, Abstract).
Claim 28 is rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of AI, Bennice, Karaletsos and in further view of XIAO et al., Pub. No.: US20200234167.
Regarding claim 28, Branavan in view of AI, Bennice and Karaletsos teach the method of claim 23.
Branavan in view of AI, Bennice and Karaletsos do not teach:
wherein generating the tree further comprises performing a backup operation to surface a highest predicted value associated with the set of proposed actions.
XIAO teaches:
wherein generating the tree further comprises performing a backup operation to surface a highest predicted value associated with the set of proposed actions.
(XIAO, “[0068] A node s that ξ(s)∈ T/L may contain edges (s, a) for all actions a ∈ A(s) and may store a set of statistics, including a softmax state value estimation Vsft(s). In some embodiments, for each edge {N(s, a), {circumflex over (Q)}sft(s, a)}, N(s,a) may be the visit count and Qsft*(s, a) may be the softmax state-action value estimation [to surface a highest predicted value associated with the set of proposed actions] (i.e.: softmax function is primarily used for probabilistic action selection, the underlying Q(s, a) values can be directly used to identify the highest predicted value associated with a set of proposed actions. Thus, the action with the highest Q -value can be surfaced as the best predicted action).
[0069]In some embodiments, operations of REWT may include iterations of 4 operations, including a select operation, an evaluate operation, an expand operation, and a backup operation [wherein generating the tree further comprises performing a backup operation].
[0070] At the select operation, the method may include generating a trajectory of nodes based on the REW method. Generating the trajectory of nodes may include a series of nodes including a root node of the search tree, so, and ending with a leaf node, sL, of the search tree.”)
XIAO, Branavan, AI, Bennice and Karaletsos are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of XIAO with teachings of Branavan, AI, Bennice and Karaletsos to enhance the reinforcement learning system by updating a multi-level data structure to improve agent control (XIAO, ¶[0004]).
Claim(s) 29 – 30 are rejected under 35 U.S.C. 103 as being unpatentable over Branavan in view of AI, Bennice, Karaletsos, XIAO and in further view of Ren et al., “Policy Optimization for Spoken Dialog Management Using Genetic Algorithm.”
Regarding claim 29, Branavan in view of AI, Bennice, Karaletsos and XIAO teach the method of claim 28.
Branavan in view of AI, Bennice, Karaletsos and XIAO do not teach:
wherein the backup operation further comprises a Bellman backup operation.
Ren teaches:
wherein the backup operation further comprises a Bellman backup operation.
(Ren, page: 5, “3.4 DM Policy Evaluation Using Dialog Corpora We describe a DM policy performance evaluation method on dialog corpus to avoid the need for deploying the DM on-line. We estimate a policy by learning its value function using a similar iterative routine to FQI on a held-out testing dialog corpus. The estimated cumulative reward when following the policy is used as a metric for performance. A similar approach has been taken in evaluating the effect of dialog state tracker on end-to-end performance of SDS [23]. To estimate the Q-function of a DM policy, the Bellman backup operation [wherein the backup operation further comprises a Bellman backup operation] (Line 10) in Algorithm 2 is changed to: Qi,t ← ri,t + γ ˆQ(si,t+1, π(si,t+1)), (4) where π is the DM policy to evaluate. After the value function converges, the average reward for all the training samples of initial dialog turn 1 N i Qi,0 is used as a metric for performance.”)
Ren, Branavan, AI, Bennice, Karaletsos and XIAO are related to the same field of endeavor (i.e.: training an agent through natural language). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teachings of Ren with teachings of Branavan, AI, Bennice, Karaletsos and XIAO to ensure effective dialog management in noisy environments (Ren, Abstract).
Regarding claim 30, Branavan in view of AI, Bennice, Karaletsos and XIAO teach the method of claim 28.
Branavan in view of AI, Bennice, Karaletsos and XIAO do not teach:
wherein the backup operation further comprises backing up the tree the distribution of rewards
Ren teaches:
wherein the backup operation further comprises backing up the tree the distribution of rewards
(Ren, page: 5, “3.4 DM Policy Evaluation Using Dialog Corpora We describe a DM policy performance evaluation method on dialog corpus to avoid the need for deploying the DM on-line. We estimate a policy by learning its value function using a similar iterative routine to FQI on a held-out testing dialog corpus. The estimated cumulative reward when following the policy is used as a metric for performance. A similar approach has been taken in evaluating the effect of dialog state tracker on end-to-end performance of SDS [23]. To estimate the Q-function of a DM policy, the Bellman backup operation [wherein the backup operation further comprises backing up the tree] (Line 10) in Algorithm 2 is changed to:
PNG
media_image36.png
28
426
media_image36.png
Greyscale
[the distribution of rewards] (i.e.: r ti is the immediate reward received after taking
action).
(i.e.: represents the discounted future value estimate. This is the second
distribution of values because it considers the expected future rewards (values) from state \( s_{i,t+1} \) onward, influenced by the policy π). where π is the DM policy to evaluate. After the value function converges, the average reward for all the training samples of initial dialog turn 1 N i Qi,0 is used as a metric for performance.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Ren with teachings of Branavan, AI, Bennice, Karaletsos and XIAO for the same reasons disclosed for claim 29.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zhu, et al. "Target-driven visual navigation in indoor scenes using deep reinforcement learning" , 2017.
An actor-critic model that uses both the goal and the current state to improve generalization. Introduce the AI2-THOR framework, which offers a rich environment with high-quality 3D scenes and a physics engine. This framework lets agents perform actions and interact with objects, allowing us to gather a large number of training samples effectively..
Kempka, et al., "Vizdoom: A doom-based ai research platform for visual reinforcement learning.", (2016).
A platform for reinforcement learning research that uses raw visual information in a semirealistic 3D world. This software, called ViZDoom, is based on the classic video game Doom and enables the development of bots that play the game using the screen visuals. ViZDoom is lightweight, fast, and easily customizable through user-defined scenarios.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to MATIYAS T MARU whose telephone number is (571)270-0902 or via email: matiyas.maru@uspto.gov. The examiner can normally be reached Monday 8:00am - Friday 4:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to
use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Michelle Bechtold can be reached on (571)431-0762. The fax phone number for the organization were
this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA)
or 571-272-1000.
/M.T.M./Examiner, Art Unit 2148
/Ryan Barrett/Primary Examiner, Art Unit 2148
Prosecution Insights