DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 3/3/2026 has been entered. Claims 1, 3, 5, 8-9, 11, 14-15, 17, and 21 have been amended. Claims 12 and 22 have been canceled. Claims 1-11 and 13-21 are pending in the application.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 17-21 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 17 recites that the action is configured to increase a likelihood of achieving the goal and that the second state is a hierarchical state configured to increase the likelihood of achieving the goal and increasing an efficiency in computation associated with achieving the goal. The terms increase a likelihood and increasing an efficiency in computation are relative terms which render the claim indefinite. The term increase, as applied to both the likelihood of achieving the goal and the efficiency in computation, is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. An increase is a comparison that requires a baseline against which the likelihood or the efficiency is measured, yet neither the claim nor the specification identifies the reference condition relative to which the likelihood of achieving the goal is increased or the efficiency in computation is increased. The specification merely repeats the claim language without supplying an objective standard; see specification paragraph [000124], which states that the first state and/or the second state is configured to increase the likelihood of achieving a goal and/or increasing an efficiency in computation associated with achieving the goal, and paragraphs [000150]-[000151], which likewise restate the functional result without identifying any baseline, metric, or measurement methodology for the asserted increase. Because the specification does not provide a standard for measuring when a likelihood or an efficiency in computation has been increased, a person of ordinary skill in the art cannot ascertain the metes and bounds of the claim. See MPEP 2173.05(b); Datamize, LLC v. Plumtree Software, Inc., 417 F.3d 1342, 1350 (Fed. Cir. 2005); Interval Licensing LLC v. AOL, Inc., 766 F.3d 1364, 1371 (Fed. Cir. 2014); cf. Enzo Biochem, Inc. v. Applera Corp., 599 F.3d 1325, 1332 (Fed. Cir. 2010) (relative limitation definite only where the specification supplied a standard). For purposes of examination, under the broadest reasonable interpretation consistent with specification paragraphs [000124], [000150], and [000151], the limitation is interpreted to require only that the recited action and the hierarchical second state be capable of producing any non-zero improvement, however small, in the probability of reaching the recited goal and in the computational cost of reaching that goal as compared to performing a non-hierarchical action or occupying a non-hierarchical state.
Claims 18-21 depend, directly or indirectly, from claim 17. Claims 18, 19, and 21 depend directly from claim 17 and claim 20 depends from claim 19. They incorporate and do not cure the indefiniteness of the increase a likelihood and increasing an efficiency in computation limitations of claim 17, and are rejected for the same reason. Claim 21 further recites that performing the option increases the likelihood of achieving the goal and increases the efficiency in computation associated with achieving the goal, and is indefinite for the same lack of an ascertainable comparative standard as set forth above with respect to claim 17.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 and 8-10 are rejected under 35 U.S.C. 103 over Sutton et al. (hereinafter Sutton) “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning” (1999), in view of Konidaris et al. (hereinafter Konidaris) “Skill Discovery in Continuous Reinforcement Learning Domains using Skill Chaining” (2009).
Regarding independent claim 1, Sutton teaches a method (Sutton: page 184, Sections 1-2, "a learning agent interacts with an environment"; develop a reinforcement-learning method over a Markov decision process):
receiving inputs associated with interactions of an agent with an environment, the interactions including a plurality of states associated with the environment and a plurality of actions associated with each state from the plurality of states (Sutton: page 184, Section 1, "the agent perceives the state of the environment, st ∈S, and on that basis chooses a primitive action, at ∈Ast"; the learning agent (an agent) perceives the state of the environment (associated with interactions of with an environment) drawn from (receiving inputs) the set of states S (the interactions including a plurality of states associated with the environment) and on each step chooses a primitive action from the action set available at that state (and a plurality of actions associated with each state from the plurality of states));
receiving an indication of a target state to be achieved by the agent in the environment (Sutton: page 206, Section 7, "assign a terminal subgoal value, g(s), to each state s in a subset of states G ⊆S. These values indicate how desirable it is for the option to terminate in each state in G"; a subset of states is designated as a subgoal (receiving an indication of) and assigned a terminal subgoal value indicating how desirable it is to terminate there (a target state to be achieved by the agent in the environment));
identifying a state sequence including a first state, a second state, and a third state from the plurality of states such that the agent can perform a first action to implement a transition from the first state to the second state and a second action to implement a transition from the second state to the third state in a consecutive manner (Sutton: page 182, Sections 0-1, "the unitary action taken at time t affects the state and reward at time t + 1"; the agent's state trajectory is a sequence of states (identifying a state sequence including a first state, a second state, and a third state from the plurality of states) connected by consecutive one-step action transitions (a first action to implement a transition from the first state to the second state and a second action to implement a transition from the second state), and the agent thereby traverses st, st+1, st+2 (to the third state in a consecutive manner) over consecutive time steps (such that the agent can perform));
setting a value associated with the transition from the first state to the hierarchical state to be equal to a value combination that is a function of a value associated with the first action and a value associated with the second action (Sutton: pages 184-185, Section 1; the value of a state is computed as a combination, e.g. a sum, that is a function of the values of the constituent actions taken from that state (setting a value associated with the transition from the first state to the hierarchical state), the Bellman equation expressing the state value as the action-probability-weighted sum of the expected reward plus the discounted value of the successor state (to be equal to a value combination that is a function of a value associated with the first action and a value associate with the second action)), setting a value associated with the transition from the hierarchical state to the third state (Sutton: page 185, Equation (3), "V ∗(s) = max π V π(s)"; the optimal value of a state is the maximum over the available actions of that action-value function (setting a value associated with the transition from the hierarchical state to the third state)).
Sutton does not expressly teach generating a hierarchical state configured to be associated with (i) a third action implementing a transition from the first state to the hierarchical state, and (ii) a fourth action implementing a transition from the hierarchical state to the third state, the first action and the second action forming an option, the hierarchical state associated with fewer actions than a non-hierarchical state; transition from the hierarchical state to the third state to be equal to a maximum value associated with the third state; determining an identifier associated with the hierarchical state; searching a dictionary to determine whether the identifier is included in the dictionary; in response to determining the identifier is not included in the dictionary, adding the identifier to the dictionary to generate an updated dictionary; and storing the updated dictionary.
However, Konidaris teaches generating a hierarchical state configured to be associated with (i) a third action implementing a transition from the first state to the hierarchical state, and (ii) a fourth action implementing a transition from the hierarchical state to the third state, the first action and the second action forming an option, the hierarchical state associated with fewer actions than a non-hierarchical state (Konidaris: page 1, Section 2, "an option created to reach one of its goal states and terminate when it does so", page 3, Section 4.1, "To create an option oT to trigger T... we must define oT's termination condition, reward function, and initiation set"; the agent creates a new option to reach a target event, the created option being a single temporally-extended decision that abstracts a span of the trajectory bounded by an entry transition and an exit transition (a hierarchical state; a third action; a fourth action; the first action and the second action forming an option), and Konidaris: page 3, Section 3, "lightweight options that use fewer features than needed to solve the overall problem are desirable"; the created option is deliberately lightweight, using fewer features than the overall task and thereby fewer constituent actions (associated with fewer actions than a non-hierarchical state)); transition from the hierarchical state to the third state to be equal to a maximum value associated with the third state (Konidaris: pages 5-6, Section 5.1, "the maximum of these values"; a newly created option's value is initialized to the maximum of the sampled transition values for that target); determining an identifier associated with the hierarchical state (Konidaris: page 3, Section 4.1, "a goal trigger function T defined over S that evaluates to 1 on states in the goal event"; each option is created for and indexed by its target-event trigger function, which identifies the option within the agent's collection (an identifier associated with the hierarchical state)), searching a dictionary to determine whether the identifier is included in the dictionary (Konidaris: page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event"; before creating an option the agent tests whether an option targeting that event already exists in its stored collection of options (searching a dictionary to determine whether the identifier is included in the dictionary)), in response to determining the identifier is not included in the dictionary, adding the identifier to the dictionary to generate an updated dictionary (Konidaris: page 5, Section 5.1, "the option was added to the agent's action repertoire. For new option o, this requires expanding the overall action-value function Q to include o"; when no such option exists the newly created option is added to the agent's repertoire by expanding the action-value structure to include it (adding the identifier to the dictionary to generate an updated dictionary)), and storing the updated dictionary (Konidaris: page 8, Section 7, "select an appropriate abstraction for a new option from a library of candidate abstractions"; the agent maintains its set of discovered options as a stored library of abstractions for reuse (storing the updated dictionary)).
Because Sutton and Konidaris are analogous art and within the same field of endeavor, specifically hierarchical reinforcement learning using the options framework, they address the same problem-solving area of discovering and efficiently reusing temporally extended abstractions in an agent-environment Markov decision process. Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the option-creation, duplicate-avoidance, repertoire-augmentation, and maximum-value-initialization bookkeeping of Konidaris with the options-and-value framework of Sutton, with a reasonable expectation of success, such that newly generated abstractions are identified, checked against the agent's stored collection, and added when absent, to teach generating a hierarchical state configured to be associated with (i) a third action implementing a transition from the first state to the hierarchical state, and (ii) a fourth action implementing a transition from the hierarchical state to the third state, the first action and the second action forming an option, the hierarchical state associated with fewer actions than a non-hierarchical state; transition from the hierarchical state to the third state to be equal to a maximum value associated with the third state; determining an identifier associated with the hierarchical state; searching a dictionary to determine whether the identifier is included in the dictionary; in response to determining the identifier is not included in the dictionary, adding the identifier to the dictionary to generate an updated dictionary; and storing the updated dictionary. This modification would have been motivated by the desire to avoid creating duplicate options and to accumulate a reusable, stored set of skills as the agent explores (Konidaris: page 3, Section 3, “options that use fewer features than needed to solve the overall problem are desirable”, page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event", page 8, Section 7, "a library of candidate abstractions").
Regarding dependent claim 2, Sutton, in view of Konidaris, teach the method of claim 1, wherein the third action and the fourth action are hierarchical actions (Konidaris: page 1, Section 2, "chooses when to execute them in the same way it chooses when to execute primitive actions"; the entry and exit transitions that bound a created option are themselves the option's own temporally-extended decisions rather than base primitive actions (the third action and the fourth action are hierarchical actions)).
Regarding dependent claim 3, Sutton, in view of Konidaris, teach the method of claim 1, computing the value combination (Sutton: Section 1, the agent computes the state value as the combination of the constituent action values via the Bellman equation), verifying that the value combination has a non-zero value (Sutton: page 206, Section 7, "the target hallway might be assigned a subgoal value of +1"; an option is created toward a subgoal carrying a positive (non-zero) terminal value), and verifying that the state sequence is non-cyclical, the generating the hierarchical state being based on the value combination having the non-zero value and the state sequence being non-cyclical (Konidaris: page 4, Section 4.3, "we require that the initiation set of an option does not overlap that of its siblings or parents"; option creation is gated by conditions on the trajectory, including that the initiation set of a new option does not overlap that of its parents so the chain does not fold back on itself (verifying that the state sequence is non-cyclical)).
Regarding dependent claim 4, Sutton, in view of Konidaris, teach the method of claim 1, wherein the hierarchical state is associated with a value combination of values from non-hierarchical states (Sutton: Section 1, the value associated with the abstraction is the combination of the values of its constituent primitive-action states (a value combination of values from non-hierarchical states), the state value being the Bellman combination of the expected reward and the discounted successor-state value).
Regarding dependent claim 5, Sutton, in view of Konidaris, teach the method of claim 1, combining the first state, the hierarchical state, and the third state to generate a sequence of states associated with the third action and the fourth action, the sequence of states forming an option sequence (Konidaris: page 4, Section 4.2, "a chain of skills leading from any state in which the agent may start to the task's goal region"; the agent strings successive options together so that one option leads to the entry region of the next, forming a sequence of states bound by the options' transitions (a sequence of states forming an option sequence)).
Regarding dependent claim 8, Sutton, in view of Konidaris, teach the method of claim 1, determining the state sequence including the first state, the second state, and the third state to be generalizable (Konidaris: page 3, Section 3, "Options that are useful across a collection of problems should have goals that have high probability of falling on the solution paths of some of those problems"; the agent targets subgoals whose options are useful across a collection of problems, that is, generalizable (determining the state sequence... to be generalizable)) and discovering the hierarchical state configured to form the option, the option being configured to implement the transition from the first state to the second state in a reusable sequence (Konidaris: page 2, Section 3, "if options are learned in an appropriate space they can be used in later tasks to speed up learning"; options learned in a space are portable and reused in later tasks (a reusable sequence)).
Regarding dependent claim 9, Sutton, in view of Konidaris, teach the method of claim 1, wherein the second state is associated with a first number of potential decisions to implement a transition from the first state to the third state, and the hierarchical state is associated with a second number of potential decisions to implement the transition from the first state to the third state (Sutton: page 182, Section 0, "a course of action persisting over a variable period of time"; invoking an option replaces a sequence of primitive decisions with a single decision, so traversing via the abstraction involves fewer potential decisions than via the base states (a first number of potential decisions; a second number of potential decisions), an option lets the agent take rather than a unitary action per step), discovering the hierarchical state and the option (Konidaris: page 1, Section 2, "an option created to reach one of its goal states"); determining that the option does not exist in the dictionary (Konidaris: page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event"); and forming the option in response to the determining that the option does not exist in the dictionary (Konidaris: Section 4.3, a new option is created precisely when no existing option already targets that event (forming the option in response to the determining that the option does not exist), the duplicate-avoidance condition gating creation).
Regarding dependent claim 10, Sutton, in view of Konidaris, teach the method of claim 1, wherein the hierarchical state is a first hierarchical state, generating a second hierarchical state associated with a plurality of actions and at least two of the first state, the second state, the third state, the first hierarchical state, or a fourth state different than the first state, the second state, the third state, and the first hierarchical state (Konidaris: page 4, Section 4.3, "More than one option may be created to reach a target event... resulting in a skill tree"; the agent creates multiple options, building a tree in which more than one option (a second hierarchical state) is created over the states and earlier options (a plurality of actions and at least two of the recited states)).
Claims 11, 13-14, and 16 rejected under 35 U.S.C. 103 over Sutton in view of Konidaris and further in view of Hafner et al. (hereinafter Hafner), US 2021/0158162 A1.
Hafner was disclosed in an IDS dated 4/27/2026.
Regarding independent claim 11, Sutton teaches an apparatus configured to (Sutton: page 184, Sections 1-2, "a learning agent interacts with an environment"; the reinforcement-learning method is executed by a computational learning system):
receive inputs associated with interactions of an agent with an environment, the interactions including a plurality of states associated with the environment and a plurality of actions associated with each state from the plurality of states (Sutton: page 184, Section 1, "the agent perceives the state of the environment, st ∈S, and on that basis chooses a primitive action, at ∈Ast"; the learning agent (an agent) perceives the state of the environment (associated with interactions of with an environment) drawn from (receive inputs) the set of states S (the interactions including a plurality of states associated with the environment) and on each step chooses a primitive action from the action set available at that state (and a plurality of actions associated with each state from the plurality of states));
receive an indication of a target state to be achieved by the agent in the environment (Sutton: page 206, Section 7, "It is possible to have many such subgoals and learn about them each independently using an off-policy learning method such as Q-learning…assign a terminal subgoal value, g(s), to each state s in a subset of states G ⊆S. These values indicate how desirable it is for the option to terminate in each state in G"; a subset of states is designated as a subgoal (receiving an indication of) and assigned a terminal subgoal value indicating how desirable it is to terminate there (a target state to be achieved by the agent in the environment));
identify a state sequence including a first state, a second state, and a third state from the plurality of states such that the agent can perform a first action to implement a transition from the first state to the second state and a second action to implement a transition from the second state to the third state in a consecutive manner (Sutton: page 182, Sections 0-1, "the unitary action taken at time t affects the state and reward at time t + 1"; the agent's state trajectory is a sequence of states (identify a state sequence including a first state, a second state, and a third state from the plurality of states) connected by consecutive one-step action transitions (a first action to implement a transition from the first state to the second state and a second action to implement a transition from the second state), and the agent thereby traverses st, st+1, st+2 (to the third state in a consecutive manner) over consecutive time steps (such that the agent can perform));
at least one of the first state, the second state, or the third state being associated with a primitive action (Sutton: pages 184-185, Section 1, "On each time step, t, the agent perceives the state of the environment st ∈ S, and on that basis chooses a primitive action, at ∈Ast. In response to each action…the environment produces one step later a numerical reward…and a next state"; the base actions selected at each state (at least one of the first state, the second state, or the third state) are the recited primitive actions, with which the hierarchical state is associated (being associated with a primitive action)).
Sutton does not expressly teach at least one of the first state, the second state, or the third state being a hierarchical state; the hierarchical state being associated with fewer actions than a state that is not a hierarchical state; determine an identifier associated with the hierarchical state; search a dictionary to determine whether the identifier associated with the hierarchical state is included in the dictionary; add, based on the determination that the identifier associated with the hierarchical state is not included in the dictionary, the identifier associated with the hierarchical state to the dictionary to generate an updated dictionary; and store the updated dictionary.
However, Konidaris teaches at least one of the first state, the second state, or the third state being a hierarchical state (Konidaris: page 1, Section 2, "an option created to reach one of its goal states and terminate when it does so", page.3, Section 4.1, "To create an option oT to trigger T... we must define oT's termination condition, reward function, and initiation set"; the agent creates a new option to reach a target event, the created option being a single temporally-extended decision that abstracts a span of the trajectory bounded by an entry transition and an exit transition (at least one of the first state, the second state, or the third state being a hierarchical state)); the hierarchical state being associated with fewer actions than a state that is not a hierarchical state (Konidaris: page 3, Section 3, "lightweight options that use fewer features than needed to solve the overall problem are desirable"; the created option is deliberately lightweight, using fewer features than the overall task and thereby fewer constituent actions (the hierarchical state being associated with fewer actions than a state that is not a hierarchical state)); determine an identifier associated with the hierarchical state (Konidaris: page 3, Section 4.1, "a goal trigger function T defined over S that evaluates to 1 on states in the goal event"; each option is created for and indexed by its target-event trigger function, which identifies the option within the agent's collection (determine an identifier associated with the hierarchical state)); search a dictionary to determine whether the identifier associated with the hierarchical state is included in the dictionary (Konidaris: page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event"; before creating an option the agent tests whether an option targeting that event already exists in its stored collection of options (search a dictionary to determine whether the identifier associated with the hierarchical state is included in the dictionary)); add, based on the determination that the identifier associated with the hierarchical state is not included in the dictionary, the identifier associated with the hierarchical state to the dictionary to generate an updated dictionary (Konidaris: page 5, Section 5.1, "the option was added to the agent's action repertoire. For new option o, this requires expanding the overall action-value function Q to include o"; when no such option exists the newly created option is added to the agent's repertoire by expanding the action-value structure to include it (add, based on the determination that the identifier associated with the hierarchical state is not included in the dictionary, the identifier associated with the hierarchical state to the dictionary to generate an updated dictionary)); and store the updated dictionary (Konidaris: page 8, Section 7, "select an appropriate abstraction for a new option from a library of candidate abstractions"; the agent maintains its set of discovered options as a stored library of abstractions for reuse (store the updated dictionary)).
Because Sutton and Konidaris are analogous art and within the same field of endeavor, specifically hierarchical reinforcement learning using the options framework, they address the same problem-solving area of discovering and efficiently reusing temporally extended abstractions in an agent-environment Markov decision process. Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the option-creation, duplicate-avoidance, repertoire-augmentation, and maximum-value-initialization bookkeeping of Konidaris with the options-and-value framework of Sutton, with a reasonable expectation of success, such that newly generated abstractions are identified, checked against the agent's stored collection, and added when absent, to teach at least one of the first state, the second state, or the third state being a hierarchical state and associated with a primitive action, the hierarchical state being associated with fewer actions than a state that is not a hierarchical state; determine an identifier associated with the hierarchical state; search a dictionary to determine whether the identifier associated with the hierarchical state is included in the dictionary; add, based on the determination that the identifier associated with the hierarchical state is not included in the dictionary, the identifier associated with the hierarchical state to the dictionary to generate an updated dictionary; and store the updated dictionary. This modification would have been motivated by the desire to avoid creating duplicate options and to accumulate a reusable, stored set of skills as the agent explores (Konidaris: page 3, Section 3, “options that use fewer features than needed to solve the overall problem are desirable”, page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event", page 8, Section 7, "a library of candidate abstractions").
Sutton and Konidaris do not expressly teach an apparatus, comprising: a memory; and a hardware processor operatively coupled to the memory, the hardware processor; a target state by implementing a machine learning model; a dictionary associated with the machine learning model.
However, Hafner teaches an apparatus, comprising: a memory; and a hardware processor operatively coupled to the memory, the hardware processor (Hafner: [0117] "The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data"; an apparatus comprising a memory and a hardware processor operatively coupled to the memory, the hardware processor configured to carry out the recited operations); a target state by implementing a machine learning model ([0039] "trains the policy neural network 120 to generate action selection outputs that can be used to select actions that maximize a cumulative measure of rewards received by the agent and that cause the agent to accomplish an assigned task"; generating action selection outputs (a target state) by training a policy neural network (by implementing a machine learning model)); a dictionary associated with the machine learning model (Hafner: [0039] "select actions that maximize a cumulative measure of rewards received by the agent and that cause the agent to accomplish an assigned task", [0018] "generating a trajectory of latent representations", [0093] "updated latent representation that characterizes a state that the environment would transition into", [0075] "processing the latent representation using a value neural network"; the agent's action-value structure and stored option dictionary being maintained as part of, and associated with, that machine learning model (the dictionary being associated with the machine learning model)).
Because Sutton, in view of Konidaris, and Hafner are analogous art and within the same field of endeavor, specifically hierarchical reinforcement learning using the options framework and neural-network machine learning models, they address the same problem-solving area of implementing a reusable-skill reinforcement-learning agent over an agent-environment Markov decision process. Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to implement the options-and-value and option-bookkeeping framework of Sutton and Konidaris using the neural-network machine learning model of Hafner, with a reasonable expectation of success, such that the agent that determines the target state, maintains the option dictionary, and performs the identifier-search, add, and store operations is implemented by a trained machine learning model, to teach an apparatus, comprising: a memory; and a hardware processor operatively coupled to the memory, the hardware processor; a target state by implementing a machine learning model; a dictionary associated with the machine learning model. This modification would have been motivated by the desire to provide compact latent representations representing the environment including rewards and state transitions to train neural networks in more data-efficient manner requiring fewer computational resources both during training and at run time (Hafner [0010]).
Regarding dependent claim 13, Sutton, in view of Konidaris and Hafner, teach the apparatus of claim 11, wherein the second state is the hierarchical state, the hierarchical state is an abstraction of a fourth state, the hierarchical state is associated with fewer actions than the fourth state (Konidaris: page 3, Section 3, "lightweight options that use fewer features"; the created option is an abstraction associated with fewer actions than the base task states it abstracts (an abstraction of a fourth state; fewer actions than the fourth state)), a third action implements a transition from the first state to the fourth state, a fourth action implements a transition from the fourth state to the third state (Sutton: Section 1, consecutive one-step action transitions connect the base states), the first action is associated with a value combination that is a function of a value associated with the third action and a value associated with the fourth action (Sutton: Section 3, the value of the abstraction is the combination of the values of its constituent action transitions via the Bellman equation), the second action is associated with a maximum value associated with the third state (Sutton: page 185, Section 1, "V ∗(s) = max π V π(s)"; the optimal value is the maximum over actions).
Regarding dependent claim 14, Sutton, in view of Konidaris and Hafner, teach the apparatus of claim 11, wherein the machine learning model is configured to implement a plurality of interactions between the agent and the environment (Hafner: [0039] "the training engine 160 trains, by using reinforcement learning techniques, the policy neural network 120 and the value neural network 130"; the machine learning model comprising the policy and value neural networks carries out the agent's repeated interactions with the environment (implement a plurality of interactions between the agent and the environment)), implement, at a first time, a first set of interactions from the plurality of interactions between the agent and the environment to transition from the first state to the third state via the second state, the first set of interactions being associated with a first value (Sutton: page 189, "for each pair of state and action, an expected cumulative discounted reward"; the trajectory taken directly through the base second state has an associated expected cumulative discounted reward (a first set of interactions ... associated with a first value)), implement, at a second time, a second set of interactions from the plurality of interactions between the agent and the environment to transition from the first state to the third state via the hierarchical state, the second set of interactions being associated with a second value (Konidaris: page 5, Section 5.1, "we therefore sampled the Q values of transitions that triggered the option's target event"; the trajectory routed through the created option (the hierarchical state) has its own sampled Q value (a second set of interactions ... associated with a second value)), the hardware processor being further configured to add the identifier associated with the hierarchical state to the dictionary to generate the updated dictionary further based on a determination that a value combination that is a function of the first value and the second value is greater than a threshold (Sutton: page 199, "the values for the interrupted policy are everywhere greater than the values of the original policy"; the option-routed behavior is adopted, and the option correspondingly added to the agent's repertoire, when the value of the trajectory routed through the option exceeds the value of the original direct trajectory, i.e., when the value combination that is a function of the first value and the second value is greater than the threshold, the recited threshold embracing zero under the broadest reasonable interpretation (a value combination ... is greater than a threshold)).
Regarding dependent claim 16, Sutton, in view of Konidaris and Hafner, teach the apparatus of claim 11, wherein the hierarchical state is generated by merging two or more states (Konidaris: page 2, Section 3, "connecting the option's goal states to every state in its initiation set"; a created option subsumes two or more base states into a single abstraction (generated by merging two or more states)).
Claim 15 is rejected under 35 U.S.C. 103 over Sutton, in view of Konidaris and Hafner, as applied in the rejection of claim 11 above, and further in view of Xu et al. (hereinafter Xu) “Meta-Gradient Reinforcement Learning” (2018).
Regarding dependent claim 15, Sutton, in view of Konidaris and Hafner, teach the apparatus of claim 11, wherein the machine learning model is associated with a set of hyperparameters used to implement a plurality of interactions between the agent and the environment (Konidaris: page 5, Section 5.1, "Sarsa (γ = 1, ϵ = 0.01) with linear function approximation"; the option-learning method is governed by a set of learning hyperparameters including the discount factor γ, the exploration rate ϵ, and the step size "αo = 0.0005"; Hafner: [0033] "A training engine 160 can train the neural networks"; the neural-network machine learning model is trained and governed by a set of training hyperparameters), the hardware processor further configured to (Hafner: [0117]): implement a set of interactions from the plurality of interactions between the agent and the environment to transition from the first state to the third state via the hierarchical state, the set of interactions being associated with receiving a reward signal (Sutton: page 184, Section 1, "the environment produces one step later a numerical reward, rt+1"; each interaction along the trajectory routed through the option (the hierarchical state) yields a numerical reward (associated with receiving a reward signal)).
Sutton, Konidaris, and Hafner do not expressly teach automatically adjust at least one hyperparameter from the set of hyperparameters in response to receiving the reward signal.
However, Xu teaches automatically adjust at least one hyperparameter from the set of hyperparameters in response to receiving the reward signal (Xu: page 3, Section 1.1, "η = {γ, λ}"; page 1, Abstract, "a gradient-based meta-learning algorithm that is able to adapt the nature of the return, online, whilst interacting and learning from the environment"; page 6, Section 3, "adapting one scalar value for γ and λ respectively"; the discount factor γ and the bootstrapping factor λ are hyperparameters that the agent automatically adjusts online from the return computed over the received reward signal (automatically adjust at least one hyperparameter ... in response to receiving the reward signal)).
Because Sutton, in view of Konidaris and Hafner, and Xu are analogous art and within the same field of endeavor, specifically reinforcement-learning agents governed by learning hyperparameters, accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate Xu's online meta-gradient adaptation of the learning hyperparameters into the reinforcement-learning machine learning model of Sutton, Konidaris, and Hafner, with a reasonable expectation of success, so that at least one hyperparameter governing the agent is automatically adjusted in response to the reward signal, to teach automatically adjusting at least one hyperparameter from the set of hyperparameters in response to receiving the reward signal. This modification would have been motivated by automatically adapting the hyperparameters online based on the agent's performance to improve performance (Xu: page 1).
Claims 17-21 are rejected under 35 U.S.C. 103 over Marvin et al. (hereinafter Marvin), US 2021/0387330 A1, in view of Konidaris, and further in view of Tamar et al. (hereinafter Tamar) “Policy Gradients with Variance Related Risk Criteria” (2012) .
Tamar was disclosed in an IDS dated 4/27/2026.
Regarding independent claim 17, Mavrin teaches a non-transitory processor-readable medium storing code representing instructions to be executed by a processor, the instructions comprising code to cause the processor to (Mavrin: [0018] “Another example embodiment is a non-transitory computer-readable medium”; the methods are embodied as processor-executable instructions stored on a computing system):
receive data associated with interactions between a first agent and a first environment associated with a domain (Mavrin: [0044] "the RL agent receives the current state of the environment, computes the reward corresponding to the current state, and generates an action from an action space", [0052] "the RL agent 102 is configured to learn skills (i.e., option policies) required to solve the first task"; the reinforcement-learning agent solves a first task within an environment and records the resulting interaction data (a first agent; a first environment));
receive information about a second environment associated with the domain, the information including a goal that is desired to be achieved in the second environment (Mavrin: [0009] "a robot reuses the knowledge learned from solving one task within an environment using reinforcement learning to efficiently solve another novel task", [0016] "learning a second policy to maximize a future cumulative reward for a second (different) task"; the agent is directed to a second related task whose objective is to maximize the future cumulative reward (a second environment; a goal), and);
implement, using a machine learning model, a second agent configured to interact with the second environment (Mavrin: [0010] "The second policy chooses an option policy from the learned option policies. The chosen option policy generates an action for the given state"; the reinforcement-learning agent operating under the second policy interacts with the second task (a machine learning model; a second agent));
identify, based on the data associated with the interactions between the first agent and the first environment, a set of actions configured to be performed by the second agent while the second agent interacts with the second environment (Mavrin: [0119] "forms an augmented action space 130 that includes the learned option policies... and a set of primitive actions"; the agent forms an augmented action space from the option policies learned in the first task together with primitive actions, and selects from that set in the second task (a set of actions configured to be performed by the second agent)); based on action being associated with a value above a predetermined threshold (Mavrin: [0010], "The second policy chooses an option policy from the learned option policies"; the second policy selects, from the augmented action space, the option policy that maximizes the future cumulative reward, that is, an action associated with a value exceeding that of the non-selected actions and hence above a predetermined threshold (a value above a predetermined threshold)).
Mavrin does not expressly teach and implement the second agent to perform an action from the set of actions and perceive the second environment to transition from a first state to a second state,
the second state being a hierarchical state and increasing an efficiency in computation associated with achieving the goal (interpreted as the second state being a hierarchical state based on the hierarchical state configured to increase the being associated with a value above a predetermined threshold and to increase an efficiency in computational associated with achieving the goal by the hierarchical state being associated with fewer actions than a state that is not a hierarchical state per the 35 U.S.C. 112(b) rejection set forth above).
However, Konidaris teaches implement the second agent to perform an action from the set of actions and perceive the second environment to transition from a first state to a second state, the second state being a hierarchical state (Konidaris: page 1, Section 2, "agent chooses when to execute them in the same way it chooses when to execute primitive actions", page 3, Section 4.1, "to reach a state on which T evaluates to 1"; the agent executes a selected option and observes the resulting transition to a state reached by the temporally extended abstraction (perform an action... perceive... transition from a first state to a second state, the second state being a hierarchical state), the agent and an option is created), the second state being a hierarchical state based on the hierarchical state being associated with a value above a predetermined threshold (Konidaris: page 4, Section 4.3 “More than one option may be created to reach a target event”, pages 5-6, Section 5.1, "action-value function Q…initialized Q(s,o) to the maximum of these values"; the created hierarchical state (option) is initialized to the maximum of the sampled transition values and is created (the second state being a hierarchical state based on the hierarchical state) and retained when its value satisfies the creation threshold (being associated with a value above a predetermined threshold)), and to increase an efficiency in computation associated with achieving the goal by the hierarchical state being associated with fewer actions than a state that is not a hierarchical state (Konidaris: page 3, Section 3, "lightweight options that use fewer features than needed to solve the overall problem are desirable", page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event"; the created option is lightweight (and to increase an efficiency in computation associated with achieving the goal by the hierarchical state being associated with), using fewer features and thereby fewer constituent actions than the overall task (fewer actions than a state that is not a hierarchical state)).
Because Mavrin and Konidaris are analogous art and within the same field of endeavor, specifically reinforcement learning with reusable temporally extended options, they address the same problem-solving area of transferring learned skills from a first task to accelerate a second task, accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the hierarchical-state characterization, value-threshold gating, and fewer-actions efficiency taught by Konidaris with the cross-task option-reuse framework of Mavrin, with a reasonable expectation of success, such that the reused action carries the second agent to a hierarchical state selected on the basis of a value above a predetermined threshold while reducing the number of actions, to teach and implement the second agent to perform an action from the set of actions and perceive the second environment to transition from a first state to a second state, the second state being a hierarchical state configured to increase the likelihood of achieving the goal based on the hierarchical state configured to increase the being associated with a value above a predetermined threshold and to increase an efficiency in computational associated with achieving the goal by the hierarchical state being associated with fewer actions than a state that is not a hierarchical state. This modification would have been motivated by the desire to avoid creating duplicate options and to accumulate a reusable, stored set of skills as the agent explores (Konidaris: page 3, Section 3, “options that use fewer features than needed to solve the overall problem are desirable”, page 4, Section 4.3, "we do not create a new option when a target event is triggered from a state already in the initiation set of an option targeting that event", page 8, Section 7, "a library of candidate abstractions").
Mavrin and Konidaris do not expressly teach the action being configured to increase a likelihood of achieving the goal, the second state being a hierarchical state configured to increase the likelihood of achieving the goal.
However, Tamar teaches the action being configured to increase a likelihood of achieving the goal, the second state being a hierarchical state configured to increase the likelihood of achieving the goal (Tamar: page 1, Abstract, "Managing risk in dynamic decision problems is of cardinal importance in many fields such as finance", pages 1-2, Section 1, “Sharpe Ratio…ratio between the expected profit and its standard deviation”, page 7, Section 5 Experiments; the reinforcement-learning policy-gradient method is optimized under a variance-related risk criterion, such as the Sharpe ratio and standard-deviation-adjusted reward, so that the selected action and resulting state increase the likelihood of achieving a desirable return (increase a likelihood of achieving the goal), and the method's applicability is demonstrated on a portfolio planning problem).
Because Mavrin, in view of Konidaris, and Tamar are analogous art and within the same field of endeavor, specifically risk-sensitive reinforcement learning with reusable temporally extended options, they address the same problem-solving area of selecting reused skills so as to increase the likelihood of a desirable outcome while transferring knowledge across tasks, accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the risk-sensitive action selection taught by Tamar with the cross-task option-reuse framework of Mavrin as modified by Konidaris, with a reasonable expectation of success, such that the reused action and the perceived hierarchical state are selected so as to increase the likelihood of achieving the goal, to teach the action being configured to increase a likelihood of achieving the goal based on the action being associated with a value above a predetermined threshold; the second state being a hierarchical state configured to increase the likelihood of achieving the goal based on the hierarchical state configured to increase the being associated with a value above a predetermined threshold and to increase an efficiency in computational associated with achieving the goal by the hierarchical state being associated with fewer actions than a state that is not a hierarchical state. This modification would have been motivated by the desire to manage risk and increase the likelihood of a desirable outcome, such as a higher Sharpe ratio, when optimizing the agent's policy (Tamar: page 1, Abstract and Section 1).
Regarding dependent claim 18, Mavrin, in view of Konidaris and Tamar, teach the non-transitory processor-readable medium of claim 17, wherein the first agent is the same as the second agent (Mavrin: [0051] "RL agent 102"; a single reinforcement-learning agent both solves the first task and is reused on the second task (the first agent is the same as the second agent), the same learns the option policies in the first task and applies them under the second policy).
Regarding dependent claim 19, Mavrin, in view of Konidaris and Tamar, teach the non-transitory processor-readable medium of claim 17, wherein the goal is a second goal, and the first environment is implemented to perform a first task, to achieve a first goal, in the domain and the second environment is implemented to perform a second task, to achieve the second goal, in the domain (Mavrin: [0009], "reuses the knowledge learned from solving one task... to efficiently solve another novel task"; the agent performs a first task with a first objective and then a related second task with a second objective in the same problem setting (a first task, to achieve a first goal; a second task, to achieve the second goal)).
Regarding dependent claim 20, Mavrin, in view of Konidaris and Tamar, teach the non-transitory processor-readable medium of claim 19, wherein the domain is at least one of financial trading, agricultural technology, or natural language processing (NLP) (Tamar: page 1, Abstract, "Managing risk in dynamic decision problems is of cardinal importance in many fields such as finance…applicability in a portfolio planning problem"; page 7, Section 5 (Experiments); the reinforcement-learning policy-gradient method optimizing a variance-related risk criterion such as the Sharpe ratio is applied to a financial-trading domain (financial trading)).
Regarding dependent claim 21, Mavrin, in view of Konidaris and Tamar, teach the non-transitory processor-readable medium of claim 17, wherein the action is a first action, and the instructions comprising code to cause the processor to implement the second agent to perform the first action include code to cause the processor to implement the second agent to perform an option that includes the first action, the option including a plurality of actions that includes the first action (Mavrin: [0043] "An option is a policy that can be executed for multiple time steps before terminating and switching to another option"; the reused action is itself an option policy executed over multiple time steps, that is, an option comprising a plurality of actions that includes the action), performing the option increasing the likelihood of achieving the goal (interpreted as performing the option increases the likelihood of achieving the goal based on the option being associated with a value above a predetermined threshold per the 35 U.S.C. 112(b) rejection set forth above) (Mavrin: [0016] the second policy chooses the option policy that maximizes the future cumulative reward, that is, an option associated with a value above a predetermined threshold), and increasing the efficiency in computation associated with achieving the goal (interpreted as and increases the efficiency in computation associated with achieving the goal by reducing a number of actions that the second agent can take relative to a state sequence that does not include the option per the 35 U.S.C. 112(b) rejection set forth above) (Konidaris: page 3, Section 3, "lightweight options that use fewer features than needed to solve the overall problem are desirable"; consistent with the broadest reasonable interpretation set forth in the 35 U.S.C. 112(b) rejection above, the recited increases are interpreted to require any non-zero improvement, however small, relative to a state sequence that does not include the option).
Allowable Subject Matter
Claims 6-7 are objected to as being dependent on a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
The closest prior arts uncovered, when taken individually or in combination do not expressly teach or render obvious the limitations recited in dependent claims 6-7 when taken in the context of the claims as a whole.
At best the closest prior arts uncovered, specifically, Sutton (1999) disclose: Learning, planning, and representing knowledge at multiple levels of temporal abstraction are key, longstanding challenges for AI. In this paper we consider how these challenges can be addressed within the mathematical framework of reinforcement learning and Markov decision processes (MDPs). We extend the usual notion of action in this framework to include options - closed-loop policies for taking action over a period of time. Examples of options include picking up an object, going to lunch, and traveling to a distant city, as well as primitive actions such as muscle twitches and joint torques. Overall, we show that options enable temporally abstract knowledge and action to be included in the reinforcement learning framework in a natural and general way. In particular, we show that options may be used interchangeably with primitive actions in planning methods such as dynamic programming and in learning methods such as Q-learning. Formally, a set of options defined over an MDP constitutes a semi-Markov decision process (SMDP), and the theory of SMDPs provides the foundation for the theory of options. However, the most interesting issues concern the interplay between the underlying MDP and the SMDP and are thus beyond SMDP theory. We present results for three such cases: (1) we show that the results of planning with options can be used during execution to interrupt options and thereby perform even better than planned, (2) we introduce new intra-option methods that are able to learn about an option from fragments of its execution, and (3) we propose a notion of subgoal that can be used to improve the options themselves. All of these results have precursors in the existing literature; the contribution of this paper is to establish them in a simpler and more general setting with fewer changes to the existing reinforcement learning framework. In particular, we show that these results can be obtained without committing to (or ruling out) any particular approach to state abstraction, hierarchy, function approximation, or the macro-utility problem (Abstract); and Kondidaris (2009) disclose: We introduce a skill discovery method for reinforcement learning in continuous domains that constructs chains of skills leading to an end-of-task reward. We demonstrate experimentally that it creates appropriate skills and achieves performance benefits in a challenging continuous domain (Abstract).
Response to Arguments
Applicant’s claim amendments and Remarks filed 4/27/2026 with respect to the claim objects set forth in the Office Action dated 10/28/2025 are persuasive and thus the said claim objects are withdrawn.
Applicant’s claim amendments and Remarks filed 4/27/2026 with respect to the 35 U.S.C. 101 rejections set forth in the Office Action dated 10/28/2025 are persuasive and thus the said 35 U.S.C. 101 rejections are withdrawn.
Applicant’s claim amendments and Remarks filed 4/27/2026 with respect to the rejections under 35 U.S.C. 102/103 have been considered but are moot because the
new ground of rejection, necessitated by Applicant’s amendment, does not rely on any references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights