DETAILED ACTION
This action is responsive to the amended claims filed on 05/26/2026. Claims 1, 3-12, and 14-20 are pending for examination.
This action is Final.
Response to Amendments/Remarks
Applicant’s arguments with respect to 35 U.S.C. 103 rejection of the claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-12, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yuan et al. (Xingdi Yuan et al., “Counting to Explore and Generalize in Text-based Games,” arXiv:1806.11525, first published June 29, 2018), hereinafter referred to as Yuan, in view of Aineto et al. (Diego Aineto, Sergio Jiménez, and Eva Onaindia, “Learning STRIPS Action Models with Classical Planning,” arXiv:1903.01153, first published March 4, 2019), hereinafter referred to as Aineto, and in further view of Wang et al. (Christopher Wang et al., “Learning a Natural-Language to LTL Executable Semantic Parser for Grounded Robotics,” arXiv:2008.03277, first published August 7, 2020), hereinafter referred to as Wang, and Hayton et al. (Thomas Hayton et al., “Narrative Planning Model Acquisition from Text Summaries and Descriptions,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 2, pp. 1709-1716, published April 3, 2020), hereinafter referred to as Hayton.
Claim 1: Regarding claim 1, Yuan teaches the following limitations:
performing an automated intervention action in a text-based environment (Yuan, page 1, section 2, “Actions (A): At each turn t, the agent issues a text command ct. The interpreter can accept any sequence of characters but will only recognize a tiny subset thereof. Furthermore, only a fraction of recognized commands will actually change the state of the world.”, Yuan teaches that an “agent issues a text command” and that recognized commands can change the state of the world. The agent is an automated reinforcement-learning agent, the command is an intervention action, and the text game is the claimed text-based environment.)
performing an inverse action in the text-based environment to reverse the automated intervention action (Yuan, page 3, col. 2, paragraph 1, “To solve easy games, the agent must learn to recall its previous directional action and to issue the command that does not reverse it (e.g., if the agent entered the current room by going east, do not now go west). Conversely, to solve medium and hard games, the agent must reverse its previous action when it enters distractor rooms to return to the chain, and also recall farther into the past to track which exits it has already passed through.”, Yuan teaches that the agent must “reverse its previous action” after entering a distractor room in order to return to the main chain. Thus, Yuan expressly teaches an action that reverses the preceding intervention and restores the agent to the preceding path or location.)
Aineto, in the field of learning symbolic action models from executions, teaches the following limitations which Yuan does not expressly teach:
wherein the automated intervention action is associated with an action template, and the action template includes one or more precondition propositions for success of the automated intervention action and one or more effect propositions that occur upon success of the automated intervention action (Aineto, page 399, col. 1, last paragraph, “When learning action models the output is, for each action, the preconditions, negative and positive effects that define the possible state transitions.”, Aineto formalizes an action as an instance of a STRIPS operator schema having a name, parameters, “preconditions, negative and positive effects.” Aineto further explains that the action is applicable when its preconditions hold and that applying the action generates the successor state according to its effects. The STRIPS operator schema therefore corresponds to the claimed action template, its preconditions correspond to the claimed precondition propositions for successful execution, and its positive and negative effects correspond to the claimed effect propositions.)
recording states of the text-based environment before and after the automated intervention action and the inverse action (Aineto, page 400, col. 2, last paragraph, “Learning STRIPS action models from fully available input knowledge, i.e. from plans where the pre and post-states of every action in the plans are known, is straightforward. When intermediate states are available, operator schemas are derived lifting the literals that change between the pre and post-state of each action execution. Preconditions of an action are derived lifting the minimal set of literals that appears in all the pre-states of the corresponding action”, teaches learning from plans in which the pre-state and post-state of each action execution are known and further teaches that an action sequence induces a corresponding sequence of states. Applied to forward intervention actions and reversing actions, Aineto teaches retaining the state immediately before and after each respective action.)
evaluating the recorded states to generate training data including pseudo-labels for logical propositions that encode an outcome of a plurality of rules, wherein the logical propositions include the one or more precondition propositions and the one or more effect propositions (Aineto, page 400, col. 2, last paragraph, “Learning from (initial, final) state pairs. This learning task amounts to observing agents acting in the world but”, teaches a set of initial/final state pairs watching only the result of their plans execution. No inter-mediate information about the actions in the plans is given.”;
Page 400, col. 1, paragraph 1, “We use F to denote the set of fluents(propositional variables) describing a state. A literall is a valuation of a fluent f∈F; i.e. either l=f or l=¬f. A set of literals L represents a partial assignment of values to fluents (without loss of generality, we will assume that L does not contain conflicting values). We use L(F)to denote the set of all literal sets on F; i.e. all partial assignments of values to fluents. A classical planning frame is a tuple Φ=〈F,A〉, where F is a set of fluents and Ais a set of actions. An action a ∈ A is defined with preconditions, pre(a)⊆L(F),positive effects, eff+(a)⊆L(F), and negative effects eff−(a)⊆L(F). We say that an action a∈A is applicable in a state s iff pre(a)⊆s. The result of applying a in s is the successor state denoted by θ(s, a)={s\eff−(a))∪eff+(a)}.An action a∈A with conditional effects is defined as as et of preconditions pre(a)and a set of conditional effects cond(a). Each conditional effect CE∈cond(a) is com-posed of two sets of literals: C⊆L(F), the condition, and E⊆L(F), the effect. An action a∈A is applicable in a state s iff pre(a)⊆s, and the triggered effects resulting from the action application are the effects whose conditions hold ins”, teach deriving operator preconditions from recurring pre-state literals and deriving effects from changed literals. The state propositions and learned precondition/effect memberships are truth-valued symbolic labels obtained by applying multiple action-model derivation rules to recorded execution states, and therefore constitute pseudo-label training information for logical propositions.)
wherein the plurality of rules includes a rule where, if a proposition is listed in the action template as an effect proposition of the one or more effect propositions of the automated intervention action but is not listed in the action template as a precondition proposition of the one or more precondition propositions of the automated intervention action, then the pseudo-label indicates that the proposition was false in an initial state before the automated intervention action (Aineto, page 400, col. 1, paragraph 1, “An action a ∈ A is defined with preconditions, pre(a) ⊆ L(F), positive effects, eff+(a) ⊆ L(F), and negative effects eff−(a) ⊆ L(F). We say that an action a ∈ A is applicable in a state s iff pre(a) ⊆ s. The result of applying a in s is the successor state denoted by θ(s, a) = {s \ eff−(a)) ∪ eff+(a)}.”; page 400, col. 2, last paragraph, “When intermediate states are available, operator schemas are derived lifting the literals that change between the pre and post-state of each action execution. Preconditions of an action are derived lifting the minimal set of literals that appears in all the pre-states of the corresponding action.” Aineto therefore teaches an action template/action schema having separately identified preconditions and positive effects, and further teaches deriving the effect portion of the schema from literals that change between the pre-action state and the post-action state. Thus, when a proposition is identified in the action schema as a positive effect because it changed from the pre-state to the post-state, while the proposition is not identified in the separately listed precondition set for the action schema, the recorded pre-state/post-state comparison indicates that the proposition was false before execution and true after execution. Accordingly, Aineto teaches the claimed rule outcome for a proposition listed as an effect proposition but not listed as a precondition proposition.)
wherein the PDDL is for generation of a sequence of actions that takes an agent from a current state to a goal state (Aineto, page 400, col. 1, paragraph 3, “Our approach is aimed at learning PDDL action schemas that follow the Strips requirement.” Aineto further illustrates a PDDL action schema including “:action,” “:parameters,” “:precondition,” and “:effect.” Aineto, page 400, col. 1, last paragraph, further teaches, “A classical planning problem is a tuple P = 〈F, A, I, G〉, where I is an initial state and G ⊆ L(F) is a goal condition. A plan for P is an action sequence π = 〈a1, …, an〉 that induces the state trajectory 〈s0, s1, …, sn〉 such that s0 = I and ai (1 ≤ i ≤ n) is applicable in si−1 and generates the successor state si = θ(si−1, ai).”, Aineto therefore first identifies the learned action schemas as PDDL action schemas and then teaches that such action/planning representations are used in a planning problem having an initial state and a goal condition to generate an action sequence that transitions from the initial state to a state satisfying the goal. The initial state at the time planning begins corresponds to the claimed current state.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to apply Aineto’s action-model learning technique to the automated executions of Yuan’s text-game agent. Yuan supplies an automated agent that changes a text environment through commands and reverses prior commands, while Aineto supplies a known technique for representing each such action by a parameterized symbolic schema and deriving the action’s preconditions and effects from observed pre-action and post-action states. The combination would have predictably produced reusable action templates and proposition labels describing the state transitions caused by Yuan’s actions, thereby enabling planning and generalization from recorded interaction data. Aineto expressly identifies learning action models from observed plan executions as the purpose of its technique, while Yuan seeks policies that generalize to unseen games; accordingly, the references provide a reason to combine their complementary teachings.
Wang, in the field of training executable semantic parsers using environment executions, teaches the following limitations which Yuan and Aineto do not teach:
A computer-implemented method for training a semantic parser (Wang, page 1, Abstract, “Children acquire their native language with apparent ease by observing how language is used in context and attempting to use it themselves. They do so without laborious annotations, negative examples, or even direct corrections. We take a step toward robots that can do the same by training a grounded semantic parser, which discovers latent linguistic representations that can be used for the execution of natural-language commands.”,
Page 3, section 2, paragraph 2, “Our work follows recent approaches that cast the problem of semantic parsing as a machine translation task. Instead of using chart parsers, as is typical for grammar-based approaches [12, 14, 13], we use an encoder-decoder sequence-to-sequence model where the input and output are sequences of tokens; in our case, natural language commands and LTL formulas, respectively.”, Wang teaches “training a grounded semantic parser.” Wang further explains that the parser is an encoder-decoder neural architecture that maps a natural-language command to a formal logical representation. Wang therefore teaches a computer-implemented method directed to training a semantic parser neural network model.)
training a semantic parser neural network model using the training data (Wang, page 1, Abstract, “Our parser is trained with pairs of sentences and executions as well as an executor. At training time, the parser hypothesizes a meaning representation for the input as a formula in LTL.”, teaches that its parser is trained with “pairs of sentences and executions.”
Page 2, figure 1, “The paradigm for training the parser is shown on the left. Given an input sentence, the parser proposes an LTL formula, z, that could encode the meaning of that sentence. That formula is executed by a robotic agent to estimate the likelihood of the observed behavior, E, given the interpretation of the sentence, i.e., determining whether many more efficient approaches to executing this formula might exist. This likelihood is used to compute the reward R(z,E). At the same time, a generator attempts to reconstruct the sentence. (right) An example of the planner in action. We use the planner described by Kuo et al. [3] which learns to execute LTL formulas end-to-end from pixels to actions. Each predicate, ORANGE and TREE here, and each operator, are neural networks which together output an action that the robot should take given the current state of the world. Not shown are recurrent connections that enable each component to keep track of execution progress.” and paragraph 1, “Second, we provide feedback from an executor: a planner trained end-to-end that is capable of executing formulas in whatever formalism we use in the prior, in our case LTL.”, Wang further teaches a neural sequence-to-sequence parser in which execution demonstrations are used by a planner/executor to score candidate logical formulas and update the parser parameters. Thus, Wang teaches training a semantic parser neural network using training information derived from recorded environment executions.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date to use Yuan/Aineto’s execution-derived logical information as supervision for Wang’s neural semantic parser. Wang expressly trains from sentence/execution pairs and uses an executor to determine whether a hypothesized formal meaning explains the observed trajectory. Aineto provides structured truth information concerning preconditions, effects, and state transitions derived from those same types of execution records. Supplying such information to Wang’s parser would have predictably reduced manual annotation and improved the parser’s ability to associate natural-language descriptions with executable action semantics. The combination therefore amounts to using known execution-derived symbolic labels in Wang’s known execution-supervised semantic-parser training process for their established purpose.
Hayton, in the field of automatically acquiring planning-domain models from natural-language descriptions, teaches the following limitation which Yuan, Aineto, and Wang do not expressly teach:
generating, using the trained semantic parser neural network model, a planning domain definition language (PDDL) for a new environment (Hayton, page 1709, Abstract, “We seek to support authors in this task by allowing natural language (NL) plot synopses to be used as a starting point from which planning domain models can be automatically acquired. We present a solution which analyses input NL text summaries, and builds structured representations from which a PDDL model is output (fully automated or author in-the loop). We introduce a novel sieve-based approach to pronoun resolution that demonstrates consistently high performance across domains. In the paper we focus on authoring of narrative planning models for use in interactive entertainment systems and games… Our results and an ex pert user assessment show that our approach enables a reduction in authoring effort required to generate baseline narrative domain models from which variants can be built.”, Hayton teaches analyzing natural-language text summaries, building structured representations, and that a “pddl model is output.” Hayton further teaches automatically acquiring a baseline planning-domain model from the input description. In the proposed combination, Hayton supplies PDDL as the known planning-domain output format from the trained neural semantic parser that produces the structured executable meaning representation.)
wherein the sequence of actions, the agent, the current state, and the goal state are associated with the new environment (Hayton, page 1710, col. 1, paragraph 2, teaches that the system assumes “a planning problem in PDDL” separated into “the domain model,” which is “a definition of the problem domain that defines the world and its behaviours,” and “an explanation of the specific problem to be solved within that world.” Hayton, page 1714, col. 1, paragraph 1, further teaches that the “system generated baseline models were sufficient to ensure generation of narrative plans corresponding to an original input synopsis when used with a suitable narrative planning problem.” Accordingly, Hayton teaches that the generated PDDL model corresponds to the particular world/domain described by the input text, and that the planning problem and generated plan are for that world/domain. Thus, the generated sequence of actions, the acting entities/characters corresponding to the claimed agent, and the starting and goal conditions of the suitable planning problem are associated with the newly modeled environment)
It would have been obvious to a person of ordinary skill in the art before the effective filing date to configure Yuan/Aineto/Wang’s trained semantic parser to output the PDDL planning representation taught by Hayton. Wang teaches mapping natural language to an executable logical formalism and expressly states that the technique is not limited to LTL, while Hayton teaches PDDL as a known machine-usable formalism automatically generated from natural-language domain descriptions. Selecting PDDL as Wang’s target formalism would have been a predictable substitution of one known executable symbolic language for another and would have enabled the generated representation to be consumed by conventional planning software to generate an action sequence from the current state to the goal state, as taught by Aineto. A person of ordinary skill would have had a reasonable expectation of success because both Wang and Hayton convert natural-language information into formal representations intended for execution or planning.
Claim 11: Regarding claim 11, claim 11 recites limitations substantially similar to claim 1, as such a similar analysis applies. Claim 11 also recites the following additional limitations for consideration which Wang further teaches:
a non-transitory computer readable storage medium comprising a computer readable program for training a semantic parser, wherein the computer readable program when executed on a computer causes the computer to perform the recited operations (Wang, pages 3, section 2, paragraph 1, “The approaches of [12, 13] present a planner and CCG-based parser to generate programs for a deterministic robotic simulator.”, Wang implements its neural semantic parser, encoder, decoder, planner scoring, and training objectives as computer-executed model components and algorithms. It would have been obvious to store the executable program instructions and trained model parameters on a conventional non-transitory computer-readable storage medium so that a computer could execute the disclosed training and inference operations.)
Claim 12: Regarding claim 12, claim 12 recites a system implementation of the substantive process recited in claim 1. Claim 12 recites the following additional limitations for consideration which Yuan further teaches:
an exploration agent configured to perform the automated intervention action and the inverse action in the text-based environment (Yuan, page 1, section 2, “Actions (A): At each turn t, the agent issues a text com mand ct. The interpreter can accept any sequence of char acters but will only recognize a tiny subset thereof. Furthermore, only a fraction of recognized commands will actually change the state of the world.”, Yuan teaches an automated reinforcement-learning agent that issues text commands, changes the game state, and reverses a prior action to exit a distractor room. Yuan’s automated text-game agent therefore corresponds to the claimed exploration agent.)
Aineto further teaches:
the exploration agent configured to record states before and after the automated intervention action and the inverse action (Aineto, page 400, col. 2, last paragraph, “Learning STRIPS action models from fully available input knowledge, i.e. from plans where the pre and post-states of every action in the plans are known, is straightforward. When intermediate states are available, operator schemas are derived lifting the literals that change between the pre and post-state of each action execution. Preconditions of an action are derived lifting the minimal set of literals that appears in all the pre-states of the corresponding action”, Aineto teaches recorded pre-states, post-states, and state sequences associated with actions in an observed plan execution. Applying that recording process to Yuan’s forward and reversing actions teaches the recited state-recording function of the exploration agent.)
a state evaluator configured to evaluate the recorded states to generate training data including the recited pseudo-labels and rule outcome (Aineto, page 400, col. 2, last paragraph, “Learning STRIPS action models from fully available input knowledge, i.e. from plans where the pre and post-states of every action in the plans are known, is straightforward. When intermediate states are available, operator schemas are derived lifting the literals that change between the pre and post-state of each action execution. Preconditions of an action are derived lifting the minimal set of literals that appears in all the pre-states of the corresponding action”, Aineto teaches deriving preconditions from pre-states, deriving effects from changed literals, and representing initial/final state pairs as labels. A software component performing these evaluations corresponds to the claimed state evaluator.)
and the PDDL is for generation of the recited current-state-to-goal-state action sequence (Aineto, page 400, col. 1, paragraph 3, “Our approach is aimed at learning PDDL action schemas that follow the Strips requirement.” Aineto further illustrates a PDDL action schema including “:action,” “:parameters,” “:precondition,” and “:effect.” Aineto, page 400, col. 1, last paragraph, further teaches, “A classical planning problem is a tuple P = 〈F, A, I, G〉, where I is an initial state and G ⊆ L(F) is a goal condition. A plan for P is an action sequence π = 〈a1, …, an〉 that induces the state trajectory 〈s0, s1, …, sn〉 such that s0 = I and ai (1 ≤ i ≤ n) is applicable in si−1 and generates the successor state si = θ(si−1, ai).”, Aineto therefore first identifies the learned action schemas as PDDL action schemas and then teaches that such action/planning representations are used in a planning problem having an initial state and a goal condition to generate an action sequence that transitions from the initial state to a state satisfying the goal. The initial state at the time planning begins corresponds to the claimed current state.)
Wang further teaches:
a system for training a semantic parser, comprising a hardware processor and a memory that stores computer program code which, when executed by the hardware processor, implements the recited components (Wang, pages 3, section 2, paragraph 1, “The approaches of [12, 13] present a planner and CCG-based parser to generate programs for a deterministic robotic simulator.”, teaches a computer-implemented neural encoder-decoder parser, planner/executor scoring process, and model-training algorithms. Implementing these disclosed software components using a processor executing code stored in memory would have been an ordinary and predictable computer implementation of Wang’s expressly computational system.)
a model trainer configured to train a semantic parser neural network model using the training data (Wang, pages 5, paragraph 1, “Formulas are scored using a planner. We adopt the one described by Kuo et al. [3] because it learns to execute LTL formulas end to end and is pretrained for the CRAFT environment that we evaluate on. Given a formula and an environment, the planner is trained to create an execution sequence; it never has access to the natural language utterance. It learns to extract features from images of the environment around the robot and acquires knowledge about LTL predicates and operators in order to execute novel formulas in novel environments. Figure 1 (right) shows an example of the planner configured to execute an LTL formula. Given an LTL formula, the planner is configured by assembling a compositional recurrent network specific to that formula; it then guides the robot to execute the formula. This compositionality enables zero-shot generalization to new formulas. Any planner could in principle be used as long as it could learn to execute formulas for the target domain and if it could score an arbitrary trajectory against a formula.”, Wang teaches updating the parameters of a neural semantic parser according to planner/executor scores obtained from command-execution training examples. Wang’s training component corresponds to the claimed model trainer.)
Hayton further teaches:
wherein the trained semantic parser neural network model generates a PDDL for a new environment (Hayton, page 1709, Abstract, “We seek to support authors in this task by allowing natural language (NL) plot synopses to be used as a starting point from which planning domain models can be automatically acquired. We present a solution which analyses input NL text summaries, and builds structured representations from which a PDDL model is output (fully automated or author in-the loop). We introduce a novel sieve-based approach to pronoun resolution that demonstrates consistently high performance across domains. In the paper we focus on authoring of narrative planning models for use in interactive entertainment systems and games… Our results and an ex pert user assessment show that our approach enables a reduction in authoring effort required to generate baseline narrative domain models from which variants can be built.”, Hayton further explains at page 1710 that the input is a natural-language description of a domain and that the “target output is a planning domain model.” Thus, Hayton teaches generating a PDDL planning-domain model corresponding to the particular world or environment described by the natural-language input.)
Claim 3: Regarding claim 3, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Wang further teaches the additional limitation:
The computer-implemented method of claim 1, wherein the training of the semantic parser neural network model includes supervised learning using the pseudo-labels (Wang, page 6, section 4.2, “Iterative Maximum Likelihood,” teaches, “First, we explore the output space by sampling K formulas from the parser. We keep the highest reward formulas ẑ* and use them as a pseudo-gold. We then maximize the likelihood of the pseudo-gold formulas over the course of 10 epochs.” Wang therefore teaches automatically selecting candidate logical formulas based on execution-derived reward and using the selected formulas as pseudo-gold target outputs for training the parser. The pseudo-gold formulas correspond to pseudo-labels because they are automatically generated labels selected without manually annotated ground-truth logical forms. Maximizing the likelihood of the parser output against those pseudo-gold target formulas constitutes supervised learning using the pseudo-labels because the parser is trained to predict the selected pseudo-gold targets from the corresponding input commands.)
Claim 4: Regarding claim 4, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Aineto further teaches the additional limitation:
The computer implemented method of claim 1, wherein the plurality of rules derives one or more pseudo-labels from the action template (Aineto, page 401, col. 1, paragraph 1, “This learning task is formalized as Λ=〈Ψ,Σ〉Σ={σ1,...,στ}is a set of(initial, final)state pairs called labels. Each label σt=(st0,stn),1≤t≤τ, comprises the final state stn resulting from executing an un-known plan πt=〈at1,...,atn〉in the initial state st0.”, teaches initial/final state pairs “called labels,” operator schemas containing preconditions and effects, and rules that derive preconditions from recurring pre-state literals and effects from changed pre-state/post-state literals. In the combined system, the rule outcome identifying a proposition’s prior truth value is derived from its membership in the learned action template’s precondition and effect sets. Thus, the pseudo-label is derived from the action template.)
Claim 5: Regarding claim 5, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Aineto further teaches the additional limitation:
The computer implemented method of claim 1, wherein the action template further includes one or more parameters that the automated intervention action accepts (Aineto, page 400, col. 2, paragraph 4, “Finally, we assume that an action a∈A is instantiated from a STRIPS operator schema ξ=〈head(ξ),pre(ξ),add(ξ),del(ξ)〉”, Aineto defines the operator header by its name and corresponding variable names and provides example operator schemas such as pickup(v1), stack(v1, v2), and unstack(v1, v2). The variables v1 and v2 are parameters accepted by the instantiated action, and therefore teach the claimed one or more action-template parameters.)
Claim 6: Regarding claim 6, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Aineto further teaches the additional limitation:
The computer implemented method of claim 1, wherein the plurality of rules further includes a rule selected from the group consisting of a first rule relating to the one or more precondition propositions for the success of the automated intervention action, a second rule relating to the one or more effect propositions for the success of the automated intervention action, and a third rule relating to the one or more precondition propositions for the success of the automated intervention action that are not canceled in the one or more effect propositions (Aineto, page 400, col. 2, paragraph 5, “pre(ξ)⊆Fv is the set of preconditions, del(ξ)⊆Fv the negative effects and add(ξ)⊆Fv the positive effects such that del(ξ)⊆pre(ξ),del(ξ)∩add(ξ)=∅and pre(ξ)∩add(ξ)=∅.”, Aineto expressly defines each operator schema using a precondition set, a positive-effect set, and a negative-effect set and separately teaches deriving the precondition and effect memberships from execution states. Aineto therefore teaches at least the first and second alternatives of the claimed Markush group. Because claim 6 requires a rule selected from the recited group, disclosure of either one of those alternatives satisfies the limitation.)
Claim 7: Regarding claim 7, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Wang further teaches the following which Yuan and Aineto fails to teach:
The computer-implemented method of claim 1, wherein the plurality of rules further include a rule that sets a noisy pseudo-label based on a determination that the automated intervention action is unsuccessful (Wang, page 1, Abstract, teaches that the parser is trained using “pairs of sentences and executions” and that, at training time, “the parser hypothesizes a meaning representation for the input as a formula in LTL.” Wang further teaches that the executor determines whether the observed trajectories are likely executions of the hypothesized meaning. Wang, page 6, section 4.2, further teaches selecting the highest-reward candidate formulas and using them as “pseudo-gold,” while formulas that do not explain the observed execution are not selected as pseudo-gold training targets. Thus, Wang teaches a rule-based execution-consistency evaluation in which an automatically generated candidate label is accepted when the corresponding execution is successful/consistent and is treated as unreliable or noisy when the corresponding execution is unsuccessful/inconsistent with the observed trajectory. Accordingly, Wang teaches setting a noisy pseudo-label based on determining that the execution associated with the automated intervention action is unsuccessful.)
Claim 8: Regarding claim 8, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 1 as set forth above. Wang further teaches the additional limitation:
The computer-implemented method of claim 1, wherein the evaluating of the recorded states includes determining a pseudo-reward for the automated intervention action, based on the recorded states and the goal state (Wang, page 5, section 4, “The model described above produces a candidate LTL formula ˆz, along with a reconstruction of the input, ˆx. Each candidate formula is used to compute a reward that incorporates the likelihood, as computed by the planner, of the observed trajectories given the hypothesized LTL formula. This plays two roles: first it ensures that the observed trajectories are actually feasible given the hypothesized LTL formula; otherwise they will have zero likelihood. Secondly, it provides a score for how rational the planner judges the behavior to be. Not all feasible paths are equally rational, and so by extension not equally likely. For example, a complex observed behavior is unlikely to be the consequence of a simple parse: it is more likely that the parser is producing an overly broad interpretation rather than the observed trajectory going out of its way to do something unnecessary. The reward is then
PNG
media_image1.png
52
459
media_image1.png
Greyscale
”, teaches that an execution demonstration contains trajectory-environment pairs and that the planner assigns a reward to a hypothesized logical formula according to whether the recorded execution is feasible and rational under that formula. The formula represents the intended commanded behavior, i.e., the goal condition, while the execution trajectory represents the recorded environment states/actions. Because the reward is an auxiliary training signal used to score candidate semantic labels rather than the environment’s ordinary task reward, it corresponds to a pseudo-reward based on the recorded trajectory and the intended goal.)
Claim 9: Regarding claim 9, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 8 as set forth above. Wang further teaches the additional limitation:
The computer-implemented method of claim 8, wherein the training of the semantic parser neural network model includes reinforcement learning using the pseudo-reward (Wang, page 6, section 4.1, “In the reinforcement learning setting, our objective is to maximize the expected reward, marginalizing over the space of possible formulas: JRL = x zR(z)pθ (ˆz | x). We use the REINFORCE algorithm [35] to learn the policy parameters with Monte-Carlo sampling. For better exploration, we use-dithering when sampling as described in 3.1. To incorporate the generator, we optimize a linear combination of this reward and the reconstruction loss, JGEN = − x logp(x | x;θGEN), so that J =JRL+αJGEN. We adjust α at training time to balance the two components. In particular, it is important to start with a small α initially, since JRL is small when θ is untrained and few candidate formulas have non-zero reward.”, teaches maximizing the expected execution-consistency reward and using the “REINFORCE algorithm” to learn the parser policy parameters.)
Claim 10: Regarding claim 10, Yuan, Aineto, Wang, and Hayton teach the limitations of claim 8 as set forth above. Yuan further teaches the additional limitation:
The computer-implemented method of claim 8, wherein the pseudo-reward for the automated intervention action is determined further based on a goal within the text-based environment (Yuan, page 2, section 3.2, “To promote exploration we use an intrinsic reward by counting state visits (Kolter & Ng, 2009; Tang et al., 2017; Martin et al., 2017; Ostrovski et al., 2017). We investigate two approaches to counting rewards.”,
Page 3, section 5.1, “The goal is to find and collect a “coin” placed in one of the rooms. The player’s initial position is at one end of the chain and the coin is at the other.”, Yuan teaches an “intrinsic reward” based on the agent’s observed states and separately teaches that the game “goal is to find and collect a ‘coin.’” The exploration reward is used to guide the text-game agent through environment states toward successful completion of that environment-defined objective. Yuan therefore teaches determining an auxiliary, state-based reward in view of a goal within the text-based environment.)
Claims 14-17 recite limitations substantially similar to claims 3-6, as such a similar analysis applies.
Claims 18-20 recite limitations substantially similar to claims 8-10, as such a similar analysis applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Edwards, A., Sahni, H., Liu, R., Hung, J., Jain, A., Wang, R., ... & Yosinski, J. (2020, November). Estimating q (s, s’) with deep deterministic dynamics gradients. In International Conference on Machine Learning (pp. 2825-2835). PMLR.
Gui, T., Liu, P., Zhang, Q., Zhu, L., Peng, M., Zhou, Y., & Huang, X. (2019, July). Mention recommendation in twitter with cooperative multi-agent reinforcement learning. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 535-544).
Cannon, J. (2011). Robot motion planning using real-time heuristic search. University of New Hampshire.
Kim, Y. C., & Yoon, W. C. (2010, February). Handling Manually Programmed Task Procedures in Human–Service Robot Interactions. In Human-Robot Interaction. IntechOpen.
Ryan, C. (1997). Evaluating the effectiveness of electronic special interest groups (Doctoral dissertation, Dublin City University).
Thrun, S. B. (1992). E cient exploration in reinforcement learning. Technical Report CMU-CS-92-102, School of Computer Science, Carnegie Mellon University.
Ye, D., Zhang, M., & Sutanto, D. (2012). Self-organization in an agent network: A mechanism and a potential application. Decision Support Systems, 53(3), 406-417.
Hausknecht, M., Ammanabrolu, P., Côté, M. A., & Yuan, X. (2020, April). Interactive fiction games: A colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 05, pp. 7903-7910).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2124
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights