DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
Para. [0052]: element 306 as recited in Fig. 3A does not exist
Para. [0053]: element 306 as recited in Fig. 3A does not exist
Para. [0054]: element 306 as recited in Fig. 3A does not exist
Para. [0055]: element 306 as recited in Fig. 3A does not exist
Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 partly recites the following limitations:
...the operations comprising...comprising a plurality of steps, the sequential data comprising a tuple for each step of the trajectory; generating verbalized sequential data from the sequential data, the verbalized sequential data for each step of the trajectory comprising one or more natural language sentences generated from the tuple for the step....
These limitations, as drafted, are a manufacture under Step 1 that under its broadest
reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A,
Prong Two because the claim recites the following additional elements:
One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations;
accessing, from the one or more computer storage media, sequential data for a trajectory;
and training a generative model on the verbalized sequential data to provide a trained generative model that generates a recommended action given a prompt specifying a current state
The additional claim elements of One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations are recited at a high-level of generality using generic computer components (i.e., using a generic device and generic memory to perform generic computer functions) such that it does not amount to a particular machine. The additional elements of accessing, from the one or more computer storage media, sequential data for a trajectory amount to mere insignificant extra-solution activity in which the limitations amount to general data gathering, manipulation and/or outputting of data (i.e., accessing from the storage media sequential data).
The additional claim elements of and training a generative model on the verbalized sequential data to provide a trained generative model that generates a recommended action given a prompt specifying a current state recites only the idea of a solution or outcome and fails to recite the details of how the solution is accomplished since no description is given as to the type of machine learning model and/or configuration used and the type of training and/or finetuning applied to the model on the data to generate a recommended action.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B because as discussed above, One or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations are recited at a high-level of generality using generic computer components (i.e., using a generic device and generic memory to perform generic computer functions) such that it does not amount to a particular machine and the additional claim elements of and training a generative model on the verbalized sequential data to provide a trained generative model that generates a recommended action given a prompt specifying a current state only recites the idea of solution and fails to recite the details of how the solution is accomplished and amounts to no more than mere recitations of “apply it.”
Furthermore, the additional elements of accessing, from the one or more computer storage media, sequential data for a trajectory are well-understood, routine, conventional activity that court decisions, such as Symantec and buySAFE cited in MPEP 2106.05(d)(II) have indicated that the mere receiving and/or sending of data over a network using a generic computer are well- understood, routine, and conventional functions when claimed in a merely generic manner (as it is here).
Accordingly, claim 1 is not patent eligible.
Claim 2 partly recites the following limitations:
The one or more computer storage media of claim 1, wherein the sequential data comprises tabular data in which each row of the tabular data comprises a step from the plurality of steps for the trajectory.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 2 is not patent eligible.
Claim 3 partly recites the following limitations:
The one or more computer storage media of claim 1, wherein generating the verbalized sequential data for a first step of the plurality of steps comprises: employing a template to map data from a first tuple for the first step into one or more natural language sentences for the first step.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 3 is not patent eligible.
Claim 4 partly recites the following limitations:
The one or more computer storage media of claim 3, wherein generating the verbalized sequential data for the first step of the plurality of steps further comprises: determining a cumulative reward for the trajectory at the first step; and appending the cumulative reward to the first tuple.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 4 is not patent eligible.
Claim 5 partly recites the following limitations:
The one or more computer storage media of claim 3, wherein generating the verbalized sequential data for the first step of the plurality of steps further comprises: converting a first continuous value of the first tuple to a discrete value.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 5 is not patent eligible.
Claim 6 partly recites the following limitations:
The one or more computer storage media of claim 1, wherein a first tuple for a first step of the plurality of steps comprises state data and action data, and wherein generating the verbalized sequential data for the first step of the plurality of steps comprises:generating a state verbalization based on the state data; and generative an action verbalization based on the action data.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 6 is not patent eligible.
Claim 7 partly recites the following limitations:
The one or more computer storage media of claim 6, wherein the state verbalization comprises a first introductory natural language sentence, a first data natural language sentence based on the state data, and a first concluding natural language sentence; and wherein the action verbalization comprises a second introductory natural language sentence, a second data natural language sentence based on the action data, and a second concluding natural language sentence.
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 7 is not patent eligible.
Claim 8 partly recites the following limitations:
The one or more computer storage media of claim 6, wherein training the generative model on the verbalized sequential data comprises....
These limitations, as drafted, are a manufacture under Step 1 and due to claim 8 incorporating the claim limitations of claim 6, which has under its broadest reasonable interpretation been found to be able to be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A,
Prong Two because the claim recites the following additional elements:
providing, to the generative model, an input comprising the state verbalization; generating, by the generative model, an output based on the state verbalization; and updating the generative model based on the output and the action verbalization.
The additional claim elements of providing, to the generative model, an input comprising the state verbalization; generating, by the generative model, an output based on the state verbalization; and updating the generative model based on the output and the action verbalization recites only the idea of a solution or outcome and fails to recite the details of how the solution is accomplished since no description is given as to the type of machine learning model and/or configuration used and the type of training and/or finetuning applied to the model on the data to generate an output.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B because as discussed above, providing, to the generative model, an input comprising the state verbalization; generating, by the generative model, an output based on the state verbalization; and updating the generative model based on the output and the action verbalization only recites the idea of solution and fails to recite the details of how the solution is accomplished and amounts to no more than mere recitations of “apply it.”
Accordingly, claim 8 is not patent eligible.
Claim 9 partly recites the following limitations:
The one or more computer storage media of claim 8, wherein the verbalized sequential data for the first step of the plurality of steps further comprises a goal verbalization and a reward verbalization; wherein the input further comprises the goal verbalization....
These limitations, as drafted, are a manufacture under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A,
Prong Two because the claim recites the following additional elements:
and wherein the generative model is updated based on the output, the action verbalization, and the reward verbalization
The additional claim elements of and wherein the generative model is updated based on the output, the action verbalization, and the reward verbalization recites only the idea of a solution or outcome and fails to recite the details of how the solution is accomplished since no description is given as to the type of machine learning model and/or configuration used and the type of training and/or finetuning applied to the model on the data to generate an output.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B because as discussed above, and wherein the generative model is updated based on the output, the action verbalization, and the reward verbalization only recites the idea of solution and fails to recite the details of how the solution is accomplished and amounts to no more than mere recitations of “apply it.”
Accordingly, claim 9 is not patent eligible.
Claim 10 partly recites the following limitations:
A computer-implemented method comprising:generating, by a verbalization component, verbalized sequential data based on sequential data for one or more trajectories....
These limitations, as drafted, are a process under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A,
Prong Two because the claim recites the following additional elements:
training, by a model training component, a generative model using the verbalized sequential data to provide a trained generative model;
generating, by the trained generative model using an input prompt, a recommended action;
and providing, by a user interface component, a recommendation for presentation based on the recommended action.
The additional claim elements of training, by a model training component, a generative model using the verbalized sequential data to provide a trained generative model; generating, by the trained generative model using an input prompt, a recommended action recites only the idea of a solution or outcome and fails to recite the details of how the solution is accomplished since no description is given as to the type of machine learning model and/or configuration used and the training and/or finetuning steps applied to the model on the data to generate a recommended action.
The additional elements of and providing, by a user interface component, a recommendation for presentation based on the recommended action amount to mere insignificant extra-solution activity in which the limitations amount to general data gathering, manipulation and/or outputting of data (i.e., providing to the user interface a recommendation for presentation).
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B because as discussed above, training, by a model training component, a generative model using the verbalized sequential data to provide a trained generative model; generating, by the trained generative model using an input prompt, a recommended action only recites the idea of solution and fails to recite the details of how the solution is accomplished and amounts to no more than mere recitations of “apply it.”
Furthermore, the additional elements of and providing, by a user interface component, a recommendation for presentation based on the recommended action are well-understood, routine, conventional activity that court decisions, such as Symantec and buySAFE cited in MPEP 2106.05(d)(II) have indicated that the mere receiving and/or sending of data over a network using a generic computer are well- understood, routine, and conventional functions when claimed in a merely generic manner (as it is here).
Accordingly, claim 10 is not patent eligible.
Claim 11 partly recites the following limitations:
The computer-implemented method of claim 10, wherein generating the verbalized sequential data based on the sequential data comprises employing one or more templates to map one or more portions of the sequential data to one or more natural language sentences.
These limitations, as drafted, are a process under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited
in the claim beyond the judicial exception.
Accordingly, claim 11 is not patent eligible.
Claim 12 partly recites the following limitations:
The computer-implemented method of claim 10, wherein generating the verbalized sequential data based on the sequential data comprises determining goal data for each step of a plurality of steps for a first trajectory from the one or more trajectories.
These limitations, as drafted, are a process under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited in the claim beyond the judicial exception.
Accordingly, claim 12 is not patent eligible.
Referring to dependent claims 13-16, they are rejected on the same basis as
dependent claims 5-8 since they are analogous claims.
Claim 17 partly recites the following limitations:
The computer-implemented method of claim 14, wherein generating the verbalized sequential data for the first step of the first trajectory from the one or more trajectories further comprises:accessing goal data for the first step of the first trajectory; generating a goal verbalization comprising a third set of one or more natural language sentences with the goal data; accessing reward data for the first step of the first trajectory; and generating a reward verbalization comprising a fourth set of one or more natural language sentences with the reward data.
These limitations, as drafted, are a process under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited in the claim beyond the judicial exception.
Accordingly, claim 17 is not patent eligible.
Claim 18 partly recites the following limitations:
The computer-implemented method of claim 17, wherein training the generative model using the verbalized sequential data comprises....
These limitations, as drafted, are a process under Step 1 and due to claim 18 incorporating the claim limitations of claim 17, which has under its broadest reasonable interpretation been found to be able to be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A,
Prong Two because the claim recites the following additional elements:
providing, to the generative model, an input comprising the state verbalization and the goal verbalization; generating, by the generative model, an output based on the input; and updating the generative model using the output, the action verbalization, and the reward verbalization
The additional claim elements of providing, to the generative model, an input comprising the state verbalization and the goal verbalization; generating, by the generative model, an output based on the input; and updating the generative model using the output, the action verbalization, and the reward verbalization recites only the idea of a solution or outcome and fails to recite the details of how the solution is accomplished since no description is given as to the type of machine learning model and/or configuration used and the training and/or finetuning steps applied to the model on the data to generate an output.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under Step 2B because as discussed above, providing, to the generative model, an input comprising the state verbalization and the goal verbalization; generating, by the generative model, an output based on the input; and updating the generative model using the output, the action verbalization, and the reward verbalization only recites the idea of solution and fails to recite the details of how the solution is accomplished and amounts to no more than mere recitations of “apply it.”
Accordingly, claim 18 is not patent eligible.
Claim 20 partly recites the following limitations:
The computer system of claim 19, wherein the verbalized sequential data used to train the generative model comprises a state verbalization and an action verbalization for each step of a plurality of steps for a trajectory in the sequential data.
These limitations, as drafted, are a system under Step 1 that under its broadest reasonable interpretation can be performed in the human mind through the use of observations, evaluations, judgements and opinion and falls under the mental process grouping. Thus, the claim recites a mental process under Step 2A, Prong One.
This judicial exception is not integrated into a practical application under Step 2A, Prong
Two because there are no additional elements recited in the claim beyond the judicial exception.
And the claims do not include additional elements that are sufficient to amount to significantly
more than the judicial exception under Step 2B because there are no additional elements recited in the claim beyond the judicial exception.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6 and 8-20 are rejected under 35 U.S.C. 103 as being unpatentable over David et al., US 11,875,123 Bl(“David”) in view of Putterman, et al. "Pretraining for language conditioned imitation with transformers." (2021)(“Putterman”).
Regarding claim 1, David teaches one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations(David, col., 13, see also fig. 10A, “[A]s shown in FIG.
l0A, the computing system (1000) may include one or more computer processor(s) (1002), non-persistent storage (1004), persistent storage (1006), a communication interface (1012)[ one or more computer storage media storing computer-useable instructions that, when used by a computing device, cause the computing device to perform operations]”), the operations comprising:
accessing, from the one or more computer storage media, sequential data for a trajectory comprising a plurality of steps, [the sequential data comprising a tuple for each step of the trajectory]( David, col., 13, see also fig. 10A, “[A]s shown in FIG. l0A, the computing system (1000) may include one or more computer processor(s) (1002), non-persistent storage (1004), persistent storage (1006)[accessing, from the one or more computer storage media]....” & David, cols. 7-8, see also fig. 3, “At Step 320, a state of an account is receiving as a second input to the advice planner... the second input to the advice planner is a vector input that
represents the state of the account... [t]he vector input are sets of input features extracted from data based on the of state of the account. For example, where the account includes financial data, the vector input include financial features extracted from the financial data. The financial features may include records of transactions, statistical attributes ( e.g., mean transaction
value, mode of transaction type, etc.), account balances, monthly income, etc[sequential data for a trajectory]...[a]t Step 340, a plan is generated as output from the advice planner. The plan comprises a first set of action logic associated with the domain. Each action logic is a discrete step in an ordered sequence for achieving a desired state of the account[a plurality of steps].”)1
and training a generative model [on the verbalized sequential data] to provide a trained generative model that generates a recommended action given a prompt specifying a current state(David, cols., 7-9, see also figs. 3 and 4, “The plan comprises a first set of action logic associated with the domain. Each action logic is a discrete step in an ordered sequence for achieving a desired state of the account...[t]he plan can be formatted according to a prompt template[given a prompt specifying a current state]... [u]pon receiving the plan as input, the LLM employs its natural language processing capabilities to analyze and understand the content of the plan, and to generate advice[to provide a trained generative model that generates a recommended action]... [a]t Step 430, the advice planner is retrained from both the first set of action logic and the second set of action logic. The advice planner incorporates both the first set of action logic, obtained from previous steps, and the newly generated second set of action logic. The retraining process involves updating the advice planner's algorithms, models, or parameters using the combined knowledge from the first and second sets of action logic[and training a generative model].”).2
While David does teach steps and sequential data, David does not teach: the sequential data comprising a tuple for each step of the trajectory; generating verbalized sequential data from the sequential data, the verbalized sequential data for each step of the trajectory comprising one or more natural language sentences generated from the tuple for the step; on the verbalized sequential data.
However, Putterman teaches:
the sequential data comprising a tuple for each step of the trajectory(Putterman, pg., 2, see also fig. 1, “[W]e are provided with a tuple of (S, A, M, P, R
)
i
[a tuple]which correspond to the states, actions, tasks, transition dynamics, and a reward or score function. A trajectory
τ
=
(
s
1
,
a
1
,
…
s
t
,
a
t
)
is a sequence of states and actions generated by the transition dynamics and the agent’s action at every timestep[for each step of the trajectory]....”);
generating verbalized sequential data from the sequential data, the verbalized sequential data for each step of the trajectory comprising one or more natural language sentences generated from the tuple for the step(Putterman, pgs., 4-5, see also fig. 1, table 1, and algorithm 1, “[W]e model the trajectories as a sequence of tokens... [t]hese text tokens are prepended to the sequence of tokens. The original words of the task are encoded using an embedding table[generating verbalized sequential data from the sequential data]...[t]he model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model... [f]or generating trajectories, we sample from the predicted action logits, and step the environment with the predicted action[the verbalized sequential data for each step of the trajectory comprising one or more natural language sentences generated from the tuple for the step].”);
on the verbalized sequential data(Putterman, pgs., 4-5, see also fig. 1, table 1, and algorithm 1, “[W]e model the trajectories as a sequence of tokens... [t]hese text tokens are prepended to the sequence of tokens. The original words of the task are encoded using an embedding table[on the verbalized sequential data]....”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of David with the teachings of Putterman the motivation to do so would be to use the sequential learning of transformers with respect to text and apply it to sequential tasks comprised of actions, states and goals for better performance(Putterman, pgs., 1-2, “Recent approaches to data-driven RL have proposed leveraging powerful sequence models... newer models, especially those based on the transformer architecture...[w]e introduce a transformer operating on text-state-action sequences – which we call Text Decision Transformer (TDT)... [w]e evaluate a single-stream architecture, Text Decision Transformer, and find it outperforms MLP baselines, highlighting sequence modeling-based approaches.”).
Regarding claim 2, David in view of Putterman teaches the one or more computer storage media of claim 1, wherein the sequential data comprises tabular data(David, col., 7, “The account state can be obtained from external sources, such as financial institutions or platforms. If the account state is obtained from external sources, the state may be retrieved in a structured format, such as JSON or XML[tabular data].” ) in which each row of the tabular data comprises a step from the plurality of steps for the trajectory(David, cols. 7-8, see also fig. 3, “If the account state is obtained from external sources, the state may be retrieved in a structured format, such as JSON or XML[in which each row of the tabular data comprises]...[e]ach action logic is a discrete step in an ordered sequence for achieving a desired state of the account[a step from the plurality of steps for the trajectory]...”).
Regarding claim 3, David in view of Putterman teaches the one or more computer storage media of claim 1, wherein generating the verbalized sequential data for a first step of the plurality of steps comprises:
employing a template to map data from a first tuple for the first step into one or more natural language sentences for the first step(David, cols. 7-8, see also fig. 3, “[T]he output of the advice planner can be a vector representation of the state of the account[from a first tuple]...[e]ach action logic represents a discrete logical step that can be organized into an ordered sequence that guides the user towards achieving a desired state of their account. The advice planner then arranges the action logic steps in the desired order to ensure a logical flow within the generated plan... [t]he plan can be formatted according to a prompt template[employing a template to map data for the first step into one or more natural language sentences for the first step]....”).
Regarding claim 4, David in view of Putterman teaches the one or more computer storage media of claim 3, wherein generating the verbalized sequential data for the first step of the plurality of steps further comprises:
determining a cumulative reward for the trajectory at the first step; and appending the cumulative reward to the first tuple(Putterman, pgs., 2-3, see also fig. 1, “[W]e are provided with a tuple of (S, A, M, P, R
)
i
[and appending the cumulative reward to the first tuple]which correspond to the states, actions, tasks, transition dynamics, and a reward or score function... [t]he score function
R
is prompted with a trajectory
τ
and a text task m and outputs the degree of success that the trajectory had with respect to the task[determining a cumulative reward for the trajectory at the first step]....”).3
Regarding claim 5, David in view of Putterman teaches the one or more computer storage media of claim 3, wherein generating the verbalized sequential data for the first step of the plurality of steps further comprises:
converting a first continuous value of the first tuple to a discrete value(Putterman, pgs., 3-4, “Performance of the model is measured by prompting the model with several different tasks and subsequently running the evaluation procedure on the generated trajectories. The score is normalized to be in the range [0, 1]... [i]f the model receives a score of 1 on a generated trajectory, then the model has succeeded in obeying the task, while a score of 0 indicates a failure[converting a first continuous value of the first tuple to a discrete value].”).4
Regarding claim 6, David in view of Putterman teaches the one or more computer storage media of claim 1, wherein a first tuple for a first step of the plurality of steps comprises state data and action data(Putterman, pgs., 4-5, see also table 1, “The model is prompted with a text tokens and a sequence of states and actions[state data and action data].”), and wherein generating the verbalized sequential data for the first step of the plurality of steps comprises:
generating a state verbalization based on the state data; and generative an action verbalization based on the action data(Putterman, pgs., 4-5, see also table 1, table 3 and algorithm 1, “The model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model... [t]he model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model... [w]ords are then converted to one-hot representation and passed through a learned embedding table. See Algorithm 1 for more details[generating a state verbalization based on the state data; and generative an action verbalization based on the action data].”).5
Regarding claim 8, David in view of Putterman teaches the one or more computer storage media of claim 6, wherein training the generative model on the verbalized sequential data comprises:
providing, to the generative model, an input comprising the state verbalization(Putterman, pgs., 4-5, see also table 1, table 3 and algorithm 1, “The model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model[providing, to the generative model, an input comprising the state verbalization]....”);
generating, by the generative model, an output based on the state verbalization; and updating the generative model based on the output and the action verbalization(Putterman, pgs., 4-5, see also table 1, table 3 and algorithm 1, “The model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model. Action logits for action
a
t
correspond to the output from state
s
t
[generating, by the generative model, an output based on the state verbalization]... [t]he model performs a forward pass over the joint sequence, outputting the logits of the distribution over next actions in parallel. We optimize the cross entropy loss between the predictions and the true actions[and updating the generative model based on the output and the action verbalization]. Pseudocode for the model and training is included in Algorithm 1.”).6
Regarding claim 9, David in view of Putterman teaches the one or more computer storage media of claim 8, wherein the verbalized sequential data for the first step of the plurality of steps further comprises
a goal verbalization and a reward verbalization wherein the input further comprises the goal verbalization; and wherein the generative model is updated based on the output, the action verbalization, and the reward verbalization(Putterman, pg., 14, see also table 1 and table 3, As algorithm 1 details:
PNG
media_image1.png
126
400
media_image1.png
Greyscale
The variable m is assigned [task] which contains the goal and reward[a goal verbalization and a reward verbalization]. And in the while not done: loop, the variable m is inputted into the TextDecisionTransfomer(m, s, a, t).sample() to output the variable action[wherein the input further comprises the goal verbalization]. Lastly in the while not done: loop, a step is taken in the environment with the outputted action i.e., env.step(action) and assigns it to the variables new_s, r, done, _ where r represents the reward verbalization and the outputted action is appended to the previous action and assigned i.e. a = a + [action] and inputted back into the generative model TextDecisionTransfomer(m, s, a, t).sample()[and wherein the generative model is updated based on the output, the action verbalization, and the reward verbalization]).7
Regarding claim 10, David teaches a computer-implemented method comprising:
training, by a model training component, a generative model [using the verbalized sequential data] to provide a trained generative model(David, cols., 7-9, see also figs. 3 and 4, “The plan comprises a first set of action logic associated with the domain. Each action logic is a discrete step in an ordered sequence for achieving a desired state of the account...[t]he plan can be formatted according to a prompt template[given a prompt specifying a current state]... [u]pon receiving the plan as input, the LLM employs its natural language processing capabilities to analyze and understand the content of the plan, and to generate advice[to provide a trained generative model]... [a]t Step 430, the advice planner is retrained from both the first set of action logic and the second set of action logic. The advice planner incorporates both the first set of action logic, obtained from previous steps, and the newly generated second set of action logic. The retraining process involves updating the advice planner's algorithms, models, or parameters using the combined knowledge from the first and second sets of action logic[training, by a model training component, a generative model],:);8
generating, by the trained generative model using an input prompt, a recommended action(David, cols., 7-9, see also figs. 3 and 4, “The plan comprises a first set of action logic associated with the domain. Each action logic is a discrete step in an ordered sequence for achieving a desired state of the account...[t]he plan can be formatted according to a prompt template[using an input prompt]... [u]pon receiving the plan as input, the LLM employs its natural language processing capabilities to analyze and understand the content of the plan, and to generate advice[generating, by the trained generative model a recommended action]....”);
and providing, by a user interface component, a recommendation for presentation based on the recommended action(David, cols., 7-9, see also figs. 3 and 4, “On the user device, the advice is received and presented to the user in the natural language format. The advice may be displayed as text on a user interface, such as a mobile application or a web page. The user can then access and read the advice provided by the advice planner and the LLM[and providing, by a user interface component, a recommendation for presentation based on the recommended action].”).
David does not teach: generating, by a verbalization component, verbalized sequential data based on sequential data for one or more trajectories; using the verbalized sequential data.
However, Putterman teaches:
generating, by a verbalization component, verbalized sequential data based on sequential data for one or more trajectories(Putterman, pgs., 4-5, see also fig. 1, table 1, and algorithm 1, “[W]e model the trajectories as a sequence of tokens... [t]hese text tokens are prepended to the sequence of tokens. The original words of the task are encoded using an embedding table[generating, by a verbalization component, verbalized sequential data]...[t]he model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model... [f]or generating trajectories, we sample from the predicted action logits, and step the environment with the predicted action[based on sequential data for one or more trajectories].”);
using the verbalized sequential data(Putterman, pgs., 4-5, see also fig. 1, table 1, and algorithm 1, “[W]e model the trajectories as a sequence of tokens... [t]hese text tokens are prepended to the sequence of tokens. The original words of the task are encoded using an embedding table...[t]he model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model[using the verbalized sequential data]....”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of David with the teachings of Putterman the motivation to do so would be to use the sequential learning of transformers with respect to text and apply it to sequential tasks comprised of actions, states and goals for better performance(Putterman, pgs., 1-2, “Recent approaches to data-driven RL have proposed leveraging powerful sequence models... newer models, especially those based on the transformer architecture...[w]e introduce a transformer operating on text-state-action sequences – which we call Text Decision Transformer (TDT)... [w]e evaluate a single-stream architecture, Text Decision Transformer, and find it outperforms MLP baselines, highlighting sequence modeling-based approaches.”).
Regarding claim 11, David in view of Putterman teaches the computer-implemented method of claim 10, wherein generating the verbalized sequential data based on the sequential data comprises employing one or more templates to map one or more portions of the sequential data to one or more natural language sentences(David, cols. 7-8, see also fig. 3, “Each action logic represents a discrete logical step that can be organized into an ordered sequence that guides the user towards achieving a desired state of their account. The advice planner then arranges the action logic steps in the desired order to ensure a logical flow within the generated plan... [t]he plan can be formatted according to a prompt template[employing one or more templates to map one or more portions of the sequential data to one or more natural language sentences]....”).9
Regarding claim 12, David in view of Putterman teaches the computer-implemented method of claim 10, wherein generating the verbalized sequential data based on the sequential data comprises determining goal data for each step of a plurality of steps for a first trajectory from the one or more trajectories(Putterman, pg., 14, see also table 1 and table 3, As algorithm 1 details:
PNG
media_image1.png
126
400
media_image1.png
Greyscale
The variable m is assigned [task] which contains goal data which is inputted into the model i.e., TextDecisionTransfomer(m, s, a, t).sample() to output an action that is taken in the environment i.e., env.step(action) to generate trajectories i.e., s, a , t = s+[new_s], a+[action], t+[t[-1]+1] while the loop is not done[determining goal data for each step of a plurality of steps for a first trajectory from the one or more trajectories] ).10
Referring to dependent claims 13-16, they are rejected on the same basis as
dependent claims 5-8 since they are analogous claims.
Regarding claim 17, David in view of Putterman teaches the computer-implemented method of claim 14, wherein generating the verbalized sequential data for the first step of the first trajectory from the one or more trajectories further comprises:accessing goal data for the first step of the first trajectory; generating a goal verbalization comprising a third set of one or more natural language sentences with the goal data; accessing reward data for the first step of the first trajectory; and generating a reward verbalization comprising a fourth set of one or more natural language sentences with the reward data(Putterman, pg., 14, see also table 1 and table 3, As algorithm 1 details:
PNG
media_image1.png
126
400
media_image1.png
Greyscale
The variable m is assigned [task] which contains the goal and reward[accessing goal data for the first step of the first trajectory; accessing reward data for the first step of the first trajectory;]. This data is as contained in the variable m is represented by the following natural language sentences as represented in table 1 as detailed below:
PNG
media_image2.png
120
690
media_image2.png
Greyscale
For the Easy split the [task] is represented by the following natural language sentences: stay on the left side, die on the first level, get as close to 500 score as you can. The goal is represented by the natural language sentence stay on the left side, die on the first level and the reward is represented by the natural language sentence get as close to 500 score as you can[generating a goal verbalization comprising a third set of one or more natural language sentences with the goal data; and generating a reward verbalization comprising a fourth set of one or more natural language sentences with the reward data] ).
Regarding claim 18, David in view of Putterman teaches the computer-implemented method of claim 17, wherein training the generative model using the verbalized sequential data comprises: providing, to the generative model, an input comprising the state verbalization and the goal verbalization; generating, by the generative model, an output based on the input(Putterman, pg., 14, see also table 1 and table 3, As algorithm 1 details:
PNG
media_image3.png
260
461
media_image3.png
Greyscale
In the function TextDecisionTransformer the embeddings are computed for the goals i.e., m_embedding=embed_m(m) and state i.e., s_embedding=embed_s(s) + embed_t(t) and concatenated to form the input embedding i.e., input_embeds=concatenate(m_embedding, input_embeds) and inputted into the transformer to predict an action i.e., pred_a(a_hidden)[ providing, to the generative model, an input comprising the state verbalization and the goal verbalization; generating, by the generative model, an output based on the input]);
and updating the generative model using the output, the action verbalization, and the reward verbalization(Putterman, pg., 14, see also table 1 and table 3, As algorithm 1 details:
PNG
media_image1.png
126
400
media_image1.png
Greyscale
While the loop is not done, a step is taken in the environment with the outputted action i.e., env.step(action) from the generative model i.e., TextDecisionTransformer(m, s, a, t).sample() and assigns the output it to the variables new_s, r, done, _ where r represents the reward verbalization and the outputted action is appended to the previous action i.e. a = a + [action] and inputted back into the generative model i.e., TextDecisionTransfomer(m, s, a, t).sample()[and updating the generative model using the output, the action verbalization, and the reward verbalization]).11
Regarding claim 19, David teaches a computer system comprising:
one or more processors; and one or more computer storage media storing computer-useable instructions that, when used by the one or more processors, causes the one or more processors to perform operations(David, col., 13, see also fig. 10A, “[A]s shown in FIG. l0A, the computing system (1000) may include one or more computer processor(s) (1002), non-persistent storage (1004), persistent storage (1006), a communication interface (1012).”) comprising:
receiving, by a user interface component, an input prompt(David, col. 6, , see also fig. 3, “At Step 310, an intent generated by a large language model is received as a first input to the advice planner. The Intent is generated by the large language model from a text received from a user device. The text data, such as in documents, emails, social media posts, and web pages, can be provided to the LLM through an interface, either directly through a user interface of the user device[receiving, by a user interface component, an input prompt]....”);
providing, by a recommendation component, the input prompt to a generative model [trained on verbalized sequential data comprising one or more natural language sentences generated from values in sequential data](David, cols., 7-8, see also fig. 3, “The plan generated by the advice planner is forwarded to the LLM as input. This can be achieved through suitable
protocols or APIs that facilitate the transmission of data between the advice planner and the LLM. The plan can be formatted according to a prompt template[providing, by a recommendation component, the input prompt to a generative model]....”);12
generating, by the generative model using the input prompt, a recommended action(David, cols., 7-8, see also fig. 3, “Upon receiving the plan as input, the LLM employs its natural language processing capabilities to analyze and understand the content of the plan, and to generate advice. The advice is generated in a natural language format[generating, by the generative model using the input prompt, a recommended action].”);
and providing, by the user interface component, a recommendation for presentation based on the recommended action(David, cols., 7-8, see also fig. 3, “At Step 360, the advice is forwarded to the user device. The advice can be communicated directly to the user, device, or sent to the advice planner where it is forwarded to the user device. The advice may be transmitted back to the user device using appropriate communication protocols or APIs...[o]n the user device, the advice is received and presented to the user in the natural language format[and providing, by the user interface component, a recommendation for presentation based on the recommended action].”).
David does not teach: trained on verbalized sequential data comprising one or more natural language sentences generated from values in sequential data.
However, Putterman teaches:
trained on verbalized sequential data comprising one or more natural language sentences generated from values in sequential data(Putterman, pgs., 4-5, see also fig.1 and algorithm 1, “Thus, a typical sequence of tokens passed into the model will resemble (see Figure 1):
τ
=
(
m
1
,
m
2
,
…
,
m
n
s
1
,
a
1
,
s
2
,
a
2
,
…
,
s
t
,
a
t
)
The model is prompted with a text tokens and a sequence of states and actions. The inputs are tokenized and passed through the model[trained on verbalized sequential data comprising one or more natural language sentences generated from values in sequential data].”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of David with the teachings of Putterman the motivation to do so would be to use the sequential learning of transformers with respect to text and apply it to sequential tasks comprised of actions, states and goals for better performance(Putterman, pgs., 1-2, “Recent approaches to data-driven RL have proposed leveraging powerful sequence models... newer models, especially those based on the transformer architecture...[w]e introduce a transformer operating on text-state-action sequences – which we call Text Decision Transformer (TDT)... [w]e evaluate a single-stream architecture, Text Decision Transformer, and find it outperforms MLP baselines, highlighting sequence modeling-based approaches.”).
Regarding claim 20, David in view of Putterman teaches the computer system of claim 19, wherein the verbalized sequential data used to train the generative model comprises a state verbalization and an action verbalization for each step of a plurality of steps for a trajectory in the sequential data(Putterman, pgs., 4-5, see also fig.1 and algorithm 1, “Thus, a typical sequence of tokens passed into the model will resemble (see Figure 1):
τ
=
(
m
1
,
m
2
,
…
,
m
n
s
1
,
a
1
,
s
2
,
a
2
,
…
,
s
t
,
a
t
)
The model is prompted with a text tokens and a sequence of states and actions[a state verbalization and an action verbalization for each step of a plurality of steps for a trajectory in the sequential data]. The inputs are tokenized and passed through the model.”).13
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over David et al., US 11,875,123 Bl(“David”) in view of Putterman, et al. "Pretraining for language conditioned imitation with transformers." (2021)(“Putterman”) and further in view of Chalvatzaki, et al., Learning to reason over scene graphs: a case study of finetuning gpt-2 into a robot language model for grounded task planning. Frontiers in Robotics and AI. 2023 Aug 15(“Chalvatzaki”)
Regarding claim 7, David in view of Putterman teaches the one or more computer storage media of claim 6, but does not teach: wherein the state verbalization comprises a first introductory natural language sentence, a first data natural language sentence based on the state data, and a first concluding natural language sentence; and wherein the action verbalization comprises a second introductory natural language sentence, a second data natural language sentence based on the action data, and a second concluding natural language sentence.
However, Chalvatzaki teaches:
wherein the state verbalization comprises a first introductory natural language sentence, a first data natural language sentence based on the state data, and a first concluding natural language sentence; and wherein the action verbalization comprises a second introductory natural language sentence, a second data natural language sentence based on the action data, and a second concluding natural language sentence(Chalvatzaki, pgs., 4-5, “In RobLM, the format definition for a NL task description must comply with the following syntactic rule (spaces added for readability):
PNG
media_image4.png
141
431
media_image4.png
Greyscale
”
As detailed above the state starts with a first introductory natural language sentence i.e., <SEP>, then the state data as represented by Context and lastly the natural language sentence of <BOS> [wherein the state verbalization comprises a first introductory natural language sentence, a first data natural language sentence based on the state data, and a first concluding natural language sentence]
As detailed above the action starts with a secondary introductory natural language sentence i.e., <BOS>, then the action data as represented by Plan and lastly the natural language sentence of <EOS>[and wherein the action verbalization comprises a second introductory natural language sentence, a second data natural language sentence based on the action data, and a second concluding natural language sentence]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of David in view of Putterman with the teachings of Chalvatzaki the motivation to do so would be to ground a language model to a given task to focus on a particular task’s domain for it to reason and execute sequential based actions in a logical manner(Chalvatzaki, pg., 2, “Our contribution is twofold: (i) we propose a novel method for linearizing the relations in a scene-graph structure representing the domain (world) to provide it as grounding context when finetuning a pretrained language model (e.g., GPT-2) for learning to draw associations between possible actions (goto, pick, etc.) and objects in the scene (e.g., kitchen, apple, etc.)... [t]he proper structure of the input context is necessary for enabling the model to reason about the combinatorics of actions with affordable objects and their logical sequence (e.g., to cook something, one must first go to the kitchen).”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." Advances in neural information processing systems 34 (2021)(details a framework that abstract reinforcement learning as a sequence modeling problem and uses a causal masked transformer to output optimal actions)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADAM C STANDKE whose telephone number is (571)270-1806. The examiner can normally be reached Gen. M-F 9-9PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Adam C Standke/
Primary Examiner
Art Unit 2129
1Examiner Notes: The claim limitations that are not in bold and contained within square brackets (i.e., [ ]) are
claim limitations that are not taught by the prior art of David.
2Examiner Notes: The claim limitations that are not in bold and contained within square brackets (i.e., [ ]) are
claim limitations that are not taught by the prior art of David.
3 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 1.
4 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 1.
5 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 1.
6 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 1.
7 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 1.
8 Examiner Notes: The claim limitations that are not in bold and contained within square brackets (i.e., [ ]) are
claim limitations that are not taught by the prior art of David.
9 According to the broadest reasonable interpretation (BRI), the use of alternative language amounts to the claim requiring one or more elements but not all.
10 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 10.
11 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 10.
12 Examiner Notes: The claim limitations that are not in bold and contained within square brackets (i.e., [ ]) are
claim limitations that are not taught by the prior art of David.
13 It would have been obvious to one of ordinary skill in the art before the effective filing date
of the claimed invention to modify the teachings of David with the above teachings of Putterman for the same rationale stated at Claim 19.
Prosecution Insights