DETAILED ACTION
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. Applicant’s submission filed 20 April 2026 [hereinafter Response], where:
Claims 11, 12, 14, 15, 17-19, 23-25, 27, and 28 have been amended.
Claims 1-10, 13 and 26 have been cancelled.
New claims 31 and 32 are presented for examination.
Claims 11, 12, 14-25, and 27-32 are pending.
Claims 11, 12, 14-25, and 27-32 are rejected.
Foreign priority is claimed to GR20200100037, filed 28 January 2020. A certified copy of this paper has been filed on 22 July 2022. Accordingly, receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
3 An information disclosure statement was submitted on 20 April 2026. The submission complies with the provisions of 37 CFR 1.97. Accordingly, the Examiner considered the information disclosure statement.
Claim Rejections - 35 U.S.C. § 112
4. The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
5. The rejection to claims 12, 17, 18, and 19 under 35 U.S.C. § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention is WITHDRAWN.
Claim Rejections - 35 U.S.C. § 101
6. 35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
7. Claims 11, 12, 14-25, and 27-32 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 11 recites a method, which is a process, and thus one of the statutory categories of patentable subject matter. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “[(b)]1 processing a representation input comprising the current observation using a representation model to generate an initial hidden state corresponding to the current environment state,” “[(c)] performing a plurality of planning iterations based on the initial hidden state to generate plan data that indicates a respective value to performing the task of the agent performing each of the set of actions in the environment and starting from the current environment state,” and “[(d)] selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data.” The activities of “[(b)] processing . . . to generate,” “[(c)] performing . . . planning iterations,” and “[(d)] selecting . . . an action” can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)).
The claim recites more details or specifics to the abstract idea of “[(c)] performing a plurality of planning iterations,” where “performing each planning iteration comprises: [(c.1)] selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs generated by: [(c.1.1)] (i) a dynamics model . . . ; and [(c.1.2)] (ii) a prediction model . . . ,” and accordingly, is merely more specific to the abstract idea.
Moreover, the claim recites “[(c.1)] selecting . . . based on outputs generated by: [(c.1.1)] a dynamics model . . . ; and [(c.1.2)] a prediction model . . . ,” where the plain meaning of such “outputs” refers to predictions or decisions made by a machine learning model based on input data. The broadest reasonable interpretation of such “outputs” cover predictive activities by a human user, (see Specification at p. 18, ll. 22-24 (“agent that is being controlled by a human user”)), and can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Accordingly, claim 11 is directed to an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not serve to integrate the abstract idea into a practical application.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the pre-processing insignificant extra-solution activity of data gathering, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Thus, claim 10 is directed to an abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. The claim includes the elements of “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not amount to significantly more than the abstract idea.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the well-understood, routine, and conventional activity of receiving data over a network, (MPEP § 2106.05(d) sub II.i), that does not amount to significantly more than the abstract idea.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Therefore, claim 11 is subject-matter ineligible.
Claim 23 recites a system, which is a product, and thus one of the statutory categories of patentable subject matter. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “[(b)]2 processing a representation input comprising the current observation using a representation model to generate an initial hidden state corresponding to the current environment state,” “[(c)] performing a plurality of planning iterations based on the initial hidden state to generate plan data that indicates a respective value to performing the task of the agent performing each of the set of actions in the environment and starting from the current environment state,” and “[(d)] selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data.” The activities of “[(b)] processing . . . to generate,” “[(c)] performing . . . planning iterations,” and “[(d)] selecting . . . an action” can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)).
The claim recites more details or specifics to the abstract idea of “[(c)] performing a plurality of planning iterations,” where “performing each planning iteration comprises: [(c.1)] selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs generated by: [(c.1.1)] (i) a dynamics model . . . ; and [(c.1.2)] (ii) a prediction model . . . ,” and accordingly, is merely more specific to the abstract idea.
Moreover, the claim recites “[(c.1)] selecting . . . based on outputs generated by: [(c.1.1)] a dynamics model . . . ; and [(c.1.2)] a prediction model . . . ,” where the plain meaning of the claimed “outputs” refers to predictions or decisions made by a machine learning model based on input data. The broadest reasonable interpretation of the claimed “outputs” cover predictive activities by a human user, (see, e.g., Specification at p. 18, ll. 22-24 (“agent that is being controlled by a human user”)), and can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Accordingly, claim 23 is directed to an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include “[a] system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations,” in which the use of generic computer components (one or more computers, one or more storage devices) to execute instructions to implement the abstract idea, (MPEP § 2106.05(f)), that does not serve to integrate the abstract idea into a practical application. Also, the claim recites additional elements of “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); see also Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not serve to integrate the abstract idea into a practical application.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the pre-processing insignificant extra-solution activity of data gathering, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Thus, claim 23 is directed to an abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. The claim includes “[a] system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations,” in which the use of generic computer components (one or more computers, one or more storage devices) to execute instructions to implement the abstract idea, (MPEP § 2106.05(f)), that does not amount to significantly more than the abstract idea. Also, the claim recites additional elements of “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); see also Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not amount to significantly more than the abstract idea.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the well-understood, routine, and conventional activity of receiving data over a network, (MPEP § 2106.05(d) sub II.i), that does not amount to significantly more than the abstract idea.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Therefore, claim 23 is subject-matter ineligible.
Claim 24 recites a “one or more computer storage media,” which is a product, and thus one of the statutory categories of patentable subject matter. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “[(b)]3 processing a representation input comprising the current observation using a representation model to generate an initial hidden state corresponding to the current environment state,” “[(c)] performing a plurality of planning iterations to generate plan data that indicates a respective value to performing the task of the agent performing each of the set of actions in the environment and starting from the current environment state,” and “[(d)] selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data.” The activities of “[(b)] processing . . . to generate,” “[(c)] performing . . . planning iterations,” and “[(d)] selecting . . . an action” can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)).
The claim recites more details or specifics to the abstract idea of “[(c)] performing a plurality of planning iterations,” where “performing each planning iteration comprises: [(c.1)] selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs generated by: [(c.1.1)] (i) a dynamics model . . . ; and [(b.1.2)] (ii) a prediction model . . . ,” and accordingly, is merely more specific to the abstract idea. Moreover, the claim recites “[(b.1)] selecting . . . based on outputs generated by: [(c.1.1)] a dynamics model . . . ; and [(c.1.2)] a prediction model . . . ,” where the plain meaning of the claimed “outputs” refers to predictions or decisions made by a machine learning model based on input data. The broadest reasonable interpretation of the claimed “outputs” cover predictive activities by a human user, (see, e.g., Specification at p. 18, ll. 22-24 (“agent that is being controlled by a human user”)), and can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Accordingly, claim 24 is directed to an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include “[o]ne or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations,” in which the use of generic computer components (one or more computers, [o]ne or more computer storage media) to execute instructions to implement the abstract idea, (MPEP § 2106.05(f)), that does not serve to integrate the abstract idea into a practical application. Also, the claim recites additional elements of “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); see also Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not serve to integrate the abstract idea into a practical application.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the pre-processing insignificant extra-solution activity of data gathering, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Thus, claim 24 is directed to an abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. The claim includes ““[o]ne or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations,” in which the use of generic computer components (one or more computers, [o]ne or more computer storage media) to execute instructions to implement the abstract idea, (MPEP § 2106.05(f)), that does not amount to significantly more than the abstract idea. Also, the claim recites additional elements of “a representation model,” “a dynamics model,” and “a prediction model,” which are described at a high level of generality, and accordingly, are generic computer components used to implement the abstract idea. (MPEP § 2106.05(f); see also Specification at p. 11, ll. 31-33 (“The representation, dynamics, and prediction models can each be implemented as a respective neural network with any appropriate neural network architecture that enables it to perform its described function.”)), and do not amount to significantly more than the abstract idea.
Also, the claim recites “[(a)] receiving a current observation characterizing a current environment state of the environment,” which is the well-understood, routine, and conventional activity of receiving data over a network, (MPEP § 2106.05(d) sub II.i), that does not amount to significantly more than the abstract idea.
The claim recites more details or specifics to the additional element of ‘[(c.1.1)] (i) a dynamics model . . . to receive input,” where “[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions and [(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state, [(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and [(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and [(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration,” and accordingly, are merely more specific to the additional element. Therefore, claim 24 is subject-matter ineligible.
Claim 12 depends directly or indirectly from claim 11. Claim 25 depends directly or indirectly from claim 23. The claims recite more details or specifics to the abstract idea of “[(c.1.1)] the dynamics model . . . output,” that includes “[(c.1.1.4)] a predicted immediate reward value that represents an immediate reward that would be received if the agent performed the input action when the environment is in the input environment state,” and accordingly, is merely more specific to the abstract idea. The claim also provides more details or specifics to the abstract idea of “[(c.1.1.4)] a predicted immediate reward value,” “[(c.1.1.4.1)] wherein the predicted immediate reward value is a numerical value that represents a progress in completing the task as a result of performing the input action when the environment is in the input environment state,” and accordingly, is merely more specific to the abstract idea. The abstract idea of these claims are not integrated into a practical application, (see MPEP § 2106.05(d)), nor do they amount to significantly more than the abstract idea, (MPEP § 2106.05(d)), because the claims recite no more than the abstract idea. Therefore, claims 12 and 25 are subject-matter ineligible.
Claim 14 depends directly or indirectly from claim 11. Claim 27 depends directly or indirectly from claim 23. The claims recite more details or specifics of abstract idea of [(b)] processing a representation input,” “[(b)] further comprises one or more previous observations characterizing one or more previous states that the environment transitioned into prior to the current state,” and accordingly, are merely more specific to the abstract idea. Therefore, claims 14 and 27 are subject-matter ineligible.
Claim 15 depends directly or indirectly from claim 11. Claim 28 depends directly or indirectly from claim 23. The claims recite more details or specifics to the additional elements of “the [(b)] representation model, [(c.1.1)] the dynamics model, and [(c.1.2)] the prediction model,” in which the models “are jointly trained end-to-end on sampled trajectories from a set of trajectory data,” and therefore, are merely more specific to the additional elements. The additional elements of the claim do not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 15 and 28 are subject-matter ineligible.
Claim 16 depends directly or indirectly from claim 11. Claims 29 depends directly or indirectly from claim 23. The claims recite more details or specifics to the additional elements of “the [(c.1.3)] representation model, [(c.1.1)] the dynamics model, and [(c.1.2)] the prediction model,” in which the models “are jointly trained end-to-end on an objective that measures, for each of a plurality of particular observations,” and accordingly, are merely more specific to the additional elements. With respect to the “objective that measures,” the abstract idea of “measures” include “(i) a policy error between the predicted policy output for the subsequent state generated conditioned on the particular observation and an actual policy that was used to select an action in response to the observation, and (ii) a value error between the value predicted for the subsequent state generated conditioned on the particular observation and an actual return received starting from the subsequent state,” which are merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 16 and 28 are subject-matter ineligible.
Claim 17 depends directly or indirectly from claim 11. The claim further recites more details or specifics to the additional element of “measures,” “wherein the objective also measures, for each of the plurality of particular observations: . . . a reward error between the predicted immediate reward for the subsequent state generated conditioned on the particular observation and an actual immediate reward corresponding to the subsequent state,” and accordingly, are merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claim 17 is subject-matter ineligible.
Claim 18 depends directly or indirectly from claim 11. The claim further recites more details or specifics to the additional elements of “ [(b)] the representation model,” in which the model “is not trained to output hidden states for reconstruction of the current observation or model semantics of the environment through the hidden states,” and accordingly, is merely more specific to the additional element. Therefore, claim 18 is subject-matter ineligible.
Claim 19 depends directly or indirectly from claim 11. The claim recites more details or specifics of the abstract idea of a “measure,” “wherein the actual return starting from each of the one or more subsequent states is a bootstrapped n-step return,” and accordingly, is merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claim 19 is subject-matter ineligible.
Claim 20 depends directly or indirectly from claim 11. The claim recites more details or specifics to the abstract idea of “[(d)] selecting, from the set of actions, an action,” which comprises “[(d.1)] selecting the action using a markov decision process (MDP) planning algorithm,” and accordingly, is merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claim 20 is subject-matter ineligible.
Claim 21 depends directly or indirectly from claim 11. Claim 30 depends directly or indirectly from claim 23. The claims recite more details or specifics of the abstract idea of “ [(c.1)] selecting the sequence of actions,” and “[(d)] selecting the action to be performed” that “are performed using a monte carlo tree search (MCTS) algorithm,” and accordingly, are merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 21 and 30 are subject-matter ineligible.
Claim 22 depends directly or indirectly from claim 11. The claim recites more details or specifics to the abstract idea of “[(d)] selecting, from the set of actions, an action,” comprises “[(d.2)] determining, from the sequences of actions in the plan data, a sequence of actions that has a maximum associated value output,” and “[(d.3)] selecting, as the action to be performed by the agent in response to the current observation, the first action in the determined sequence of actions,” and accordingly, is merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 21 and 30 are subject-matter ineligible.
Claim 31 depends directly or indirectly from claim 11. Claim 32 depends directly or indirectly from claim 23. The claims recite more details or specifics of the abstract idea of “[(c)] performing the plurality of planning iterations,” “wherein performing the plurality of planning iterations based on the initial hidden state to generate the plan data comprises [(c.2)] performing the plurality of planning iterations without using a simulator of the environment and without reconstructing the current observation,” and accordingly, are merely more specific to the abstract idea. Therefore, claims 31 and 32 are subject-matter ineligible.
Claim Rejections – 35 U.S.C. § 103
8. The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
9. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. § 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
10. This application currently names joint inventors. In considering patentability of the claims the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
11. Claims 11, 12, 23, 24, and 25 are rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer] and US Published Application 20170213150 to Arel et al. [hereinafter Arel].
Regarding claims 11, 23, and 24, Xie teaches [a] method for selecting, from a set of actions, actions to be performed by an agent interacting with an environment to cause the agent to perform a task (Xie, Abstract), [a] system comprising one or more computers and one or more storage devices (Xie, abstract, teaches “[m]achine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects [(that is, a “robot” inherently is a system comprising one or more computers and one or more storage devices)]”) of claim 23, and [o]ne or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform (Xie, abstract, teaches “[m]achine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects [(that is, a “robot” inherently includes [o]ne or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform)]”), comprising:
[(a)] receiving a current observation characterizing a current environment state of the environment (Xie, left column of p. 3, “III. Capabilities for Improvisational Tool Use,” first full paragraph, teaches “autonomously collecting data of diverse object interactions, training predictive models of low-level sensory observations (i.e. action-conditioned video prediction [17]) [(that is, “low-level sensory observations” is receiving a current observation characterizing a current environment state of the environment)], and using these models to make plans to achieve goals involving tools”);
[(b)] processing a representation input comprising the current observation using a representation model to generate an initial hidden state corresponding to the current environment state (Xie, left column of p. 3, “III. Capabilities for Improvisational Tool Use,” first full paragraph, teaches “low-level sensory observations such as pixels include all information about the environment that the robot can currently perceive [(that is, comprising the current observation)], and hence are general to a wide range of objects and situations, including non-rigid and deformable objects”; Xie, right column of p. 4, “2) Action Proposal Model Training,” last partial paragraph, teaches “we use an RNN, in particular an LSTM net [(that is, a representation model)], that uses this embedding to initialize the hidden state [(that is, processing a representation input comprising the current observation using a representation model”);
[(c)] performing a plurality of planning iterations to generate plan data (Xie, left column of p. 2, “I. Introduction,” first full paragraph, teaches “[o]ur method uses video prediction to reason about potential robot actions, constructing plans to manipulate novel objects on the fly, in less than a second [(that is, to generate a plan)]”; Xie, left column of p. 6, “B. Test-Time Control, 2) Planning with Demonstration Guidance,” first paragraph, teaches that “[p]lanning with [guided visual foresight (GVF)] at test time is illustrated in Figure 2 (right) and Algorithm 1. The user first specifies the task by clicking on the pixels that shall be moved and the corresponding goal-pixels. The planner searches for actions using the cross entropy method (CEM) [23], a common iterative sampling-based optimization procedure [(that is, “the planner” is performing a plurality of planning iterations to generate plan data)]”); that indicates a respective value to performing the task of the agent performing each of the set of actions in the environment and starting from the current environment state (Xie, left column of p. 4, “IV. Demonstration-Guided Visual Planning,” first & second paragraphs, teaches that, “[f]or example, the user might specify that three pieces of trash need to be moved to a location within a dustpan. Then, the current observation is passed to the action proposal model [(that is, starting from the current environment state)], which returns a sampling distribution that is used to sample a certain number of action sequences. . . . We feed each of the sampled action sequences into the video prediction model to predict their outcome as a video. We then rank these predictions using a cost function [(that is, “cost” indicates a respective value to performing the task)] determined by the human-specified goal, and refine the best samples further. Lastly, the robot recomputes action plans after several control cycles. . . . [D]emonstrations can be effectively used to guide the planning process towards tool-related behaviors, while the predictive model is used to fully construct and refine a sequence of actions for completing the task [(that is, of the agent performing each of the set of actions in the environment and starting from the current environment state)]”), wherein performing each planning iteration comprises:
[(c.1)] selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs (Xie, right column of p. 3, “IV. Demonstration-Guided Visual Planning,” second paragraph, teaches “[w]e use [kinesthetic demonstration] data to train an action proposal model to obtain a distribution over action sequences [(that is, “action sequences” is selecting a sequence of actions to be performed by the agent)] conditioned on the initial image [(that is, starting from the current environment state)] based on actions taken by the demonstrator”) generated by:
[(c.1.1)] (i) a dynamics model (Xie, right column of p. 5, “4) Predictive Model Training,” first paragraph, teaches that “[t]he forward pass of the dynamics model is summarized in the following two equations:
PNG
media_image1.png
77
461
media_image1.png
Greyscale
[(that is, a dynamics model)].” The model is trained with stochastic gradient descent using a `2 image reconstruction loss”) that is configured through training (Xie, right column of p. 4, “2) Action Proposal Model Training,” first paragraph, teaches an “action proposal model is parameterized as an autoregressive recurrent neural network (RNN). It is trained with the following maximum likelihood objective [(that is, a dynamics model that is configured through training)]”) to receive as input
[(c.1.1.1)] a) a hidden state corresponding to an input environment state and [(c.1.1.2)] b) an input action from the set of actions (Xie, left column at p. 11, “A. Video Prediction Model Implementation Details,” first paragraph, teaches that “[a]t every time-step an action at [(that is, an input action from the set of actions)] is passed into the model along with the hidden state ht, [(that is, to receive as input a hidden state corresponding to an input environment state)] producing a new state ht+1 and a flow field
F
^
t
+
1
←
t
which is used to transform the image via bi-linear sampling”) and
[(c.1.1.3)] to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state (Xie, left column at p. 11, “A. Video Prediction Model Implementation Details,” first paragraph, teaches that “[a]t every time-step an action at is passed into the model along with the hidden state ht, producing a new state ht+1 and a flow field
F
^
t
+
1
←
t
[(that is, “a new state ht+1 “ is a dynamics model . . . to generate as output at least a hidden state corresponding to a predicted next environment state that the environment would transition into if the agent performed the input action when the environment is in the input environment state)]”), . . . , and
[(c.1.1.5)] wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model (Xie, right column at p. 4, “2) Action Proposal Model Training,” first paragraph, teaches “[t]he [initial state] encoder ge [(that is, the representation model)] encodes the initial image I1 and state s1 to provide the input to the LSTM at the first timestep [(that is, wherein, at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model)]”) and,
[(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration (Xie, right column of p. 4, “2) Action Proposal Model Training,” first paragraph, teaches “the action encoder ga [(that is, the dynamics model)] encodes the previous action [(that is, a preceding step of the planning iteration)] to provide to the LSTM cell and future timesteps”); and
[(c.1.2)] (ii) a prediction model (Xie, left column at p. 6, “2) Planning with Demonstration Guidance,” first paragraph, teaches “After rolling out the video prediction model
ƒ
γ
[(that is, a prediction model)] using Equation 7
PNG
media_image2.png
39
391
media_image2.png
Greyscale
we obtain M different predicted probability distributions
P
1
:
H
m
, which are ranked using the cost function c”) that is configured to
[(c.1.2.1)] receive as input the hidden state corresponding to the predicted next environment state (Xie, right column of p. 5, “4) Predictive Model Training,” first paragraph, teaches “[t]he model, which is implemented as a recurrent convolutional neural network, ꝭ parameterized by ү, has a hidden state ht and takes in a previous image and an action at each step of the rollout”; Xie, left column of p. 11, “VII. Appendix, A. Video Prediction Model Implementation Details,” first paragraph, teaches “[a]t every time-step an action at is passed into the model along with the hidden state ht, producing a new state ht+1 [(that is, as input the hidden state)] and a flow field
F
^
t
+
1
←
t
[(that is, a prediction model that is configured to receive as input the hidden state corresponding to the predicted next environment state)] which is used to transform the image via bi-linear sampling”) and to
[(c.1.2.2)] generate as output
[(b.1.2.2.1)] a) a predicted policy output that defines a score distribution over the set of actions (Xie, Algorithm 1 Guided Visual Foresight (test time), teaches a guided visual foresight (GVF) model [Examiner annotations in dashed-line text boxes]:
PNG
media_image3.png
461
858
media_image3.png
Greyscale
Xie, left column of p. 6, “2) Planning with Demonstration Guidance,” first paragraph, teaches “After rolling out the video prediction model
ꝭ
ү
using Equation 7 we obtain M different predicted probability distributions
P
^
1
:
H
m
, which are ranked using the cost function c. We then fit a Gaussian distribution to the best k action samples (see line 10). In later CEM iterations, actions are sampled from the fitted Gaussians (line 7) [(that is, a) a predicted policy output that defines a score distribution over the set of actions)]”) and
* * *
and [(d)] selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data (Xie, Fig. 2, teaches an action-conditioned video prediction model,[Examiner annotations in dashed-line text boxes]:
PNG
media_image4.png
477
887
media_image4.png
Greyscale
Xie, Fig. 2 caption, teaches “[o]ur guided visual foresight (GVF) approach, at training time (left) and test time (right). Our method incorporates demonstrations and autonomous data collection to learn a video prediction model and action proposal model that enable the robot to solve both a diverse range of goals that require tool use [(that is, selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data)]. We incorporate the action proposal model both for training data for the video prediction model and for improving the sampling-based planner at test time. The test time procedure is further detailed in Algorithm 1 [shown above]”).
Though Xie teaches a pixel distance cost function that evaluates how far a designated pixel is from the goal pixels, Xie, however, does not explicitly teach –
* * *
[(c.1) selecting a sequence of actions . . . . (c.1.1)] (i) a dynamics model . . . to receive as input . . . . (c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, . . . ; and
* * *
But Schäfer teaches –
* * *
[(c.1) selecting a sequence of actions . . . . (c.1.1)] (i) a dynamics model . . . to receive as input . . . . (c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation (Schäfer ¶ 0010 teaches “the recurrent hidden layer being formed by hidden states with a plurality of hidden state variables in a second state space with a second dimension, with the second dimension being smaller than the first dimension [(that is, the hidden state has a lower dimensionality . . . than the current observation)]”), . . . ; and
* * *
Xie and Schäfer are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states.
Thus, it would have been obvious to a person having ordinary skill of the art as of the effective filing date of the Applicant’s invention to modify Xie pertaining to a model-based RL planner with the state space dimensionality reduction of Schäfer .
The motivation to do so is because “inventively known learning and/or optimization methods are employed in a state space of reduced dimensions determined via a recurrent neural network. These learning and optimization methods can for example be reinforcement learning methods which are sufficiently well known from the state of the art . . . .” (Schäfer ¶ 0014).
Though Xie and Schäfer teach a pixel distance cost function that evaluates how far a designated pixel is from the goal pixels in a reduced dimensionality state space, the combination of Xie and Schäfer, however, does not explicitly teach –
* * *
[(c.1.2) (ii) a prediction model that is configured to
* * *
(c.1.2.2) generate as output]
* * *
[(c.1.2.2.2)] b) a value output that represents a value of the environment being in the predicted next environment state to performing the task; and
* * *
But Arel teaches -
* * *
[(c.1.2) (ii) a prediction model that is configured to
* * *
(c.1.2.2) generate as output]
* * *
[(c.1.2.2.2)] b) a value output that represents a value of the environment being in the predicted next environment state to performing the task (Arel ¶ 0083 “the system instead uses state value supervised learning models that are configured to receive a state representation representing a given state and to generate a state value estimate [(that is, a value output)] that is an estimate of the long-term value of the environment having transitioned into the given state, e.g., of the return received starting from the environment being in the state [(that is, a value output that represents a value of the environment being in the predicted next environment state to performing the task)]. For example, the system can use these state value supervised learning models [(that is, a prediction model)] in conjunction with a transition model [(that is, a dynamics model)] that receives a state and an action as input and predicts a state that is most likely to be the state the environment transitions into as a result of the actor performing the action in response to the given state representation to select the action to be performed by the agent”); and
* * *
Xie, Schäfer, and Arel are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie and Schäfer pertaining to a model-based RL planner in a reduced dimensional state space with the immediate rewards expectation of Arel.
The motivation to do so is because “supervised learning models can be trained in a scalable manner to effectively select actions in response to new state representations without adversely affecting their performance when the environment is in other states.” (Arel ¶ 0007).
Regarding claims 12 and 25, the combination of Xie, Schäfer, and Arel teaches all of the limitations of claims 11 and 23, as described above in detail.
Arel teaches -
[(c.1.1.3)] wherein the dynamics model (see above, where Arel ¶ 0083 teaches a transitions model [(that is, a dynamics model)]) also generates as output a predicted immediate reward value that represents an immediate reward that would be received if the agent performed the input action when the environment is in the input environment state (Arel ¶ 0019 teaches “[g]enerally, the reward is a numeric value that is received from the environment as it transitions into a given state and is a function of the state of the environment. While the agent is interacting with the environment, the reinforcement learning system selects actions to be performed by the agent in order to maximize the expected return. Generally, the expected return is a function of the rewards anticipated to be received over time in response to future actions performed by the agent. That is, the return is a function of future rewards [(that is, “future rewards” are a predicted immediate reward value)] received starting from the immediate reward received in response to the agent performing the selected action. For example, possible definitions of return that the reinforcement learning system attempts to maximize may include a sum of the future rewards, a discounted sum of the future rewards, or an average of the future rewards [(that is, output a predicted immediate reward value that represents an immediate reward that would be received if the agent performed the input action when the environment is in the input environment state)]”),
[(c.1.1.3.1)] wherein the predicted immediate reward value is a numerical value that represents a progress in completing the task as a result of performing the input action when the environment is in the input environment state (Arel ¶ 0019 teaches “[g]enerally, the reward is a numeric value that is received from the environment as it transitions into a given state and is a function of the state of the environment. While the agent is interacting with the environment, the reinforcement learning system selects actions to be performed by the agent in order to maximize the expected return. Generally, the expected return is a function of the rewards anticipated to be received over time in response to future actions performed by the agent [(that is, “rewards anticipated to be received over time” is the predicted immediate reward value is a numerical value that represents a progress in completing the task)]. That is, the return is a function of future rewards received starting from the immediate reward received in response to the agent performing the selected action [(that is, as a result of performing the input action when the environment is in the input environment state)]”).
12. Claims 14-16 and 27-29 are rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel] and Liang et al., “VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control,” arXiv (2018) [hereinafter Liang].
Regarding claims 14 and 27, the combination of Xie, Schäfer, and Arel teaches all of the limitations of claims 11 and 23, respectively, as described above in detail.
Though Xie, Schäfer, and Arel teach a model receiving a state representation in a reduced dimensional space to output an output for a state-action pair, the combination of Xie, Schäfer, and Arel, however, does not explicitly teach –
[(c.1.3)] wherein the representation input further comprises
[(c.1.3.1)] one or more previous observations characterizing one or more previous states that the environment transitioned into prior to the current state.
But Liang teaches -
[(c.1.3)] wherein the representation input further comprises
[(c.1.3.1)] one or more previous observations characterizing one or more previous states that the environment transitioned into prior to the current state (Liang at p. 10, “Involvement of AVF,” second paragraph, teaches “[i]n Critic network of [actor-critic] algorithm, hidden information of each time illustrated in Fig 5 comes from MDN-RNN route layer. And historical information in former n time steps is utilized for current state value estimation. . . . For the state value function estimation, it is required to combine both context information 𝑐𝑡 derived from previously hidden information {ht−1, ht−2, . . , ht−n} and current state information [(that is, one or more previous observations characterizing one or more previous states that the environment transitioned into prior to the current state)] as
PNG
media_image5.png
53
266
media_image5.png
Greyscale
Where zt is the latent representation of state in time step t, ct is the context vector with attention, [. , . ] is the concatenation of vectors and {W, b, Wv, bv} is the set of parameters to learn in attention-based value neural network. Fig 5 [as earlier shown above] reveals the learning process of state value v.”)
Xie, Schäfer, Arel, and Liang are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Liang teaches a model-based reinforcement learning algorithm with attention mechanism embedded.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Schäfer, and Arel pertaining to a model-based RL planner in a reduced dimensional state space implementing an Attention-based Value Function (AVF) model of Liang.
The motivation to do so is because in “[universal complex tasks, the bottlenecks encountered includes] the low efficiency of data utilization in model-free reinforcement algorithms . . . . In contrast, the model-based reinforcement learning algorithms can reveal underlying dynamics in learning environments and seldom suffer the data utilization problem. To address the problem, a model-based reinforcement learning algorithm with attention mechanism embedded is proposed as an extension of World Models [in which an] agent can learn optimal policies through less interactions with actual environment, and final experiments demonstrate the effectiveness of our model in control problem.” (Liang, Abstract).
Regarding claims 15 and 28, the combination of Xie, Schäfer, and Arel teaches all of the limitations of claims 11 and 23, respectively, as described above in detail.
Though Xie, Schäfer, and Arel teach a model receiving a state representation in a reduced dimensional space to output an output for a state-action pair, the combination of Xie, Schäfer, and Arel, however, does not explicitly teach –
wherein [(c.1.3)] the representation model, [(b.1.1)] the dynamics model, and [(b.1.2)] the prediction model are jointly trained end-to-end on sampled trajectories from a set of trajectory data.
But Liang teaches -
wherein [(c.1.3)] the representation model, [(b.1.1)] the dynamics model, and [(b.1.2)] the prediction model are jointly trained end-to-end on sampled trajectories from a set of trajectory data (Liang at p. 12, “4.3.1 Pretraining Details,” first paragraph, teaches “The purpose of VAE [(that is, the dynamics model)], MDN-RNN [(that is, the prediction model)], AVF [(that is, the representation model)] and Controller is to learn representations of states and the dynamic transitions in the environment at the same time, but massive parameters and complexity of network structures make it tough and time-consuming to train VMAV-C. Hence, synchronously pretraining VMAV [(that is, are jointly trained end-to-end)] is the required step in our experiments. To achieve this aim, we collect 2000 episodes with random policy strategy through a series of interactions with actual environment as {episode = {(xt, at, xt+1, rt+1, dt+1)}} in Step 0. These rollouts/screenshots of the environment serve as the training dataset for VAE, and we assume the sampling has approximately encompassed the dynamic information of the environment, especially the state representations and concerning transitions [(that is, jointly trained end-to-end on sampled trajectories from a set of trajectory data)]”).
Xie, Schäfer, Arel, and Liang are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Liang teaches a model-based reinforcement learning algorithm with attention mechanism embedded.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Schäfer, and Arel pertaining to a model-based RL planner in a reduced dimensional state space implementing an Attention-based Value Function (AVF) model of Liang.
The motivation to do so is because in “[universal complex tasks, the bottlenecks encountered includes] the low efficiency of data utilization in model-free reinforcement algorithms . . . . In contrast, the model-based reinforcement learning algorithms can reveal underlying dynamics in learning environments and seldom suffer the data utilization problem. To address the problem, a model-based reinforcement learning algorithm with attention mechanism embedded is proposed as an extension of World Models [in which an] agent can learn optimal policies through less interactions with actual environment, and final experiments demonstrate the effectiveness of our model in control problem.” (Liang, Abstract).
Regarding claims 16 and 29, the combination of Xie, Schäfer, Arel, and Liang teaches all of the limitations of claims 15 and 28, as described above in detail.
Liang teaches
wherein [(c.1.3)] the representation model, [(b.1.1)] the dynamics model, and [(b.1.2)] the prediction model are jointly trained end-to-end (Liang at p. 12, “4.3.1 Pretraining Details,” first paragraph, teaches “[t]he purpose of VAE [(that is, the dynamics model)], MDN-RNN [(that is, the prediction model)], AVF [(that is, the representation model)] and Controller is to learn representations of states and the dynamic transitions in the environment at the same time, but massive parameters and complexity of network structures make it tough and time-consuming to train VMAV-C. Hence, synchronously pretraining VMAV [(that is, are jointly trained end-to-end)] is the required step in our experiments”) on an objective that measures (Liang at p. 4, “3. Background,” first paragraph, teaches “The main goal of reinforcement learning is to capture some policy to maximize the cumulative rewards, rewards, which means selecting proper action given some states [(that is, the “policy π” is jointly trained end-to-end on an objective that measures)]”), for each of a plurality of particular observations:
for each of one or more subsequent states that follow the state represented by the particular observation in the trajectory:
(i) a policy error between the predicted policy output for the subsequent state generated conditioned on the particular observation and an actual policy that was used to select an action in response to the observation (Liang at p. 5, “Proximal Policy Optimization (PPO),” second paragraph, teaches “in which Kullback-Leibler(KL) divergence between old policy and updated policy is considered in objective function [(that is, “divergence” is a policy error)], and the KL divergence in each state point can be bounded as well”; Liang at p. 5, Proximal Policy Optimization (PPO),” second paragraph, teaches a “surrogate loss function in original TRPO can be formulated as
PNG
media_image6.png
61
556
media_image6.png
Greyscale
Where π is some stochastic policy,
π
θ
o
l
d
is the parameters in policy in last time [(that is, an actual policy that was used to select an action in response to the observation and πθ is the predicted policy output for the subsequent state generated conditioned on the particular observation)], and At estimates the advantage function of performing at conditioned on the state st at time step t”), and
* * *
Arel teaches –
* * *
for each of one or more subsequent states that follow the state represented by the particular observation in the trajectory:
* * *
(ii) a value error between the value predicted for the subsequent state generated conditioned on the particular observation and an actual return received starting from the subsequent state (Arel ¶ 0059 teaches “[t]he system can measure the performance of the supervised learning models as of a given sequence representation [(that is, “a given sequence,” is starting from the subsequent state)] based on estimation errors between the value function estimates and the actual returns for the state representations before the given state representation in the sequence. The estimation error may be the difference between the value function estimate and the actual return, the square of the difference between the value function estimate and the actual return, or any other appropriate machine learning error measure for the models [(that is, for each of one or more subsequent states that follow the state represented by the particular observation in the trajectory: . . . (ii) a value error between the value predicted for the subsequent state generated conditioned on the particular observation and an actual return received starting from the subsequent state)]”).
13. Claim 17 is rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel], Liang et al., “VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control,” arXiv (2018) [hereinafter Liang], and Mongillo et al., “The Misbehavior of Reinforcement Learning,” IEEE (2014) [hereinafter Mongillo].
Regarding claim 17, the combination of Xie, Schäfer, Arel, and Liang teaches all of the limitations of claim 16, as described above in detail.
Though Xie, Schäfer, Arel, and Liang teach reinforcement learning having measures on predictive performance, the combination of Xie, Schäfer, Arel, and Liang, however, do not explicitly teach –
wherein the objective also measures, for each of the plurality of particular observations:
for each of the one or more subsequent states that follow the state represented by the particular observation in the trajectory:
a reward error between a predicted immediate reward for the subsequent state generated conditioned on the particular observation and an actual immediate reward corresponding to the subsequent state.
But Mongillo teaches –
wherein the objective also measures, for each of the plurality of particular observations:
for each of the one or more subsequent states that follow the state represented by the particular observation in the trajectory:
a reward error between the predicted immediate reward for the subsequent state generated conditioned on the particular observation and an actual immediate reward corresponding to the subsequent state (Mongillo, right column of p. 530, “A. Value-Based Learning,” first paragraph, teaches “[w]e consider the sequence of states, actions, and rewards of an agent interacting with an environment. In each cycle, the agent, being in state s and taking action a, updates its estimate of the corresponding state–action value function Q(a,s) according to
PNG
media_image7.png
84
550
media_image7.png
Greyscale
where η > 0 is the learning rate, and
δ
≡
r
+
Q
(
a
'
,
s
'
)
-
Q
(
a
,
s
)
is the reward prediction error (RPE) [(that is, a reward error)], with a’ and s’ being the next action and the next state, respectively [(that is, the predicted immediate reward for the subsequent state-generated conditioned on the particular observation )], and r is the obtained reward in the cycle [(that is, “Q(a,s)” and “obtained reward r” is an actual immediate reward corresponding to the subsequent state)]. The RPE is a basic quantity that plays a central role in all value-based methods. Roughly speaking, it is a measure of how good the agent is at predicting the consequences of its behavior”).
Xie, Schäfer, Arel, Liang, and Mongillo are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Liang teaches a model-based reinforcement learning algorithm with attention mechanism embedded. Mongillo teaches reinforcement learning is a collection of methods devised to find the optimal policy, a (possibly stochastic) mapping from observations (including hidden observations) to actions, that realizes the goal of the agent, who may receive only partial information due to the hidden state.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Schäfer, Arel, and Liang, pertaining to a model-based RL planner in a reduced dimensional state space implementing an Attention-based Value Function (AVF) model with the reward prediction error of Mongillo.
The motivation to do so is “[reward prediction error (RPE)] is a basic quantity that plays a central role in all value-based methods. Roughly speaking, it is a measure of how good the agent is at predicting the consequences of its behavior”. (Mongillo, right column of p. 530, “A. Value-Based Learning,” first paragraph).
14. Claim 18 is rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel], Liang et al., “VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control,” arXiv (2018) [hereinafter Liang], and Yamato et al., “Recognizing Human Action in Time-Sequential Images using Hidden Markov Model,” IEEE (1992) [hereinafter Yamato].
Regarding claim 18, the combination of Xie, Schäfer, Arel, and Liang teaches all of the limitations of claim 15, as described above in detail.
Though Xie, Schäfer, Arel, and Liang teach joint training with each of the system models, the combination of Xie, Schäfer, Arel, and Liang do not explicitly teach -
wherein [(b.1.3)] the representation model are not trained to output hidden states for reconstruction of the current observation or to model semantics of the environment through the hidden states.
But Yamato teaches -
wherein [(b.1.3)] the representation model is not trained to output hidden states for reconstruction of the current observation or model semantics of the environment through the hidden states observation (Yamato, right paragraph of p. 379, “1. Introduction,” first partial paragraph, teaches “[o]ur purpose is to recognize human action from time-sequential images, not obtaining geometric representations of human bodies. Since we focus on recognition, we avoid reconstruction because the representation obtained by geometric reconstruction is not essential. Instead, we utilize low-level image features in a bottom-up manner [(that is, wherein the representation model is not trained to output hidden states for reconstruction of the current observation . . . .)]”;
[Examiner notes that “planning iterations” similarly are directed to actions, and accordingly, reconstruction is not essential, similar to the teachings of Yamato]).
Xie, Schäfer, Arel, Liang, and Yamato are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Liang teaches a model-based reinforcement learning algorithm with attention mechanism embedded. Yamato teaches the non-necessity of reconstructing observations when it is action recognition being performed.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s claimed invention to modify the combination of Xie, Schäfer, Arel and Liang pertaining to a model-based RL planner in a reduced dimensional state space with the reconstruction avoidance of Yamato.
The motivation to do so is because “However, the reconstruction procedures are neither robust nor reliable for real images. This is because real images are usually too noisy to permit easy model fitting. Thus extracting successful high-level representations from images is very difficult.” (Yamato, right column of p. 379, “1. Introduction,” first partial paragraph).
15. Claim 19 is rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel], Liang et al., “VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control,” arXiv (2018) [hereinafter Liang], and François-Lavet et al., “An Introduction to Deep Reinforcement Learning,” arXiv (2019) [hereinafter François].
Regarding claim 19, the combination of Xie, Schäfer, Arel, and Liang teaches all of the limitations of claim 15, as described above in detail.
Through Xie, Schäfer, Arel, and Liang teach the use of a return function relating to the agent performing selected actions in response to a state representation, the combination of Xie, Arel, and Liang, however, does not explicitly teach –
wherein the actual return starting from the subsequent state is a bootstrapped n-step return.
But François teaches -
wherein the actual return starting from the subsequent state is a bootstrapped n-step return (François at p. 33, “To Bootstrap or not to Bootstrap?,” first paragraph, teaches “Bootstrapping has both advantages and disadvantages. On the negative side, using pure bootstrapping methods (such as in DQN) are prone to instabilities when combined with function approximation because they make recursive use of their own value estimate at the next time-step. On the contrary, methods such as n-step Q-learning rely less on their own value estimate [(that is, return)] because the estimate used is decayed by үn for the nth step backup [(that is, the actual return starting from the subsequent state is a bootstrapped n -step return)]”).
Xie, Schäfer, Arel, Liang, and François are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Liang teaches a model-based reinforcement learning algorithm with attention mechanism embedded. François teaches the use of a target value to update Q-network parameters based on the immediate reward and the following steps in the return.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Arel, and Liang, pertaining to a model-based RL planner in a reduced dimensional state space implementing an Attention-based Value Function (AVF) model with the bootstrapping return of François.
The motivation to do so is because “[b]ootstrapping also has advantages [of allowing] learning from off-policy samples.” (François at p. 33, “To Bootstrap or not to Bootstrap,” second paragraph).
16. Claims 20-22 and 30 are rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel], and François-Lavet et al., “An Introduction to Deep Reinforcement Learning,” arXiv (2019) [hereinafter François].
Regarding claim 20 the combination of Xie, Schäfer, and Arel teaches all of the limitations of claim 11, as described above in detail.
Though Xie, Schäfer, and Arel teaches the use of hidden states in reinforcement learning, the combination of Xie, Schäfer, and Arel does not explicitly teach –
[(c)] wherein selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data comprises
[(c.1)] selecting the action using a markov decision process (MDP) planning algorithm.
But François teaches -
[(c)] wherein selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data comprises
[(c.1)] selecting the action using a markov decision process (MDP) planning algorithm (François, Fig. 10.1, teaches a partially observable Markov decision process (POMDP), where an agent must make action decisions based on observations that do not fully reveal the underlying state (that is, a hidden state) [Examiner annotations in dashed-line text boxes]:
PNG
media_image8.png
392
608
media_image8.png
Greyscale
François, Fig. 10.1 caption, teaches the figure is an “[i]llustration of a POMDP. The actual dynamics of the POMDP is depicted in dark while the information that the agent can use to select the action at each step [(that is, selecting the action)] is the whole history Ht depicted in blue. [(that is, selecting the action using a Markov decision process (MDP) planning algorithm)]”).
Xie, Schäfer, Arel, and François are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. François teaches the use of a MPD to select agent actions.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Schäfer, and Arel, pertaining to a model-based RL planner in a reduced dimensional state space implementing with the MPD for action selection of François.
The motivation to do so is because “deep RL opens up many new applications in domains such as healthcare, robotics, smart grids, finance, and many more. This manuscript provides an introduction to deep reinforcement learning models, algorithms and techniques. Particular focus is on the aspects related to generalization and how deep RL can be used for practical applications.” (François, Abstract).
Regarding claim 21, the combination of Xie, Schäfer, Arel, and François teaches all of the limitations of claim 20, as described above in detail.
François teaches -
wherein selecting the sequence of actions for each planning iteration and selecting the action to be performed by the agent are performed using a monte carlo tree search (MCTS) algorithm (François at p. 47, “6.1.1 Lookahead search,” first and second paragraphs, teaches “[a] lookahead search in an MDP iteratively builds a decision tree where the current state is the root node. . . . Monte-Carlo tree search (MCTS) techniques (Browne et al., 2012) are popular approaches to lookahead search [(that is, wherein selecting the sequence of actions . . . and selecting the action to be performed by the agent are performed using a monte carlo tree search (MCTS) algorithm)]”).
Regarding claim 22, the combination of Xie, Schäfer, Arel, and François teaches all of the limitations of claim 20, as described above in detail.
Arel teaches -
wherein [(c)] selecting, from the set of actions, an action to be performed by the agent in response to the current observation based on the generated plan data comprises:
[(c.2)] determining, from the sequences of actions in the plan data, a sequence of actions that has a maximum associated value output (Arel ¶ 0019 teaches “While the agent is interacting with the environment, the reinforcement learning system selects actions to be performed by the agent in order to maximize the expected return. Generally, the expected return is a function of the rewards anticipated to be received over time in response to future actions performed by the agent. That is, the return is a function of future rewards received starting from the immediate reward received in response to the agent performing the selected action [(that is, determining, from the sequences of actions in the plan data, a sequence of actions that has a maximum associated value output)]”); and
[(c.3)] selecting, as the action to be performed by the agent in response to the current observation, the first action in the determined sequence of actions (Arel, claim 1, teaches “selecting an action to be performed by a computer-implemented agent that interacts with an environment by performing actions selected from a set of actions [(that is, selecting, as the action performed by the agent in response to the current observation, the first action in the determined sequence of actions)]”).
Regarding claim 30, the combination of Xie, Schäfer, and Arel teaches all of the limitations of claim 23, as described above in detail.
Though Xie, Schäfer, and Arel teaches the use of hidden states in reinforcement learning, the combination of Xie, Schäfer, and Arel does not explicitly teach –
wherein selecting the sequence of actions for each planning iteration and selecting the action to be performed by the agent are performed using a monte carlo tree search (MCTS) algorithm.
But François teaches -
wherein selecting the sequence of actions for each planning iteration and selecting the action to be performed by the agent are performed using a monte carlo tree search (MCTS) algorithm (François at p. 47, “6.1.1 Lookahead search,” first and second paragraphs, teaches “[a] lookahead search in an MDP iteratively builds a decision tree where the current state is the root node. . . . Monte-Carlo tree search (MCTS) techniques (Browne et al., 2012) are popular approaches to lookahead search [(that is, wherein selecting the sequence of actions . . . and selecting the action to be performed by the agent are performed using a monte carlo tree search (MCTS) algorithm)]”).
Xie, Schäfer, Arel, and François are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. François teaches the use of a MPD to select agent actions.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s invention to modify the combination of Xie, Schäfer, and Arel, pertaining to a model-based RL planner in a reduced dimensional state space implementing with the MPD for action selection of François.
The motivation to do so is because “deep RL opens up many new applications in domains such as healthcare, robotics, smart grids, finance, and many more. This manuscript provides an introduction to deep reinforcement learning models, algorithms and techniques. Particular focus is on the aspects related to generalization and how deep RL can be used for practical applications.” (François, Abstract).
17. Claims 31 and 32 are rejected under 35 U.S.C. § 103 as being unpatentable over Xie et al., “Improvisation through Physical Understanding: Using Novel Objects as Tools with visual Foresight,” arXiv (2019) [hereinafter Xie] in view of US Published Application 20100049339 to Schäfer et al. [hereinafter Schäfer], US Published Application 20170213150 to Arel et al. [hereinafter Arel], and Yamato et al., “Recognizing Human Action in Time-Sequential Images using Hidden Markov Model,” IEEE (1992) [hereinafter Yamato].
Regarding claims 31 and 32, the combination of Xie, Schäfer, and Arel teaches all of the limitations of claims 11 and 23, respectively, as described above in detail.
Xie teaches -
[(c)] wherein performing the plurality of planning iterations based on the initial hidden state to generate the plan data comprises
[(c.2)] performing the plurality of planning iterations without using a simulator of the environment (Xie, left column of p. 2, “II. Related Work, Learning from Demonstrations,” first paragraph, teaches that “[p]rior work has leveraged demonstrations to accelerate model-free reinforcement learning either in simulation or the real world, overcoming the well-known exploration problem) . . . .
Though Xie, Schäfer, and Arel teaches the use of hidden states in reinforcement learning, the combination of Xie, Schäfer, and Arel does not explicitly teach –
[(c) wherein performing the plurality of planning iterations based on the initial hidden state to generate the plan data comprises
(c.2) performing the plurality of planning iterations] . . . without reconstructing the current observation.
But Yamato teaches -
[(c) wherein performing the plurality of planning iterations based on the initial hidden state to generate the plan data comprises
(c.2) performing the plurality of planning iterations] . . . without reconstructing the current observation (Yamato, right paragraph of p. 379, “1. Introduction,” first partial paragraph, teaches “Our purpose is to recognize human action from time-sequential images, not obtaining geometric representations of human bodies. Since we focus on recognition, we avoid reconstruction because the representation obtained by geometric reconstruction is not essential. Instead, we utilize low-level image features in a bottom-up manner [(that is, performing the plurality of planning iterations . . . without reconstructing the current observation )]”;
[Examiner notes that “planning iterations” similarly are directed to actions, and accordingly, reconstruction is not essential, similar to the teachings of Yamato]).
Xie, Schäfer, Arel, and Yamato are from the same or similar field of endeavor. Xie teaches training a model with both a visual and physical understanding of multi-object interactions by combining diverse demonstration data with self-supervised interaction data to build generalizable models and the demonstration data to guide a model-based RL planner to solve complex tasks. Schäfer teaches learning and optimization methods known per se can be employed after a suitable dimensional reduction of the state space of the states. Arel teaches a reinforcement learning system selects actions to maximize a return, which is a function of immediate rewards to be performed by the agent. Yamato teaches the non-necessity of reconstructing observations when it is action recognition being performed.
Thus, it would have been obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant’s claimed invention to modify the combination of Xie, Schäfer, and Arel pertaining to a model-based RL planner in a reduced dimensional state space with the reconstruction avoidance of Yamato.
The motivation to do so is because “However, the reconstruction procedures are neither robust nor reliable for real images. This is because real images are usually too noisy to permit easy model fitting. Thus extracting successful high-level representations from images is very difficult.” (Yamato, right column of p. 379, “1. Introduction,” first partial paragraph).
Response to Argument
18. Examiner has fully considered Applicant’s arguments, and responds below, accordingly.
Section 101
19. “Applicant notes that the MPEP now explicitly recognizes that "improvements as to how the machine learning model itself operates" constitutes an improvement in computer functionality. MPEP 2106.04(d) (https://vvww.uspto.gov/web/offices/pac/mpep/ANCDesjardins-Memo-12-5-25 .pdf).
The present Specification similarly describes technical improvements that can be achieved by the claimed subject matter with respect to how machine learning models (including a representation model, a dynamics model, and a prediction model) operate to control an agent interacting with an environment to cause the agent to perform a task by selecting, from a set of actions, actions to be performed by the agent. As explained in paragraph 0013 of the Specification,
"This specification describes effectively performing planning for selecting actions to be performed by an agent when controlling the agent in an environment for which a perfect or very high-quality simulator is not available. In particular, tree-based planning methods have enjoyed success in challenging domains where a perfect simulator that simulates environment transition is available. However, in real-world problems the dynamics governing the environment are typically complex and unknown, and planning approaches have so far failed to yield the same performance gains. The described techniques use a learned model combined with an MDP planning algorithm e.g., a tree-based search with a learned model to achieves high quality performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. The described techniques learn a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the action-selection policy, the value function, and, when relevant, the reward, allowing for excellent results to be achieved on a variety of domains where conventional planning techniques had failed to show significant improvement."
The capability of the machine learning models to control an agent to perform a task "in an environment for which a perfect or very high-quality simulator is not available" and "without any knowledge of [the environment's] underlying dynamics" represents "improvements as to how the machine learning model itself operates."
The claims "include the components or steps of the invention that provide the improvement described in the specification (MPEP § 2106.05(a))." (Response at pp. 10-11).
In view of the amended claims, Applicant submits that “[t]he claims thus recite a specific combination of steps - including "[(b)] processing a representation input comprising the current observation using a representation model to generate an initial hidden state," "[(c)] performing a plurality of planning iterations based on the initial hidden state," and "[(c.1)] selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs generated by: [(c.1.1)] (i) a dynamics model that . . . [(c.1.1.3)] generate[s] as output at least a hidden state . . . and [(c.1.2)] (ii) a prediction model that . . . [(c.1.2.2)] generate[s] as output [(c.1.2.2.1)] a) a predicted policy output . . . and [(c.1.2.2.2)] b) a value output" - to generate plan data that can be used to select an action to be performed by the agent.
Further, by using a dynamics model and a prediction model that operate on hidden states that have "[(c.1.1.4)] a lower dimensionality, simpler modality, or both than the current observation," as recited in amended claim 11, "the applicability of the described planning techniques can be extended into these complex tasks with no significant increase in computational overhead of the planning process. Thus, the described techniques can be used to control agents for tasks with large discrete action spaces, continuous action spaces, or hybrid action spaces with reduced latency and reduced consumption of computational resources while still maintaining effective performance." At paragraph 0014 of the specification.” (Response at pp. 12-13).
Examiner Response:
For Step 2A Prong Two, the rejection above identifies any additional elements by specifically pointing to claim features/limitations/steps recited in the claim beyond the identified judicial exception (abstract idea), and evaluates the integration of the judicial exception into a practical application by explaining that the claim as a whole, looking at the additional elements individually and in combination, do not integrate the judicial exception into a practical application using the considerations set forth in MPEP §§ 2106.04(d), 2106.05(a)-(c) and (e)-(h).
Under Step 2A Prong Two, “Integration” may be based on the improvements in the functioning of a computer or an improvement to any other technology or technical field. (MPEP § 2106.04(d)(1)). The evaluation requires, [i]n sum, that (1) the specification should be evaluated to determine if the disclosure provides sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement. Next, (2) if the specification sets forth such an improvement, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement.
By way of example to Desjardins, the MPEP provides under Step 2A Prong Two that “the [Desjardins] specification identified improvements as to how the machine learning model itself operates, including training a machine learning model to learn new tasks while protecting knowledge about previous tasks to overcome the problem of ‘catastrophic forgetting’ encountered in continual learning systems. Importantly, the [appeals review panel (ARP)] evaluated the claims as a whole in discerning at least the limitation ‘adjust the first values of the plurality of parameters to optimize performance of the machine learning model on the second machine learning task while protecting performance of the machine learning model on the first machine learning task’ reflected the improvement disclosed in the specification. Accordingly, the claims as a whole integrated what would otherwise be a judicial exception instead into a practical application at Step 2A Prong Two, and therefore the claims were deemed to be outside any specific, enumerated judicial exception (Step 2A: NO).” (MPEP § 2106.04(d) sub III; see “Advance Notice of Change to the MPEP in light of Ex Parte Desjardins” (05 December 2025) at p. 2)).
Under the first leg of MPEP § 2106.05(d)(1), Applicant points to the Specification as explaining the complexity of real-world problems, where
in real-world problems the dynamics governing the environment are typically complex and unknown, and planning approaches have so far failed to yield the same performance gains. The described techniques use a learned model combined with an MDP planning algorithm e.g., a tree-based search with a learned model to achieves high quality performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. The described techniques learn a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the action-selection policy, the value function, and, when relevant, the reward, allowing for excellent results to be achieved on a variety of domains where conventional planning techniques had failed to show significant improvement.
(Response at pp. 10-11 (quoting Specification ¶ 0013)). Though the disclosure may explicitly set forth an improvement, the disclosure does so in a conclusory manner (i.e., a bare assertion of an improvement without the detail necessary to be apparent to a person of ordinary skill in the art). (MPEP § 2106.04(d)(1)). In other words, the claims do not explain how the abstract ideas and/or additional elements are achieved (e.g., “receiving a current observation,” “processing . . . to generate an initial hidden state,” “([c.1)] selecting a sequence of actions,” etc.). Also, by way of example, though dependent claims recite a “predicted immediate reward value,” no description within the claim explains how the “predicted immediate reward value” is achieved or procured. (see, e.g., claim 12).
The disclosure speaks to the use of a “planning engine” that may be “logically described as a tree, the plan data 122 generated by using the planning engine 120 may be represented by any of a variety of convenient data structures, e.g., as multiple triples or as an adjacency list,” (Specification ¶ 0034; see also Specification ¶ 0035 (“generate the sequence of actions by repeatedly . . . selecting an action a according to the compiled statistics . . . .for example by maximizing over an upper confidence bound [using argmax]”)), by way of example.
In this respect, without more, the claims do not serve to integrate a judicial exception (i.e., abstract idea) into a practical application of the exception because the claims do not apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize or preempt the judicial exception. (2024 SME Guidance, 89 Fed. Reg. 137 at p. 58136 (17 July 2024)).
Accordingly, as set out above in detail, the instant claims are subject-matter ineligible.
Section 103
20. “Applicant respectfully submits that the cited portions of Xie and Arel, either alone or in combination, do not disclose or suggest this combination of features of amended claim 11.
With respect to the rejection of features that relate to the dynamics model, the Office Action cited a portion of Xie that reads as follows:
In practice, the first two images passed into the model are ground truth images, called context frames. At every time-step an action at is passed into the model along with the hidden state ht, producing a new state ht+ 1 and a flow field F t+ 1 ←t which is used to to transform the image via bi-linear sampling.
[Xie, A Video Prediction Model Implementation details, on page 11]
Applicant respectfully submits that the cited portion of Xie does not disclose or suggest how to perform a planning iteration using the dynamics model as recited by amended claim 11.
In particular, the cited portion of Xie does not disclose that
at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model.
Instead, the cited portion of Xie discloses that "the first two images passed into the model are ground truth images, called context frames."
Arel does not alleviate the deficiency of Xie in disclosing these features.
The cited portion of Arel reads "For example, the system can use these state value supervised learning models in conjunction with a transition model that receives a state and an action as input and predicts a state that is most likely to be the state the environment transitions into . . ." at paragraph 0083.
Thus, the cited portion of Arel does not disclose or suggest
(i) a dynamics model that is configured through training to receive as input a) a hidden state corresponding to an input environment state . . . , wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation,
as recited by amended claim 11.
Therefore, Applicant respectfully submits that amended claim 11 is in condition for allowance. Independent claims 23 and 24 and their respective dependent claims are allowable for corresponding reasons.” (Response at pp. 14-15).
Examiner Response:
Examiner finds persuasive Applicant’s arguments and/or amendments that the cited prior art does not teach or disclose
* * *
[(c.1.1.4)] wherein the hidden state has a lower dimensionality, simpler modality, or both than the current observation, and
* * *
(claim 1, lines 13-15). In this respect, Examiner relies upon the teachings of Schäfer, as set out above in detail.
Also, Applicant argues that the cited references do not teach the “planning iterations” of the claims, where:
* * *
[(c.1) selecting a sequence of actions to be performed by the agent starting from the current environment state based on outputs generated by:]
[(c.1.1)] (i) a dynamics model that is configured through training to receive as input . . .
[(c.1.1.1)] a) a hidden state corresponding to an input environment state . . . ,
[(c.1.1.5)] at a first step of the planning iteration, the hidden state corresponding to the input environment state is the initial hidden state generated by the representation model, and
[(c.1.1.6)] at a subsequent step of the planning iteration, the hidden state corresponding to the input environment state is the hidden state generated by the dynamics model in a preceding step of the planning iteration
* * *
(claim 1, lines 19-22 (emphasis added by Examiner showing amended language)). Xie, however, teaches the emphasized limitations, as described above in detail. (see, Xie, right column of p. 4, “2) Action Proposal Model Training,” first paragraph).
Moreover, the rejections hereinabove clearly sets forth which claim limitations are taught by each of the prior art references, and the reason why it would be obvious to a person having ordinary skill in the art as of the effective filing date of the Applicant's invention to combine their teachings, and Applicant has not explained why the cited prior art references cannot be combined in the manner set forth in the rejection.
Conclusion
21. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
22. The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure:
(US Published Application 20220035346 to Mercangoez et al.) teaches The set of trained models is then used to output the predictions, by inputting online measurement results in an original space to two trained models whose outputs are fed, as reduced space inputs and reduced space initial states, to a third trained model. The third trained model processes the reduced space inputs to reduced space predictions. They are fed to a fourth trained model, which outputs the predictions in the original space.
(Xiong et al., “Dynamic Travel Mode Searching and Switching Analysis Considering Hidden Model Preference and Behavioral Decision Processes,” Springer (2015)) teaches conceptual framework to model the travel mode searching and switching dynamics. In the proposed model, each hidden state represents the latent modal preference of each traveler. The empirical application suggests that the states can be interpreted as car loving and carpool/transit loving, respectively. At each time period, transitions between the states are functions of time-varying covariates such as travel time and travel cost of the habitual modes. Once applied with travel demand and/or traffic simulation models, the proposed model can describe time-dependent multimodal behavior responses to various planning/policy stimuli.
23. Any inquiry concerning this communication or earlier communications from the Examiner should be directed to KEVIN L. SMITH whose telephone number is (571) 272-5964. Normally, the Examiner is available on Monday-Thursday 0730-1730.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, KAKALI CHAKI can be reached on 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.L.S./
Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
1 Examiner adds these limitation identifiers for the limited purpose of evaluating the claims for subject matter eligibility under MPEP § 2106, and not for the purpose of oversimplifying the claims.
2 Examiner adds these limitation identifiers for the limited purpose of evaluating the claims for subject matter eligibility under MPEP § 2106, and not for the purpose of oversimplifying the claims.
3 Examiner adds these limitation identifiers for the limited purpose of evaluating the claims for subject matter eligibility under MPEP § 2106, and not for the purpose of oversimplifying the claims.
Prosecution Insights