Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . under the first inventor to file provisions of the AIA .
Claims 1-6, and 11-20 are pending and are being examined. Claims 7-10 are non-elected.
Information Disclosure Statement
The information disclosure statement (IDS) was submitted on 12/26th/2023. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Examiner's Note
The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well.
Claim Rejections - 35 USC § 101
101 Rejection
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-6, and 11-20 is rejected under 35 USC § 101 because the claimed invention is directed to non-statutory subject matter
Step 1 Analysis:
Claims 1-6, and 11-14 are directed to a method, which is directed to a process, one of the statutory categories. Claims 15-20 are directed to a system which is directed to a machine, one of the statutory categories.
Regarding Claim 1:
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 2A Prong 1 Analysis:
Claim 1 recites in part process steps which, under the broadest reasonable interpretation, are a series of mental processes including an observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper. If a claim, under its broadest reasonable interpretation, covers a mental process or a mathematical concept but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas. The claim recites in part:
assign, to each action index in a set of action indices, a respective action from the original action space Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator assigning an action to an index). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
process the current observation to generate a policy output that comprises a respective score for each of the action indices Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator assigning a score to an action index). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
selecting an action index from the set of action indices using the policy output Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator selecting an action index). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
selecting, as an action to be performed by the agent in response to the current observation, the action that was assigned to the selected action index by the discretization neural network by processing the current observation Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator selecting an action to be performed by a robot). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2 Analysis:
The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
A method performed by one or more computers for controlling an agent to interact with an environment by performing actions from an original action space is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
receiving a current observation characterizing a current state of an environment is recited at a high-level of generality and amounts to insignificant extra-solution activity of gathering data for use in the claimed process. As described in MPEP 2106.05(g).
processing the current observation using a discretization neural network that is configured to process the current observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
wherein a total number of action indices in the set of action indices is less than a total number of actions in the original action space is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
processing a policy input comprising the current observation using a policy neural network that is configured to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
causing the agent to perform the selected action in response to the current observation is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Step 2B Analysis:
Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, the additional elements of:
A method performed by one or more computers for controlling an agent to interact with an environment by performing actions from an original action space is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
receiving a current observation characterizing a current state of an environment is recited at a high-level of generality and amounts to extra-solution activity of gathering data for use in the claimed process. As described in MPEP 2106.05(g). The courts have found limitations directed to gathering information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), “receiving or transmitting data over a network”).
processing the current observation using a discretization neural network that is configured to process the current observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
wherein a total number of action indices in the set of action indices is less than a total number of actions in the original action space is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
processing a policy input comprising the current observation using a policy neural network that is configured to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
causing the agent to perform the selected action in response to the current observation is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101.
The additional limitations of the dependent claims contain no additional elements that provide a practical application or amount to significantly more than the abstract idea and are addressed briefly below.
Dependent claim 2:
Step 2A Prong 1: The claim recites similar mental processes as the independent claim by dependency
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the action space is a continuous action space is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the action space is a continuous action space is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
For the reasons above, claim 2 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 3:
Step 2A Prong 1: The claim recites similar mental processes as the independent claim by dependency
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the respective score for each of the action indices is a Q-value that represents an estimated return to be received if the agent performs the action that was assigned to the selected action index by the discretization neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the respective score for each of the action indices is a Q-value that represents an estimated return to be received if the agent performs the action that was assigned to the selected action index by the discretization neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
For the reasons above, claim 3 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 4:
Step 2A Prong 1:
generate an encoded representation of the observation Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator describing an observation in encoded text). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
regress an action from the original action space that is assigned to the action index Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator regressing an action from a list of actions that is assigned to a particular number). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the discretization neural network comprises: an encoder neural network that includes one or more neural network layers and processes the observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
a respective decoder neural network that includes one or more hidden layers for each of the action indices, wherein the respective decoder neural network for each action index processes the encoded representation for the action index to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the discretization neural network comprises: an encoder neural network that includes one or more neural network layers and processes the observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
a respective decoder neural network that includes one or more hidden layers for each of the action indices, wherein the respective decoder neural network for each action index processes the encoded representation for the action index to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
For the reasons above, claim 4 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 5:
Step 2A Prong 1: The claim recites the same mental processes as the independent claim by dependency
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the discretization neural network has been trained on a set of demonstration transitions is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the discretization neural network has been trained on a set of demonstration transitions is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
For the reasons above, claim 5 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 6:
Step 2A Prong 1: The claim recites the same mental processes as the independent claim by dependency
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the policy neural network has been trained through a discrete action reinforcement learning technique after the training of the discretization neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the policy neural network has been trained through a discrete action reinforcement learning technique after the training of the discretization neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
For the reasons above, claim 6 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 11:
Step 2A Prong 1:
generate a policy output that comprises a respective score for each of the action indices Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator assigning a score to each action index). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
after training the discretization neural network, using the trained discretization neural network to train a policy neural network that is configured to process a policy input comprising a current observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
after training the discretization neural network, using the trained discretization neural network to train a policy neural network that is configured to process a policy input comprising a current observation to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
For the reasons above, claim 11 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 12:
Step 2A Prong 1:
Generating ….. an experience tuple that comprises the current observation, the selected action, a next observation received in response to the agent performing the selected action, and a reward value received in response to the agent performing the selected action Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator generating a tuple consisting of an observation, action, next observation, and reward). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
using the trained discretization neural network to train a policy neural network is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
using the trained discretization neural network is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
storing the experience tuple in a replay memory for use in training the policy neural network on experience tuples using a discrete action reinforcement learning technique is recited at a high-level of generality and amounts to extra-solution activity of storing data for use in the claimed process.
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
using the trained discretization neural network to train a policy neural network is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
using the trained discretization neural network is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
storing the experience tuple in a replay memory for use in training the policy neural network on experience tuples using a discrete action reinforcement learning technique is recited at a high-level of generality and amounts to extra-solution activity of storing data for use in the claimed process. As described in MPEP 2106.05(g). The courts have found limitations directed to storing information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), "storing and retrieving information in memory").
For the reasons above, claim 12 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 13:
Step 2A Prong 1:
replacing the selected action in the experience tuple with a closest action to the selected action from the actions that are assigned to any of the action indices Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator replacing an action in a tuple with an action from a list). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
the replay memory stores one or more experience tuples is recited at a high-level of generality and amounts to extra-solution activity of storing data for use in the claimed process. As described in MPEP 2106.05(g).
that include a selected action that would not be assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
for each of the one or more experience tuples, prior to using the experience tuple to train the policy neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
by the trained discretization neural network by processing the corresponding observation in the experience tuple is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
the replay memory stores one or more experience tuples is recited at a high-level of generality and amounts to extra-solution activity of storing data for use in the claimed process. As described in MPEP 2106.05(g). The courts have found limitations directed to storing information electronically, recited at a high level of generality, to be well-understood, routine, and conventional (see MPEP 2106.05(d)(II), "storing and retrieving information in memory").
that include a selected action that would not be assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
for each of the one or more experience tuples, prior to using the experience tuple to train the policy neural network is recited at a high level of generality and amounts to no more than generally linking the use of a judicial exception to a particular technological environment or field of use MPEP 2106.05(h). As explained by the Supreme Court, a claim directed to a judicial exception cannot be made eligible "simply by having the applicant acquiesce to limiting the reach of the patent for the formula to a particular technological use." Diamond v. Diehr, 450 U.S. 175, 192 n.14, 209 USPQ 1, 10 n. 14 (1981). Thus, limitations that amount to merely indicating a field of use or technological environment in which to apply a judicial exception do not amount to significantly more than the exception itself, and cannot integrate a judicial exception into a practical application.
by the trained discretization neural network by processing the corresponding observation in the experience tuple is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
For the reasons above, claim 13 is rejected as being directed to non-patentable subject matter under §101.
Dependent claim 14:
Step 2A Prong 1:
select action indices that maximize expected returns given how actions are assigned to action indices by the trained discretization neural network Under the broadest reasonable interpretation, this limitation is a process step that covers a mental process including observation, evaluation, judgment or opinion that could be performed in the human mind or with the aid of pencil and paper (such as an operator selecting an action list that includes actions that would allow a robot to make a left turn). If a claim, under its broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components, then it falls within the “Mental Process” grouping of abstract ideas.
Step 2A Prong 2: The judicial exception is not integrated into a practical application. In particular, the claim recites the additional element of:
training the policy neural network comprises training the policy neural network to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
Step 2B: In accordance with Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more that the judicial exception. As discussed above, the additional elements of:
training the policy neural network comprises training the policy neural network to is recited at a high-level of generality and amounts to no more than mere instructions to apply the exception using a generic computer component (See MPEP 2106.05(f)).
For the reasons above, claim 14 is rejected as being directed to non-patentable subject matter under §101.
Claims 15-20 are the system claims corresponding to the method claims 1-6, respectively, therefore, they are rejected based upon the same rationale as the rejection of claims 1-6.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over TANG (Discretizing Continuous Action Space for On-Policy Optimization - 2020), in view of XU (PARTICLE-BASED ADAPTIVE DISCRETIZATION FOR CONTINUOUS CONTROL USING DEEP REINFORCEMENT LEARNING - 2020).
Regarding claim 1, TANG teaches A method performed by one or more computers for controlling an agent to interact with an environment by performing actions from an original action space, the method comprising: receiving a current observation characterizing a current state of an environment ([Page 1, Sec. 2.1] In the standard formulation of Markov Decision Process (MDP), an agent starts with an initial state S0 at time t=0. At time t ≥ 0 , the agent is in St, takes an action at, receives a reward rt and transitions to a next state St+1).
processing the current observation using a discretization neural network that is configured to process the current observation ([Page 3, Sec. 3.1] The discrete policy is parameterized as follows. As in prior works (Schulman et al., 2015b; 2017b), the policy πθ is a neural network that takes state s as input, through multiple layers of transformation it will encode the state into a hidden vector h(s) = fθ(s). For the jth action in the ith dimension of the action space, we output a logit. For any dimension i, the K logits are combined by soft-max to compute the probability of choosing action j).
to assign, to each action index in a set of action indices, a respective action from the original action space, wherein a total number of action indices in the set of action indices is less than a total number of actions in the original action space ([Page 4, Sec. 3] Without loss of generality, we assume the action space A = [-1, 1]m. We discretize each dimension of the action space into K equally spaced atomic actions. The set of atomic action for any dimension i is
PNG
media_image1.png
44
290
media_image1.png
Greyscale
. The examiner notes that YANG teaches assigning to each action index in a set of action indices (k-1), a respective action from the original action space (k), wherein a total number of action indices in the set of action indices is less than a total number of actions in the original action space.
processing a policy input comprising the current observation using a policy neural network that is configured to process the current observation to generate a policy output that comprises a respective score for each of the action indices; selecting an action index from the set of action indices using the policy output; selecting, as an action to be performed by the agent in response to the current observation, the action that was assigned to the selected action index by the discretization neural network by processing the current observation ([Page 3, Sec. 3.1] The discrete policy is parameterized as follows. As in prior works (Schulman et al., 2015b; 2017b), the policy πθ is a neural network that takes state s as input, through multiple layers of transformation it will encode the state into a hidden vector h(s) = fθ(s). For the jth action in the ith dimension of the action space, we output a logit. For any dimension i, the K logits are combined by soft-max to compute the probability of choosing action j. The examiner notes that TANG teaches processing a state as input and outputs a logit that is used to compute a probability score of choosing an action j from an action index).
However, TANG is not relied upon to explicitly teach causing the agent to perform the selected action in response to the current observation. On the other hand, XU teaches causing the agent to perform the selected action in response to the current observation ([Page 4, first para.] Sampling through the mixture of Gaussians can be done by two steps. First, we perform sampling on the categorical distribution to choose a particle, j, for each dimension, k, based on the weights. Then, we can draw samples from the Gaussian distribution represented by the chosen particles. Sampling through the weighted particles is stratified, which is considered optimal in terms of variance [35]. The examiner notes that XU teaches selecting an action for a robot to perform based on a chosen particle. The examiner further notes that TANG and XU are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s selection process to incorporate causing the agent to perform the selected action in response to the current observation as taught by XU [Page 4, first para.] to explore actions at several different locations in the action space [0021]).
Regarding claim 2, TANG teaches the action space is a continuous action space ([Page 1, Sec. 1] A straightforward solution is to discretize the continuous action space).
Claims 15-16 are the system claims corresponding to the method claims 1-2 respectively, therefore, they are rejected based upon the same rationale as the rejection of claims 1-2.
Claims 3, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over TANG (Discretizing Continuous Action Space for On-Policy Optimization - 2020), in view of XU (PARTICLE-BASED ADAPTIVE DISCRETIZATION FOR CONTINUOUS CONTROL USING DEEP REINFORCEMENT LEARNING - 2020), further in view of BADIA (US20230059004A1).
Regarding claim 3, TANG teaches the method of claim 1, however, TANG is not relied upon to explicitly teach the respective score for each of the action indices is a Q-value that represents an estimated return to be received if the agent performs the action that was assigned to the selected action index by the discretization neural network. On the other hand, BADIA teaches the respective score for each of the action indices is a Q-value that represents an estimated return to be received if the agent performs the action that was assigned to the selected action index by the discretization neural network ([0080] The training engine 208 can then train the action selection neural network on the trajectory using a reinforcement learning technique. The reinforcement learning technique can be, e.g., a Q-learning technique, e.g., a Retrace Q-learning technique or a Retrace Q-learning technique with a transformed Bellman operator, such that the action selection neural network is a Q neural network and the action scores are Q values that estimate expected returns that would be received if the corresponding actions were performed by the agent. The examiner notes that TANG and XU are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s selection process to incorporate the respective score for each of the action indices is a Q-value that represents an estimated return to be received if the agent performs the action that was assigned to the selected action index by the discretization neural network as taught by BADIA [0080] to estimate expected returns that would be received if the corresponding actions were performed by the agent [0080]).
Claim 17 is the system claim corresponding to the method claim 3, therefore, it is rejected based upon the same rationale as the rejection of claim 3.
Claims 4-6, 11, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over TANG (Discretizing Continuous Action Space for On-Policy Optimization - 2020), in view of XU (PARTICLE-BASED ADAPTIVE DISCRETIZATION FOR CONTINUOUS CONTROL USING DEEP REINFORCEMENT LEARNING - 2020), further in view of ALLSHIRE (LASER: Learning a Latent Action Space for Efficient Reinforcement Learning - 2021).
Regarding claim 4, TANG teaches an encoder neural network that includes one or more neural network layers and processes the observation to generate an encoded representation of the observation ([Page 3, Sec. 3.1] The discrete policy is parameterized as follows. As in prior works (Schulman et al., 2015b; 2017b), the policy πθ is a neural network that takes state s as input, through multiple layers of transformation it will encode the state into a hidden vector h(s) = fθ(s). For the jth action in the ith dimension of the action space, we output a logit. For any dimension i, the K logits are combined by soft-max to compute the probability of choosing action j. The examiner notes that TANG teaches encoding an observation into an observation representation).
However, TANG is not relied upon to explicitly teach a respective decoder neural network that includes one or more hidden layers for each of the action indices, wherein the respective decoder neural network for each action index processes the encoded representation for the action index to regress an action from the original action space that is assigned to the action index. On the other hand, ALLSHIRE teaches a respective decoder neural network that includes one or more hidden layers for each of the action indices, wherein the respective decoder neural network for each action index processes the encoded representation for the action index to regress an action from the original action space that is assigned to the action index ([Page 6652, Sec. IV.A] The function f : A¯→ A defined in Sec. III for mapping from latent actions to control inputs in the original action space will be represented by a latent state-dependent variational decoder neural network, DθD: Sr × A¯ → A parameterized by θD, where ˆa = DθD (sr , ¯a) is the reconstruction of a, an action in the original space that would have resulted in ¯a ∼ EθE (a). The examiner notes that TANG and ALLSHIRE are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate a respective decoder neural network that includes one or more hidden layers for each of the action indices, wherein the respective decoder neural network for each action index processes the encoded representation for the action index to regress an action from the original action space that is assigned to the action index as taught by ALLSHIRE [Page 6652, Sec. IV.A] to map the policy’s latent actions back to the original action space [Page 6652, Sec. IV.A]).
Regarding claim 5, TANG teaches the method of claim 1. however, TANG is not relied upon to explicitly teach the discretization neural network has been trained on a set of demonstration transitions. On the other hand, ALLSHIRE teaches the discretization neural network has been trained on a set of demonstration transitions ([Page 6654, Sec. V.B.Exp1] The aim of this experiment is to test whether policy learning is more efficient in a learned latent action space than in an original action space. First, we train a RL policy to convergence on a set of tasks. We then sample 1,000 episodes from this expert policy on each task to form a dataset of expert experiences, which we use to train SAC on action spaces learned by LASER and its ablations. dataset of experiences (consisting of state, action, next state tuples) and train LASER on this dataset. The examiner notes that TANG and ALLSHIRE are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate the discretization neural network has been trained on a set of demonstration transitions as taught by ALLSHIRE [Page 6654, Sec. V.B.Exp1] to test whether policy learning is more efficient in a learned latent action space than in an original action space [Page 6654, Sec. V.B.Exp1]).
Regarding claim 6, TANG teaches the method of claim 5. however, TANG is not relied upon to explicitly teach the policy neural network has been trained through a discrete action reinforcement learning technique after the training of the discretization neural network. On the other hand, ALLSHIRE teaches the policy neural network has been trained through a discrete action reinforcement learning technique after the training of the discretization neural network ([Sec. IV.A, second column] After learning an action space representation with LASER, an RL policy π : S → A¯ can then be trained in this latent action space using the decoder to map the policy’s latent actions back to the original action space. The examiner notes that ALLSHIRE teaches training a neural network policy using reinforcement learning after learning an action space representation. The examiner notes that TANG and ALLSHIRE are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate the policy neural network has been trained through a discrete action reinforcement learning technique after the training of the discretization neural network as taught by ALLSHIRE [Sec. IV.A] to map the policy’s latent actions back to the original action space [Sec. IV.A]).
Regarding claim 11, TANG teaches process a policy input comprising a current observation to generate a policy output that comprises a respective score for each of the action indices ([Page 3, Sec. 3.1] The discrete policy is parameterized as follows. As in prior works (Schulman et al., 2015b; 2017b), the policy πθ is a neural network that takes state s as input, through multiple layers of transformation it will encode the state into a hidden vector h(s) = fθ(s). For the jth action in the ith dimension of the action space, we output a logit. For any dimension i, the K logits are combined by soft-max to compute the probability of choosing action j).
However, TANG is not relied upon to explicitly after training the discretization neural network, using the trained discretization neural network to train a policy neural network that is configured to. On the other hand, ALLSHIRE teaches after training the discretization neural network, using the trained discretization neural network to train a policy neural network that is configured to ([Sec. IV.A, second column] After learning an action space representation with LASER, an RL policy π : S → A¯ can then be trained in this latent action space using the decoder to map the policy’s latent actions back to the original action space. The examiner notes that ALLSHIRE teaches training a neural network policy using reinforcement learning after learning an action space representation. The examiner notes that TANG and ALLSHIRE are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate after training the discretization neural network, using the trained discretization neural network to train a policy neural network that is configured to as taught by ALLSHIRE [Sec. IV.A] to map the policy’s latent actions back to the original action space [Sec. IV.A]).
Claims 18-20 are the system claims corresponding to the method claims 4-6 respectively, therefore, they are rejected based upon the same rationale as the rejection of claims 4-6.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over TANG (Discretizing Continuous Action Space for On-Policy Optimization - 2020), in view of XU (PARTICLE-BASED ADAPTIVE DISCRETIZATION FOR CONTINUOUS CONTROL USING DEEP REINFORCEMENT LEARNING - 2020), further in view of ALLSHIRE (LASER: Learning a Latent Action Space for Efficient Reinforcement Learning – 2021), further in view of JI (US20220121920A1).
Regarding claim 12, TANG teaches the method of claim 11, however, TANG is not relied upon to explicitly teach generating, using the trained discretization neural network, an experience tuple that comprises the current observation, the selected action, a next observation received in response to the agent performing the selected action, and a reward value received in response to the agent performing the selected action; and storing the experience tuple in a replay memory for use in training the policy neural network on experience tuples using a discrete action reinforcement learning technique. On the other hand, JI teaches generating, using the trained discretization neural network, an experience tuple that comprises the current observation, the selected action, a next observation received in response to the agent performing the selected action, and a reward value received in response to the agent performing the selected action; and storing the experience tuple in a replay memory for use in training the policy neural network on experience tuples using a discrete action reinforcement learning technique ([0014] The memory 121 is configured to store empirical data. The empirical data includes a coordination pattern label, observations at a current moment, action vectors at the current moment, a "pseudo reward" at the current moment, observations at a next moment, and so on. The examiner notes that TANG and JI are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate generating, using the trained discretization neural network, an experience tuple that comprises the current observation, the selected action, a next observation received in response to the agent performing the selected action, and a reward value received in response to the agent performing the selected action; and storing the experience tuple in a replay memory for use in training the policy neural network on experience tuples using a discrete action reinforcement learning technique as taught by JI [0014] to improve the efficiency of multi-agent learning [0013]).
Claims 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over TANG (Discretizing Continuous Action Space for On-Policy Optimization - 2020), in view of XU (PARTICLE-BASED ADAPTIVE DISCRETIZATION FOR CONTINUOUS CONTROL USING DEEP REINFORCEMENT LEARNING - 2020), further in view of ALLSHIRE (LASER: Learning a Latent Action Space for Efficient Reinforcement Learning – 2021), further in view of JI (US20220121920A1), further in view of CHEN (Transfer with Action Embeddings for Deep Reinforcement Learning – 2021).
Regarding claim 13, TANG teaches the method of claim 12, however, TANG is not relied upon to explicitly teach:
the replay memory stores one or more experience tuples that include a selected action that would not be assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple.
for each of the one or more experience tuples, prior to using the experience tuple to train the policy neural network, replacing the selected action in the experience tuple with a closest action to the selected action from the actions that are assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple
On the other hand, CHEN teaches the replay memory stores one or more experience tuples that include a selected action that would not be assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple ([Page 4, Sec. 4.2] The agent executes action at = g(^at), receives reward rt, observes next state st+1, and stores the transition (st; ^at; r; st+1) to the replay buffer B (Lines 7-8). Then it updates the policy model and the action embeddings and transition model accordingly following SAC loss [Haarnoja et al., 2018] and Equation (A.1) (Lines 9-12). The examiner notes that CHEN teaches storing a tuble that includes proro-action ^at that is not performed. The examiner further notes that TANG and CHEN are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate the replay memory stores one or more experience tuples that include a selected action that would not be assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple as taught by CHEN [Page 4, Sec. 4.2] to map the output of the policy model (continuous embedding space) to the original discrete action space [Page 3, Sec. 4.2]).
Furthermore, CHEN teaches for each of the one or more experience tuples, prior to using the experience tuple to train the policy neural network, replacing the selected action in the experience tuple with a closest action to the selected action from the actions that are assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple ([Page 3, Sec. 4.2] Then the real action performed is chosen by a nearest neighbor in the learned action embeddings:
PNG
media_image2.png
46
314
media_image2.png
Greyscale
where g(.) is a mapping from a continuous space to a discrete space. It returns an action in A that is closest to proto-action ^a in embedding space by L2 distance. The agent executes action at = g(^at), receives reward rt, observes next state st+1, and stores the transition (st; ^at; r; st+1) to the replay buffer B (Lines 7-8). Then it updates the policy model and the action embeddings and transition model accordingly following SAC loss [Haarnoja et al., 2018] and Equation (A.1) (Lines 9-12). Then it updates the policy model and the action embeddings and transition model accordingly following SAC loss [Haarnoja et al., 2018] and Equation (A.1) (Lines 9-12). The examiner notes that CHEN teaches updasting the policy model and the action embeddings and transition model with the performed action which is the nearest neighbor to the proto-action in the leaned action embeddings. The examiner further notes that TANG and CHEN are both directed to machine learning and both are reasonably analogous to the claimed invention. Therefore, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified TANG’s model to incorporate for each of the one or more experience tuples, prior to using the experience tuple to train the policy neural network, replacing the selected action in the experience tuple with a closest action to the selected action from the actions that are assigned to any of the action indices by the trained discretization neural network by processing the corresponding observation in the experience tuple as taught by CHEN [Page 3, Sec. 4.2] to map the output of the policy model (continuous embedding space) to the original discrete action space [Page 3, Sec. 4.2]).
Regarding claim 14, TANG teaches training the policy neural network comprises training the policy neural network to select action indices that maximize expected returns given how actions are assigned to action indices by the trained discretization neural network ([Page 1, Sec. 2.1] The objective is to search for the optimal policy that achieves maximum reward. The examiner notes that TANG [Page 3, Sec. 3] teaches discretizing each dimension of the action space into K equally spaced atomic actions. TANG also teaches training the network with an objective of finding actions that maximize the reward).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
REED (US20200104680A1)
“REED teaches a method for for training an action selection policy neural network, wherein the action selection policy neural network is configured to process an observation characterizing a state of an environment to generate an action selection policy output, wherein the action selection policy output is used to select an action to be performed by an agent interacting with an environment”
PIETQUIN (US20210397959A1)
“PIETQUIN teaches a method for for training a neural network used to select actions performed by an agent interacting with an environment by performing actions that cause the environment to transition states”
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAMCY ALGHAZZY whose telephone number is (571)272-8824. The examiner can normally be reached on M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, OMAR FERNANDEZ RIVAS can be reached on (571) 272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAMCY ALGHAZZY/Examiner, Art Unit 2128
/OMAR F FERNANDEZ RIVAS/Supervisory Patent Examiner, Art Unit 2128
Prosecution Insights