Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is responsive to the Application filed on 8/2/2023. Claims 1- 19 and 22 are pending in the case. Claims 1 and 22 are independent claims.
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12, 14, 17-19 and 22 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more.
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined whether the claim is directed to one of the four statutory categories of invention, i.e., process, machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the statutory categories, the second step in the analysis is to determine whether the claim is directed to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception (e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the claims integrate the judicial exception into a practical application. If itis determined at step 2A, Prong 2 that the claims do not integrate the judicial exception into a practical application, the analysis proceeds to determining whether the claim is a patent-eligible application of the exception (Step 2B). If an abstract idea is present in the claim, any element or combination of elements in the claim must be sufficient to ensure that the claim integrates the judicial exception into a practical application, or else amounts to significantly more than the abstract idea itself. Applicant is advised to consult the 2019 PEG for more details of the analysis.
Step 1 Analysis: Is the claim to a process, machine, manufacture or composition of matter? See MPEP § 2106.03.
Claims 1-19 are drawn to a method, claim 22 is drawn to an apparatus, therefore each of these claim groups falls under one of four categories of statutory subject matter (machine/products/apparatus, process/method, manufactures and compositions of mater; Step 1). Nonetheless, the claims are directed to a judicially recognized exception of an abstract idea without significant more (Step 2A, see below). Independent claims 1 and 22 are non-verbatim but similar in claim construction, hence share the same rationale that the claimed inventions are directed to non-statutory subject matter as follows:
As to claim 1:
Claim 1 recites “A computer implemented method for configuring a reinforcement learning agent to perform an efficient reinforcement learning procedure, wherein the reinforcement learning agent comprises a model trained using a machine learning process to determine actions to be performed by the reinforcement learning agent, the method comprising: using the model to determine an action to perform, based on values of a set of features obtained in an environment; determining, for a first feature in the set of features, a first indication of a relative contribution of the first feature, compared to other features in the set of features, to the determination of the action by the model; and determining a reward to be given to the reinforcement learning agent in response to performing the action, based on the first feature and the first indication.”
Step 2A Prong One Analysis: Does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1).
Yes, the limitation “determine actions to be performed” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III). Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of observe someone’s action, which is an observation or evaluation that is practically capable of being performed in the human mind with the assistance of pen and paper. See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “determine an action to perform, based on values of a set of features obtained in an environment” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III). Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of observe someone’s action based on environment, which is an observation or evaluation that is practically capable of being performed in the human mind with the assistance of pen and paper. See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “determining, for a first feature in the set of features, a first indication of a relative contribution of the first feature, compared to other features in the set of features, to the determination of the action by the model” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Yes, the limitation “determining a reward to be given to … in response to performing the action, based on the first feature and the first indication” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A Prong Two Analysis: Does the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d).
No, this limitation “a reinforcement learning agent to perform an efficient reinforcement learning procedure, wherein the reinforcement learning agent comprises a model trained using a machine learning process” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception and reciting only the idea of a solution or outcome, i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is unclear how the “reinforcement learning agent” is used nor the specification makes it clear how these actions are performed. Thus, these additional elements are recited in a manner that represent no more than mere instructions to apply the judicial exceptions on a computer. See MPEP § 2106.05(f) and § 2106.04(d).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea when considered as an ordered combination and as a whole.
Step 2B Analysis: Does the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05.
First, the additional elements directed to generally linking the use of a judicial exception to a particular technological environment or field of use are deemed insufficient to transform the judicial exception to a patentable invention because the claimed limitations generally link the judicial exception to the technology environment, see MPEP 2106.05(h). However, they are included below for the sake of completeness.
Second, the additional elements mere application of the abstract idea or mere instructions to implement an abstract idea on a computer are deemed insufficient to transform the judicial exception to a patentable invention because the limitations generally apply the use of a generic computer and/or process with the judicial exception. See MPEP 2106.05(f). However, they are included below for the sake of completeness.
No, this limitation “a reinforcement learning agent to perform an efficient reinforcement learning procedure, wherein the reinforcement learning agent comprises a model trained using a machine learning process” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception and reciting only the idea of a solution or outcome, i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is unclear how the “reinforcement learning agent” is used nor the specification makes it clear how these actions are performed. Thus, these additional elements are recited in a manner that represent no more than mere instructions to apply the judicial exceptions on a computer. See MPEP § 2106.05(f) and § 2106.04(d).
Thus, considering the additional elements individually and in combination and the claims as a whole, the additional elements do not provide significantly more than the abstract idea. The claims are not eligible subject matter.
Therefore, in examining elements as recited by the limitations individually and as an ordered combination, as a whole the independent claim limitations do not recite what have the courts have identified as “significantly more”.
Furthermore, regarding dependent claims 2-12, 14, 17-19 which are dependent on claim 1, the claims are directed to a judicial exception without significantly more as highlighted below in the claim limitations by evaluating the claim limitations under Step 2A and 2B:
Dependent claim 2
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “penalising the reinforcement learning agent if the first indication indicates that the first feature most strongly contributed to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 3
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “rewarding the reinforcement learning agent if the first indication indicates that the first feature most strongly contributed to the determination of the action by the model and the first feature is a correct feature with which to have determined the action.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 4
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “penalising the reinforcement learning agent if the first indication indicates that the first feature least strongly contributed to the determination of the action by the model and the first feature is a correct feature with which to have determined the action.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 5
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “modifying values in a reward function based on the action, the first feature and the first indication” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 6
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “decreasing the respective reward in the reward function by a predetermined increment if: iv) the action is an incorrect action but the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is a correct feature with which to have determined the action; v) the action is a correct action but the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action; or vi) the action is an incorrect action and the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 7
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “determining, for a second feature in the set of features, a second indication of a relative importance of the second feature, compared to other features in the set of features, in the determination of the action by the model; and wherein the step of determining a reward to be given to the reinforcement learning agent in response to performing the action is further based on the second feature and the second indication.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 8
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “initiating the determined action” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 9
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “obtaining updated values of the set of features after the action is performed; and using the values of the set of features, the determined action, the determined reward, and the updated values of the set of features as training data to train the model.” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No.
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No.
Dependent claim 10
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No, this limitation “performed by a node in a communications network and the set of features are obtained by the communications network” amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform “perform[ing]”. Therefore, the additional limitation is insignificant extra-solution activity to the judicial exception, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No, this limitation “performed by a node in a communications network and the set of features are obtained by the communications network” amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform “perform[ing]”. Therefore, the additional limitation is insignificant extra-solution activity to the judicial exception, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(g). Furthermore the additional element is directed to receiving or transmitting data over a network / performing repetitive calculations / electronic recordkeeping / storing and retrieving information in memory / electronically scanning or extracting data from a physical document, which the courts have recognized as well‐understood, routine, and conventional when they are claimed in a generic manner. See MPEP § 2106.05(d)(II).
Dependent claim 11
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Yes, the limitation “adjustment of operational parameters” is the abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion). See MPEP § 2106.04(a)(2)(III).
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No, this limitation “communications network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(f)(2).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No, this limitation “communications network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception, or merely uses a computer in its ordinary capacity as a tool to perform an existing process, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(f)(2).
Dependent claim 12
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) This limitation “determining a tilt angle for an antenna in the communications network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? This limitation “determining a tilt angle for an antenna in the communications network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Dependent claim 14
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) This limitation “determining movements of a mobile robot or autonomous vehicle receiving instructions through the communications network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? This limitation “determining movements of a mobile robot or autonomous vehicle receiving instructions through the communications network” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Dependent claim 17
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) This limitation “determining the first indication of the relative importance of the first feature is performed using an explainable artificial intelligence, XAI, process” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? This limitation “determining the first indication of the relative importance of the first feature is performed using an explainable artificial intelligence, XAI, process” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Dependent claim 18
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) This limitation “the reinforcement learning agent is a deep reinforcement learning agent” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? This limitation “the reinforcement learning agent is a deep reinforcement learning agent” is an additional element that generally links the use of the judicial exception to a particular technological environment or field of use, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(h).
Dependent claim 19
Incorporates the rejection of independent claim
Step 2A Prong 1: does the claim recite an abstract idea, law of nature, or natural phenomenon? See MPEP § 2106.04(II)(A)(1). Incorporates the abstract idea of independent claim.
Step 2A prong 2: the claim recite additional elements that integrate the judicial exception into a practical application? See MPEP § 2106.04(d) No, this limitation “neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception and reciting only the idea of a solution or outcome, i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is unclear how the “NN” is used nor the specification makes it clear how these actions are performed. Thus, these additional elements are recited in a manner that represent no more than mere instructions to apply the judicial exceptions on a computer. See MPEP § 2106.05(f) and § 2106.04(d). No, this limitation “trained to take as input values of the set of features and output Q-values for possible actions” amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform “train”. Therefore, the additional limitation is insignificant extra-solution activity to the judicial exception, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(g).
Step 2B: the claim recite additional elements that amount to significantly more than the judicial exception? See MPEP § 2106.05. and Is the additional element recognized as well-understood, routine, and conventional? No, this limitation “neural network” is an additional element that amounts to adding the words “apply it” (or an equivalent) with the judicial exception and reciting only the idea of a solution or outcome, i.e., the claim fails to recite details of how a solution to a problem is accomplished because it is unclear how the “NN” is used nor the specification makes it clear how these actions are performed. Thus, these additional elements are recited in a manner that represent no more than mere instructions to apply the judicial exceptions on a computer. See MPEP § 2106.05(f) and § 2106.04(d). No, this limitation “trained to take as input values of the set of features and output Q-values for possible actions” amounts to mere data gathering. It is necessary to acquire the data in order to use the recited judicial exception to perform “train”. Therefore, the additional limitation is insignificant extra-solution activity to the judicial exception, and as such is deemed insufficient to transform the judicial exception to a patentable invention. See MPEP §§ 2106.04(d), 2106.05(g).
The dependent claims as analyzed above, do not recite limitations that integrated the judicial exception into a practical application. In addition, the claim limitations do not include additional elements that are sufficient to amount to significantly more than the judicial exception (Step 2B). Therefore, the claims do not recite any limitations, when considered individually or as a whole, that recite what the courts have identified as “significantly more”, see MPEP 2106.05; and therefore, as a whole the claims are not patent eligible.
As shown above, the dependent claims do not provide any additional elements that when considered individually or as an ordered combination, amount to significantly more than the abstract idea identified. Therefore, as a whole the dependent claims do not recite what the courts have identified as “significantly more” than the recited judicial exception.
Therefore, claims 1-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception and does not recite, when claim elements are examined individually and as a whole, elements that the courts have identified as “significantly more” than the recited judicial exception.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1- 16 and 22 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by
Robinson et al (WO 2020224910 A1).
Referring to claims 1 and 22, Robinson discloses a computer implemented method for configuring a reinforcement learning agent to perform an efficient reinforcement learning procedure, (page 22, 1.29-32 of Robinson, “the agents may make user of recurrent neural networks (such as recurrent autoencoders) determining actions based on the present, on past sates) wherein the reinforcement learning agent comprises a model trained using a machine learning process to determine actions to be performed by the reinforcement learning agent, the method comprising:
using the model to determine an action to perform, based on values of a set of features obtained in an environment; (page 24 of Robinson, “The agent is configured to determine an action 80 in response to the current state (e.g. an action to adjust the configuration of the associated sensor(s) to focus on more important features within the environment). At each time-step, the node is configured to determine an action 80 based on encoded data 60 (be that the encoded sensor data from the decoder 12 or the predicted encoded sensor data from the parent node). Reinforcement learning can then be used to train the system to learn the optimal actions for various states. Whilst this embodiment relates to the determination of an action at each time-step, in alternative embodiments, an action may be determined after a predefined number of time-steps or after a certain criteria has been reached (e.g. at a specified time-step).)
determining, for a first feature in the set of features, a first indication of a relative contribution of the first feature, compared to other features in the set of features, to the determination of the action by the model; (page 27, 1.11-27 of Robinson, predefined features that are identified as being important) and
determining a reward to be given to the reinforcement learning agent in response to performing the action, based on the first feature and the first indication. (page 27, 1.21-23 of Robinson, a value, reward or cost function implemented by each agent (the action generator) to determine the next action may include a weighting towards predefined features that are identified as being important)
Referring to claim 2, Robinson discloses the method as in claim 1 wherein the step of determining a reward comprises: penalising the reinforcement learning agent if the first indication indicates that the first feature most strongly contributed to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action. (page 10, l.3-9 of Robinson, the machine learning agent may be biased to select actions that focus on predefined features of interest (e.g., user defined features) and whilst the agent may be biased towards one or more features of interest, it could equally be considered to be biased away from features of low importance. Page 27, l.3-6, by optimsing based on the prediction error gradients (biasing the system towards areas that provide the greatest learning), the actions in the system are biased towards regions of novelty (e.g., regions with new features less observed with lighter uncertainty about their subsequent states) and/or changes in the environment (e.g., previously observed objects moving within the environment))
Referring to claim 3, Robinson discloses the method as in claim 1 wherein the step of determining a reward comprises: rewarding the reinforcement learning agent if the first indication indicates that the first feature most strongly contributed to the determination of the action by the model and the first feature is a correct feature with which to have determined the action. (page 10, l.3-9 of Robinson, the machine learning agent may be biased to select actions that focus on predefined features of interest (e.g., user defined features) and whilst the agent may be biased towards one or more features of interest, it could equally be considered to be biased away from features of low importance. Page 27, l.3-6, by optimsing based on the prediction error gradients (biasing the system towards areas that provide the greatest learning), the actions in the system are biased towards regions of novelty (e.g., regions with new features less observed with lighter uncertainty about their subsequent states) and/or changes in the environment (e.g., previously observed objects moving within the environment))
Referring to claim 4, Robinson discloses the method as in claim 1 wherein the step of determining a reward comprises: penalising the reinforcement learning agent if the first indication indicates that the first feature least strongly contributed to the determination of the action by the model and the first feature is a correct feature with which to have determined the action. (page 10, l.3-9 of Robinson, the machine learning agent may be biased to select actions that focus on predefined features of interest (e.g., user defined features) and whilst the agent may be biased towards one or more features of interest, it could equally be considered to be biased away from features of low importance. Page 27, l.3-6, by optimsing based on the prediction error gradients (biasing the system towards areas that provide the greatest learning), the actions in the system are biased towards regions of novelty (e.g., regions with new features less observed with lighter uncertainty about their subsequent states) and/or changes in the environment (e.g., previously observed objects moving within the environment))
Referring to claim 5, Robinson discloses the method as in claim 1 wherein the step of determining a reward comprises: modifying values in a reward function based on the action, the first feature and the first indication. (page 27, I.21-23, a value, reward or cost function implemented by each agent (the action generator) to determine the next action may include a weighting towards predefined features that are identified as being important)
Referring to claim 6, Robinson discloses the method as in claim 5 wherein the values in the reward function are modified by: decreasing the respective reward in the reward function by a predetermined increment if: iv) the action is an incorrect action but the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is a correct feature with which to have determined the action; v) the action is a correct action but the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action; or vi) the action is an incorrect action and the first indication indicates that the first feature contributed most strongly to the determination of the action by the model and the first feature is an incorrect feature with which to have determined the action. (page 10, l.3-9 of Robinson, the machine learning agent may be biased to select actions that focus on predefined features of interest (e.g., user defined features) and whilst the agent may be biased towards one or more features of interest, it could equally be considered to be biased away from features of low importance. Page 27, l.3-6, by optimsing based on the prediction error gradients (biasing the system towards areas that provide the greatest learning), the actions in the system are biased towards regions of novelty (e.g., regions with new features less observed with lighter uncertainty about their subsequent states) and/or changes in the environment (e.g., previously observed objects moving within the environment))
Referring to claim 7, Robinson discloses the method as in claim 1, further comprising: determining, for a second feature in the set of features, a second indication of a relative importance of the second feature, compared to other features in the set of features, in the determination of the action by the model; and wherein the step of determining a reward to be given to the reinforcement learning agent in response to performing the action is further based on the second feature and the second indication. (page 27, I.21-23, a value, reward or cost function implemented by each agent (the action generator) to determine the next action may include a weighting towards predefined features that are identified as being important. page 10, l.3-9 of Robinson, the machine learning agent may be biased to select actions that focus on predefined features of interest (e.g., user defined features) and whilst the agent may be biased towards one or more features of interest, it could equally be considered to be biased away from features of low importance. Page 27, l.3-6, by optimsing based on the prediction error gradients (biasing the system towards areas that provide the greatest learning), the actions in the system are biased towards regions of novelty (e.g., regions with new features less observed with lighter uncertainty about their subsequent states) and/or changes in the environment (e.g., previously observed objects moving within the environment))
Referring to claim 8, Robinson discloses the method as in claim 1 further comprising: initiating the determined action. (page 25, I.31- page 26, l.4, “The agent can be trained via reinforcement learning. That is, each time an action is determined based on an input state and the action is applied to the environment (the configuration of the sensor is adapted), an updated state is received (updated sensor data) which is then used to determine a reward (based on a reward function) or, conversely, a loss (based on a loss function). The parameters of the agent are then updated to minimise the loss or maximise the reward. The remainder of the application discusses the use of a loss function. Having said this, a reward function may equally be used (for instance, by taking the inverse of the parameters that are included in the loss function). Accordingly, for the purposes of this application, the maximisation of a reward function is considered equivalent to the minimisation of a cost function.”)
Referring to claim 9, Robinson discloses the method as in claim 8 further comprising: obtaining updated values of the set of features after the action is performed; and using the values of the set of features, the determined action, the determined reward, and the updated values of the set of features as training data to train the model. (page 25, I.31- page 26, l.4, “The agent can be trained via reinforcement learning. That is, each time an action is determined based on an input state and the action is applied to the environment (the configuration of the sensor is adapted), an updated state is received (updated sensor data) which is then used to determine a reward (based on a reward function) or, conversely, a loss (based on a loss function). The parameters of the agent are then updated to minimise the loss or maximise the reward. The remainder of the application discusses the use of a loss function. Having said this, a reward function may equally be used (for instance, by taking the inverse of the parameters that are included in the loss function). Accordingly, for the purposes of this application, the maximisation of a reward function is considered equivalent to the minimisation of a cost function.”)
Referring to claim 10, Robinson discloses the method as in claim 1 wherein the method is performed by a node in a communications network and the set of features are obtained by the communications network. (page 31, 1.22-27 of Robinson, the various nodes within the system share information to allow for better decisions to be taken by each node. Each communication link within the network of nodes may be governed by dedicated communication modules within each node. There may … adjust the amount of data shared between the nodes)
Referring to claim 11, Robinson discloses the method as in claim 10 wherein the reinforcement learning agent is configured for adjustment of operational parameters of the communications network. (page 28, l. 25-31, “The amount of data that is transferred between nodes can be changed by changing the amount of compression performed by a node (e.g. the size of the latent space). To achieve this, the node receives a variable bandwidth hyperparameter 58. This defines an initial setting for the amount of data shared between the nodes. The variable bandwidth hyperparameter 68 may be chosen by the user in accordance with the technical criteria for the system (e.g. the latency requirements and transmission overheads within the system). The variable bandwidth hyperparameter 58 is input into the encoder 12, which generates the encoded sensor data 60. The decoder 14 determines, from the encoded sensor data 60, a bandwidth action 78. The bandwidth action 78 defines the bandwidth for the node (the amount of data to be sent to the parent node). The agent is configured to adjust the bandwidth (adjust the size of the latent space) based on a trained reinforcement learning model. The agent (via the decoder 14) outputs a bandwidth action 78. The bandwidth action 78 is an action on the bandwidth to adjust the bandwidth (adjust the size of the latent space).)
Referring to claim 12, Robinson discloses the method as in claim 10 wherein the reinforcement learning agent is for use in determining a tilt angle for an antenna in the communications network. (p. 11 I. 5-11 of Robinson: "the action comprises one or more of: adjusting a resolution of the corresponding sensor; adjusting a focus of the corresponding sensor; and directing the corresponding sensors to sense an updated region. Sensing an updated region may be via an actuator moving the sensor or the sensor configuring itself to adjust its sensitivity in a certain direction (e.g. the adjustment of a phased array of antennas))
Referring to claim 13, Robinson discloses the method as in claim 12 wherein: the set of features comprise signal quality, signal coverage and a current tilt angle of the antenna; the action comprises an adjustment to the current tilt angle of the antenna; and the reward is further based on a change in one or more key performance indicators related to the antenna, as a result of changing the tilt angle of the antenna according to the adjustment. (p. 11 I. 5-11 of Robinson: "the action comprises one or more of: adjusting a resolution of the corresponding sensor; adjusting a focus of the corresponding sensor; and directing the corresponding sensors to sense an updated region. Sensing an updated region may be via an actuator moving the sensor or the sensor configuring itself to adjust its sensitivity in a certain direction (e.g. the adjustment of a phased array of antennas))
Referring to claim 14, Robinson discloses the method as in claim 10 wherein the method is for use in determining movements of a mobile robot or autonomous vehicle receiving instructions through the communications network. (page 34, l.4-14, “The data acquired by these sensors can be used as training data for further improving autonomous control systems. Having said this, it would be inefficient and expensive to report back 100% of sensor data acquired by vehicles within a fleet. Not only would this cause a significant power and bandwidth drain on the vehicles, but it would also require unfeasibly large data storage capacity to collect. There is therefore a need to more effectively identify the data that is most useful for training autonomous vehicles.”)
Referring to claim 15, Robinson discloses the method as in claim 14 wherein: the set of features comprise sensor data from the mobile robot related to distances between the mobile robot and other objects surrounding the mobile robot; the action comprises sending an instruction to the mobile robot to instruct the mobile robot to perform a movement; and the reward is based on the changes to the distances between the mobile robot and other objects surrounding the mobile robot, as a result of the mobile robot performing the movement. (page 25, I.31- page 26, l.4, “The agent can be trained via reinforcement learning. That is, each time an action is determined based on an input state and the action is applied to the environment (the configuration of the sensor is adapted), an updated state is received (updated sensor data) which is then used to determine a reward (based on a reward function) or, conversely, a loss (based on a loss function). The parameters of the agent are then updated to minimise the loss or maximise the reward. The remainder of the application discusses the use of a loss function. Having said this, a reward function may equally be used (for instance, by taking the inverse of the parameters that are included in the loss function). Accordingly, for the purposes of this application, the maximisation of a reward function is considered equivalent to the minimisation of a cost function.” And page 34, l.4-14, “The data acquired by these sensors can be used as training data for further improving autonomous control systems. Having said this, it would be inefficient and expensive to report back 100% of sensor data acquired by vehicles within a fleet. Not only would this cause a significant power and bandwidth drain on the vehicles, but it would also require unfeasibly large data storage capacity to collect. There is therefore a need to more effectively identify the data that is most useful for training autonomous vehicles.”)
Referring to claim 16, Robinson discloses the method as in claim 1 wherein the method is performed by a mobile robot or autonomous vehicle and wherein the reinforcement learning agent is for use in determining movements of the mobile robot or autonomous vehicle. (page 34, l.4-14, “The data acquired by these sensors can be used as training data for further improving autonomous control systems. Having said this, it would be inefficient and expensive to report back 100% of sensor data acquired by vehicles within a fleet. Not only would this cause a significant power and bandwidth drain on the vehicles, but it would also require unfeasibly large data storage capacity to collect. There is therefore a need to more effectively identify the data that is most useful for training autonomous vehicles.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Robinson et al (WO 2020224910 A1) In view of “Explainability in Deep Reinforcement Learning,”, Heuillet et al, 12/18/2020.
Referring to claim 17, Robinson discloses the method as in claim 1. Robinson does not specifically discloses wherein the step of determining the first indication of the relative importance of the first feature is performed using an explainable artificial intelligence, XAI, process.
However, Heuillet discloses wherein the step of determining the first indication of the relative importance of the first feature is performed using an explainable artificial intelligence, XAI, process. (Abstract of Heuillet).
Robinson and Heuillet are analogous art because both references concern reinforcement learning interpretating actions. Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Robinson’s reinforcement learning with XAI type of reinforcement learning as taught by Heuillet. The motivation for doing so would have been to improve on reinforcement learning related to actions as feedback to learning of the model.
Referring to claim 18, Robinson discloses the method as in claim 1. Robinson does not specifically discloses wherein the reinforcement learning agent is a deep reinforcement learning agent.
However, Heuillet discloses wherein the reinforcement learning agent is a deep reinforcement learning agent. . (Abstract of Heuillet).
Robinson and Heuillet are analogous art because both references concern reinforcement learning interpretating actions. Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Robinson’s reinforcement learning with XAI type of reinforcement learning as taught by Heuillet. The motivation for doing so would have been to improve on reinforcement learning related to actions as feedback to learning of the model.
Referring to claim 19, Robinson discloses the method as in claim 1. Robinson does not specifically discloses wherein the model is a neural network, trained to take as input values of the set of features and output Q-values for possible actions that could be taken in the communications network.
However, Heuillet discloses herein the model is a neural network, trained to take as input values of the set of features and output Q-values for possible actions that could be taken in the communications network. (page 11 of Heuillet).
Robinson and Heuillet are analogous art because both references concern reinforcement learning interpretating actions. Accordingly, it would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Robinson’s reinforcement learning with XAI type of reinforcement learning as taught by Heuillet. The motivation for doing so would have been to improve on reinforcement learning related to actions as feedback to learning of the model.
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure:
Price (US 20220188623 A1): A policy based on a compound reward function is learned through a reinforcement learning algorithm at a learning network. The policy is used to choose an action of a plurality of possible actions. A state-action value network is established for each of the two or more reward terms. The state-action value networks are separated from the learning network. A human-understandable output is produced to explain why the action was taken based on each of the state action value networks.
Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
In the interests of compact prosecution, Applicant is invited to contact the examiner via electronic media pursuant to USPTO policy outlined MPEP § 502.03. All electronic communication must be authorized in writing. Applicant may wish to file an Internet Communications Authorization Form PTO/SB/439. Applicant may wish to request an interview using the Interview Practice website: http://;www.uspto.gov/patent/laws-and-regulations/interview-practice.
Applicant is reminded Internet e-mail may not be used for communication for matters under 35 U.S.C. § 132 or which otherwise require a signature. A reply to an Office action may NOT be communicated by Applicant to the USPTO via Internet e- mail. If such a reply is submitted by Applicant via Internet e-mail, a paper copy will be placed in the appropriate patent application file with an indication that the reply is NOT ENTERED. See MPEP § 502.03(II).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAIMEI JIANG whose telephone number is (571)270-1590. The examiner can normally be reached M-F 9-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela D Reyes can be reached at 571-270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAIMEI JIANG/Primary Examiner, Art Unit 2142
Prosecution Insights