DETAILED ACTION
This action is responsive to claims filed 09/12/2023. Claims 1, 8–13, and 19 have been amended. Claim 18 has been cancelled and there are no new claims.
Claims 1–17 and 19 are pending for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/12/2023 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered and attached by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title
Claims 1–17 and 19 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 1 is directed to a method i.e., a process.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
“estimating levels of variability of stochastic processes underlying the behavior of respective ones of the plurality of agents”
“selecting an agent having a lowest level of variability of its underlying stochastic process”
“select an action based on its updated local policy”
“generate an expected next state based on the selected action”
These limitations, under their broadest reasonable interpretation, cover mental processes, concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See MPEP 2106.04(a)(2). In particular, with the aid of pen and paper, a human can estimate levels of variability within agents performing statistical processes, select which agent they think is the lowest level, selected new actions for the agent, and generate and expected state based on the action.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A method of performing multi-agent reinforcement learning in a system including a plurality of agents that execute actions on an environment based on respective local policies of the agents, the method comprising” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
“instructing the selected agent to update its local policy, select an action based on its updated local policy, and generate an expected next state based on the selected action” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Updating agent policies, selecting new actions, and generating the next state merely invokes computers or other machinery as a tool to perform an existing process of reinforcement learning.
“repeatedly selecting a next agent having a next lowest level of variability of its underlying stochastic process and instructing the next selected agent to update its local policy based on previously generated expected next states of previously selected agents” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Repeatedly updating agent policies, selecting new actions, and generating the next state merely invokes computers or other machinery as a tool to perform an existing process of reinforcement learning.
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 2:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1 above). This claim merely recites an additional limitation which is directed to mere pre-solution activity of establishing inputs. The additional limitation:
“initially generating a random ranking of the plurality of agents” — This limitation is insignificant extra-solution activity and is merely pre-solution activity of establishing inputs. See MPEP 2106.05(g)(1).
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
“initially generating a random ranking of the plurality of agents” — This limitation is directed to the activity of insignificant extra-solution activity and is merely pre-solution activity of establishing inputs. See MPEP 2106.05(g)(1); See Bilski v. Kappos, 561 U.S. 593, 611-12, 95 USPQ2d 1001, 1010 (2010). This limitation is well-understood, routine, and conventional because it involves retrieving information from a stored list to initialize parameters. See MPEP 2106.05(d)(II); See Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
Thus, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception (see MPEP 2106.05(I.), failing step 2B.
Regarding claim 3:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 2 which included an abstract idea (see rejection for claim 2 above). This claim merely recites a further limitation on the multi-agent reinforcement learning in a system limitation which is directed to a generic computer component performing a generic computer function such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional limitation:
“instructing the agents to execute the selected actions” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Agents acting on instructions to execute selected actions merely invokes computers or other machinery as a tool to perform an existing process.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 4:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 4 depends from claim 3 (see analysis of claim 3 above) which is directed to a method i.e., a process.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
“determining actual next states of the agents after executing the selected actions”
“comparing the actual next states of the agents after executing the selected actions to the expected next states of the agents”
“generating a ranking of the agents by variability of their underlying stochastic processes based on the comparison of the actual next states of the agents after executing the selected actions to the expected next states of the agents”
These limitations, under their broadest reasonable interpretation, cover mental processes, concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See MPEP 2106.04(a)(2). In particular, with the aid of pen and paper, a human can determine the next steps of agents after they perform actions, compare the expected values, and generate a ranking based on the comparison.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application.
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 5:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 4 which included an abstract idea (see rejection for claim 4 above). This claim merely recites a further limitation on the multi-agent reinforcement learning in a system limitation which is directed to a generic computer component performing a generic computer function such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional limitations:
“for each agent, incrementing a counter when the expected next state of the agent matches the actual next state of the agent” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Incrementing a counter based on matching metrics merely invokes computers or other machinery as a tool to perform an existing process.
“wherein the ranking of the agents by variability of their underlying stochastic processes is based on values of their respective counters in ascending order” — These limitations are merely a continuation of the abstract idea in claim 1. Under their broadest reasonable interpretation, they cover mental processes, concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See MPEP 2106.04(a)(2). In particular, with aid of pen of paper, a human can generate a ranking based on a comparison wherein the comparison includes counters.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding claim 6:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 5 which included an abstract idea (see rejection for claim 5 above). This claim merely recites a further limitation on the multi-agent reinforcement learning in a system limitation which is directed to a generic computer component performing a generic computer function such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional limitation:
“normalizing the counter values by dividing the counter values by a number of elapsed time steps since the counters were started” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Normalizing data for use in neural networks merely invokes computers or other machinery as a tool to perform an existing process.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding claim 7:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 1 which included an abstract idea (see rejection for claim 1 above). This claim merely recites a further limitation on the multi-agent reinforcement learning in a system limitation which is directed to a generic computer component performing a generic computer function such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional limitation:
“iteratively updating a ranking of the agents based on variability of their underlying stochastic processes sequentially updating local policies of the agents” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Iteratively updating data for use in neural networks merely invokes computers or other machinery as a tool to perform an existing process.
“simultaneously executing selected actions based on the updated local policies until the ranking of agents does not change between successive iterations” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Executing agent processes in parallel in a reinforcement learning scheme merely invokes computers or other machinery as a tool to perform an existing process.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding claim 8:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 5 which included an abstract idea (see rejection for claim 5 above). This claim merely recites a further limitation on the multi-agent reinforcement learning in a system limitation which is directed to a generic computer component performing a generic computer function such that it amounts no more than mere instructions to apply the exception using a generic computer component. The additional limitation:
“wherein the underlying stochastic processes of the agents comprise Markov Decision Processes” — This limitation is directed to the field of use (see MPEP 2106.05(h)) as it merely limiting the fields of the stochastic processes to that of Markov Decision processes.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding Claim 9:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 9 is directed to a master node i.e., a machine.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
This claim is directed to machinery to perform the method of claim 1. Therefore, this claim inherently includes all the limitations (including the abstract ideas) of claim 1.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A master node for controlling multi-agent reinforcement learning configured to perform operations according to Claim 1” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 10:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 10 is directed to a master node i.e., a machine.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
This claim is directed to machinery to perform the method of claim 1. Therefore, this claim inherently includes all the limitations (including the abstract ideas) of claim 1.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A master node for controlling multi-agent reinforcement learning, comprising: a processing circuit; and a memory coupled to the processing circuit, wherein the memory comprises computer readable program instructions that, when executed by the processing circuit, cause the computing device to perform operations according to Claim 1” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 11:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 11 is directed to a computer program i.e., a method.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
This claim is directed to a computer program executed on machinery to perform the method of claim 1. Therefore, this claim inherently includes all the limitations (including the abstract ideas) of claim 1.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A computer program comprising program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according to Claim 1” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 12:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 12 is directed to a computer program product comprising a non-transitory storage medium i.e., a machine.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
This claim is directed to machinery to perform the method of claim 1. Therefore, this claim inherently includes all the limitations (including the abstract ideas) of claim 1.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according to Claim 1” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding Claim 13:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 13 is directed to a method i.e., a process.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
“generating a ranking of the plurality of agents based on levels of variability of stochastic processes underlying the behavior of respective ones of the plurality of agents;”
“sequentially selecting the agents in order based on the ranking and updating their local policies, wherein the local policy of a selected agent is updated conditioned on an expected next state of at least one previously selected agent”
These limitations, under their broadest reasonable interpretation, cover mental processes, concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See MPEP 2106.04(a)(2). In particular, with the aid of pen and paper, a human can generate a ranking for the agents based on stochastic processes and can select agents in order of the ranking among other requirements.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A method of performing multi-agent reinforcement learning in a system including a master node and a plurality of agents that execute actions on an environment based on respective local policies of the agents, the method comprising” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
“simultaneously (410) executing actions by agents based on their updated local policies” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Executing agent processes in parallel in a reinforcement learning scheme merely invokes computers or other machinery as a tool to perform an existing process.
“updating (412) the ranking of the plurality of agents in response to executing the actions” — This limitation amounts to no more than mere instructions to apply the exception and is the equivalent to mere instruction to implement the abstract idea on a computer. See MPEP 2106.05(f). Updating agent rankings in reinforcement learning merely invokes computers or other machinery as a tool to perform an existing process of reinforcement learning.
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Regarding claims 14–15, although varying in scope, the limitations of claims 14–15 are substantially the same as the limitations of claims 5–6, respectively. Thus, claims 14–15 are rejected using the same reasoning and analysis as claims 5–6 above, respectively.
Regarding claims 16:
The claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim is dependent on claim 13 which included an abstract idea (see rejection for claim 13 above). This claim merely recites a further limitation on the sequentially selecting limitation which is directed to an abstract idea that can be performed in the human mind. The additional limitations:
“wherein updating the local policy of an agent comprises selecting an action based on an updated local policy of the agent, and generating an expected next state based on the selected action” — These limitations are merely a continuation of the abstract idea in claim 13. Under their broadest reasonable interpretation, they cover mental processes, concepts performed in the human mind (including an observation, evaluation, judgment, opinion). See MPEP 2106.04(a)(2). In particular, with aid of pen of paper, a human can sequentially select agents in order based on an updated policy and can generate expected next states based on actions.
Thus, the judicial exception is not integrated into a practical application (see MPEP 2106.04(d)I.), failing Step 2A Prong 2. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception under step 2B.
Regarding claims 17, although varying in scope, the limitations of claim 17 are substantially the same as the limitations of claim 2. Thus, claim 17 is rejected using the same reasoning and analysis as claim 2 above.
Regarding Claim 19:
Step 1 — Is the claim to a process, machine, manufacture, or composition of matter?
Yes, claim 19 is directed to a computer program product comprising a non-transitory storage medium i.e., a machine.
Step 2A — Prong 1 Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
This claim is directed to machinery to perform the method of claim 13. Therefore, this claim inherently includes all the limitations (including the abstract ideas) of claim 13.
Step 2A — Prong 2 — Does the claim recite additional elements that integrate the judicial exception into a practical application?
No, there are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according to Claim 13” — This limitation is reciting generic computer components at a high-level of generality (i.e., as a generic computer component performing a generic computer function) such that it amounts no more than mere instructions to apply the exception using a generic computer component. See MPEP 2106.05(f).
Step 2B — Does the claim recite additional elements that amount to significantly more than the judicial exception?
No, there are no additional elements that amount to significantly more than the judicial exception.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Duan et al., (US 20180373997 A1), hereinafter “Duan”, in view of Gu et al., (US 20170228662 A1), hereinafter “Gu”.
Regarding claim 1, Duan teaches:
A method of performing multi-agent reinforcement learning in a system including a plurality of agents that execute actions on an environment based on respective local policies of the agents, the method comprising (Duan ¶Abstract, ¶0034: “the method begins with operating a reinforcement learning model using a state-action table with a set of environment states, a set of software agent states of at least one software agent, a set of actions corresponding to the set of environmental states and software agent states, a plurality of policies of transitioning from the environmental states and software agent states to actions, rules that determine a scalar immediate reward based on the transitioning, and rules that describe what the at least one software agent observes. An unstable state is identified from a series of values of the set of actions in the state-action table in which the series of values differ from each other by a settable threshold. Policies or factors are selected to split the unstable state that has been identified” and “The ‘reinforcement learning model’ includes a set of environment and software agent states, a set of actions of the software agent, policies of transitioning from the environmental and software agent states to actions, rules that determine the scalar immediate reward based on the transitioning and rules that describe what the software agent observes”):
estimating levels of variability of stochastic processes underlying the behavior of respective ones of the plurality of agents (Duan Figs. 5–6, ¶0044, ¶0053: “The score of open window is varying after each trip, i.e. trip 2, trip 3, trip 4, trip 5, and trip 6, as shown in table 420. An unstable state is identified by detecting this variation or vibration in the score” and “Shown is an evaluation of whether the new updated score in the state-action table 610 based on feedback from the vehicle/sensors is stable 620. The output is one of stable meaning the same value from a series of values of the set of actions in the state-action table 520 in which the series of values are within a certain range from each other by a settable threshold or differ from each other by a settable threshold which mean “vibrating” or unstable. Another output possible is a middle status that is not stable or not unstable according to the thresholds”—[wherein the convergence evaluator estimates variability by evaluating whether action scores vary, vibrate, or remains stable over feedback iterations]); and
selecting an agent having a lowest level of variability of its underlying stochastic process (Duan Fig. 5, ¶0048–0052: “convergence evaluation 554 which identifies unstable or stable states from a series of value of the set action in the state action table 520 … if a stable state is identified, the state merger is processed in step 560 for identified stable state in the state action table 520”—[wherein a stable state has the lowest variability because its action-value series remains within the settable threshold]);
instructing the selected agent to update its local policy, select an action based on its updated local policy, [and generate an expected next state based on the selected action] (Duan Figs. 3, 5, ¶0039–0043, ¶0047: “FIG. 3 illustrates an update to a state-action table 300 based on user-feedback, according to an embodiment of the present invention. The states and actions change from a time T1 on the left to a time T2 on the right. More specifically, shown are rows of states 310 and the corresponding actions 350. Action scores in the table on the right for T2 are adjusted based on user feedback and/or sensor readings. For example: After taken on action, adjust the score based on user feedback and sensor readings the Air quality function Func(inPM25, inTVOC, inHumidity, . . . ) provides: If the value of Func( ) increases, the action has positive effect, then increase its score. If the value of Func( ) doesn't change or decreases, decrease its score. After decreasing the score of “open window” by 20, “inner loop” becomes the chosen action of this state” and “Also in the action generator 530 is an action selector 534. Once the current state which the software agent is in is determined, the Action Selection selects which action to take. Normally, the action with highest reward will be selected. An Action Selection notification is sent to automotive 540 and feedback received by score refiner 552.”—[wherein action score is updated in the state-action table and the action is selected based on the updated state-action scores/rewards which are defined in the local policy]); and
repeatedly selecting a next agent having a next lowest level of variability of its underlying stochastic process and instructing the next selected agent to update its local policy [based on previously generated expected next states of previously selected agents, select an action based on its updated local policy, and generate an expected next state based on the selected action] (Duan Fig. 5, ¶0049–0052: “In the case that an unstable state has been identified, the convergence evaluator feeds into a factor selector 568 and state splitter 560 for state-action table 520. One or more policies or factors are used to split the unstable state that has been identified. Examples of policies include regression model, a Pearson correlation coefficient, or mutual information between rows of the state-action table … On the other hand if a stable state is identified, the state merger is processed in step 560 for identified stable state in the state action table 520”—[wherein state stability is evaluated repeatedly and updates the state-action table]).
Duan does not appear to explicitly teach:
[instructing the selected agent to update its local policy, select an action based on its updated local policy,] and generate an expected next state based on the selected action;
[repeatedly selecting a next agent having a next lowest level of variability of its underlying stochastic process and instructing the next selected agent to update its local policy] based on previously generated expected next states of previously selected agents, select an action based on its updated local policy, and generate an expected next state based on the selected action.
However, Gu teaches:
generate an expected next state based on the selected action (Gu Fig. 3, ¶0052: “The system processes the particular observation and the selected action using a state transition model (330) to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selection action in response to the particular observation. In other words, the system uses the state transition model to create an imaginary trajectory for the agent”—[wherein the state-transition model generates the expected next state based on the action]);
selecting an agent having a lowest level of variability of its underlying stochastic process (Gu ¶: “”—[wherein]); and
based on previously generated expected next states of previously selected agents, select an action based on its updated local policy, and generate an expected next state based on the selected action (Gu Figs. 1–4, ¶0026, ¶0052, ¶0059–0065: “The policy subnetwork 112 is a neural network that is configured to receive the observation 105 and process the observation 105 to generate an ideal point 122 in the continuous space of actions. The ideal point 122 represents an action that, if performed in response to the observation, is expected to produce a maximum Q value” and “The system processes the particular observation and the selected action using a state transition model (330) to determine a next observation characterizing a next state” and “The experience tuple includes (1) a training observation that characterizes a training state of the environment, (2) an action performed by the agent in response to the training observation, (3) a reward received as a result of the agent performing the action in response to the training observation, and (4) a subsequent observation characterizing a subsequent state of the environment … The system determines an update to the current values of the parameters of subnetworks of the reinforcement learning system using an error between the Q value for the particular action and the target Q value (480)”—[wherein selecting an action from an updated policy network and using a state transition model to generate the expected next state for that action is based on prior generated/observed subsequent states in the experience tuples updating policy parameters]).
The methods of Duan, the teachings of Gu, and the instant application are analogous art because they pertain to reinforcement learning using agent policies.
It would be obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the methods of Duan with the teachings of Gu to provide expected next state generation with agent and state selection regarding policy updates. One would be motivated to do so to apply reinforcement learning techniques for prepetition and policy updates to the state-action/policy framework to improve adjustment based on feedback and stability (Gu ¶0011: “Accuracy and efficiency of trained reinforcement learning systems can be improved by providing the reinforcement learning system with a subsystem that allows actions other than an optimal action to be selected by the currently trained system”).
Regarding claim 2, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
initially generating a random ranking of the plurality of agents (Duan Fig. 5, ¶0046: “The software again may choose any action as a function of the history or can randomize is action selection”—[wherein the software chooses to randomize the action selection process i.e., randomize the ranking]).
Regarding claim 3, Duan in view of Gu teaches all the limitations of claim 2.
Duan teaches:
instructing the agents to execute the selected actions (Duan Fig. 5, ¶0047–0048: “Also in the action generator 530 is an action selector 534. Once the current state which the software agent is in is determined, the Action Selection selects which action to take. Normally, the action with highest reward will be selected. An Action Selection notification is sent to automotive 540 and feedback received by score refiner 552. A feedback controller 550 includes a score refiner 552. The score refiner or reward refiner is used to review the score or reward of action based on the feedback (results) after the action is taken”—[wherein selecting, sending notifications, and incorporating feedback is instructing agents to execute the selected action]).
Regarding claim 4, Duan in view of Gu teaches all the limitations of claim 3.
Gu teaches:
determining actual next states of the agents after executing the selected actions (Gu Fig. 4, ¶0059: “The experience tuple includes (1) a training observation that characterizes a training state of the environment, (2) an action performed by the agent in response to the training observation, (3) a reward received as a result of the agent performing the action in response to the training observation, and (4) a subsequent observation characterizing a subsequent state of the environment”);
comparing the actual next states of the agents after executing the selected actions to the expected next states of the agents (Gu Fig. 4, ¶0063–0065: “generate a new value estimate for the subsequent state (460). The new value estimate for the new subsequent state is an estimate of an expected return resulting from the environment being in the subsequent state … The system combines the reward and the new value estimate to generate a target Q value … determines an update to the current values of the parameters of subnetworks of the reinforcement learning system using an error between the Q value for the particular action and the target Q value (480)”—[wherein the predicted /expected value information is compared to target value information derived from the actual subsequent state]); and
Duan teaches:
generating a ranking of the agents by variability of their underlying stochastic processes based on the comparison of the actual next states of the agents after executing the selected actions to the expected next states of the agents (Duan Figs. 5, 6, ¶0048–0053: “A feedback controller 550 includes a score refiner 552. The score refiner or reward refiner is used to review the score or reward of action based on the feedback (results) after the action is taken. Also includes in the feedback controller 550 is a convergence evaluation 554 which identifies unstable or stable states from a series of value of the set action in the state action table 520. In the case that an unstable state has been identified, the convergence evaluator feeds into a factor selector 568 and state splitter 560 for state-action table 520. One or more policies or factors are used to split the unstable state that has been identified. Examples of policies include regression model, a Pearson correlation coefficient, or mutual information between rows of the state-action table. In one embodiment, the selection of the unstable state to split is based upon the one or more polices or factors with a high correlation between a numerical value of the policies or factors and a score adjustment trend. In another embodiment, the selection of the unstable state to split is based upon at least one categorical value for the policies or factors with a low correlation between the categorical value and a value for stableness. On the other hand if a stable state is identified, the state merger is processed in step 560 for identified stable state in the state action table 520. FIG. 6 is a state-action table of convergence evaluator 554 of FIG. 5, according to an embodiment of the present invention. Shown is an evaluation of whether the new updated score in the state-action table 610 based on feedback from the vehicle/sensors is stable 620. The output is one of stable meaning the same value from a series of values of the set of actions in the state-action table 520 in which the series of values are within a certain range from each other by a settable threshold or differ from each other by a settable threshold which mean “vibrating” or unstable. Another output possible is a middle status that is not stable or not unstable according to the thresholds”—[wherein the updates include stability/instability determinations (i.e., rankings) based on feedback from action values]).
The same motivation that was utilized for combining Duan with Gu, as set forth in claim 1, is equally applicable to claim 4.
Regarding claim 5, Duan in view of Gu teaches all the limitations of claim 4.
Duan teaches:
for each agent, incrementing a counter when the expected next state of the agent matches the actual next state of the agent; wherein the ranking of the agents by variability of their underlying stochastic processes is based on values of their respective counters in ascending order (Duan Figs. 4, 6, ¶0044, ¶0053: “FIG. 4 illustrates identifying unstable state and splitting the state into more than one state. The score of open window is varying after each trip, i.e. trip 2, trip 3, trip 4, trip 5, and trip 6, as shown in table 420. An unstable state is identified by detecting this variation or vibration in the score. Once this variation is detected, the row for the unstable state is spit into “N” or more rows as shown in table 430. In this example, the single row of table 410 that was unstable is divided into four rows in table 430 as shown. In this example, infinity is 650 and the domain is uniformly divided into four (4) regions. It is important to note that other methods for dividing up a range of numbers may be used including using exponent and logarithmic scales”—[wherein the system repeatedly tracks stable/unstable results over a series of action values by incrementing a counter (e.g., trip 2, trip 3, trip 4, trip 5, and trip 600)])
Regarding claim 6, Duan in view of Gu teaches all the limitations of claim 5.
Gu teaches:
normalizing the counter values by dividing the counter values by a number of elapsed time steps since the counters were started (Gu ¶0025, ¶0028: “The expected return is the time-discounted total future reward resulting from the environment being in the state characterized by the observation” and “the reinforcement learning system 100 can select the action represented by the ideal point as the action to be performed by the agent with probability 1-ε and select a random action with probability ε. As another example, the system can sample a point from a noise distribution and then select an action that is represented by the point that is equal to (the sampled point+the ideal point)”—[wherein the system normalizes the expected return by time-discounting total future rewards creating a distribution]).
The same motivation that was utilized for combining Duan with Gu, as set forth in claim 1, is equally applicable to claim 6.
Regarding claim 7, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
iteratively updating a ranking of the agents based on variability of their underlying stochastic processes sequentially updating local policies of the agents (Duan Figs. 4–5, ¶0039–0043, ¶0048: “More specifically, shown are rows of states 310 and the corresponding actions 350. Action scores in the table on the right for T2 are adjusted based on user feedback and/or sensor readings For example: After taken on action, adjust the score based on user feedback and sensor readings the Air quality function Func(inPM25, inTVOC, inHumidity, . . . ) provides: If the value of Func( ) increases, the action has positive effect, then increase its score. If the value of Func( ) doesn't change or decreases, decrease its score. After decreasing the score of “open window” by 20, “inner loop” becomes the chosen action of this state” and “A feedback controller 550 includes a score refiner 552. The score refiner or reward refiner is used to review the score or reward of action based on the feedback (results) after the action is taken. Also includes in the feedback controller 550 is a convergence evaluation 554 which identifies unstable or stable states from a series of value of the set action in the state action table 520”—[wherein the updates, including scores (i.e., rankings), are processed iteratively based on feedback to update local policies]), and
simultaneously executing selected actions based on the updated local policies until the ranking of agents does not change between successive iterations (Duan Figs. 5–6, ¶0051–0053: “In another embodiment, the selection of the unstable state to split is based upon at least one categorical value for the policies or factors with a low correlation between the categorical value and a value for stableness. On the other hand if a stable state is identified, the state merger is processed in step 560 for identified stable state in the state action table 520. FIG. 6 is a state-action table of convergence evaluator 554 of FIG. 5, according to an embodiment of the present invention. Shown is an evaluation of whether the new updated score in the state-action table 610 based on feedback from the vehicle/sensors is stable 620. The output is one of stable meaning the same value from a series of values of the set of actions in the state-action table 520 in which the series of values are within a certain range from each other by a settable threshold or differ from each other by a settable threshold which mean ‘vibrating’ or unstable. Another output possible is a middle status that is not stable or not unstable according to the thresholds”]).
Regarding claim 8, Duan in view of Gu teaches all the limitations of claim 5.
Gu teaches:
wherein the underlying stochastic processes of the agents comprise Markov Decision Processes (Gu ¶0018, ¶0052–0059: “the system receives data characterizing the current state of the environment and selects an action from a predetermined set of actions to be performed by the agent in response to the received data. Data characterizing a state of the environment will be referred to in this specification as an observation” and “The system processes the particular observation and the selected action using a state transition model (330) to determine a next observation characterizing a next state that the environment would have transitioned into if the agent had performed the selection action in response to the particular observation … The system obtains an experience tuple (410). The experience tuple includes (1) a training observation that characterizes a training state of the environment, (2) an action performed by the agent in response to the training observation, (3) a reward received as a result of the agent performing the action in response to the training observation, and (4) a subsequent observation characterizing a subsequent state of the environment”—[wherein Gu teaches states/observations, actions, rewards, subsequent state, and a state-transition model i.e., Markov Decision Process]).
The same motivation that was utilized for combining Duan with Gu, as set forth in claim 1, is equally applicable to claim 8.
Regarding claim 9, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
a master node for controlling multi-agent reinforcement learning configured to perform operations according to Claim 1 (Duan Fig. 11, ¶0046–0048, ¶0061: “FIG. 11 illustrates one example of a processing node 1100, in accordance with an embodiment the present invention. This example is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein and the processing node 1100 is capable of being implemented and/or performing any one or more of the functionalities set forth herein”]).
Regarding claim 10, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
a master node for controlling multi-agent reinforcement learning, comprising: a processing circuit; and a memory coupled to the processing circuit, wherein the memory comprises computer readable program instructions that, when executed by the processing circuit, cause the computing device to perform operations according to Claim 1. (Duan Fig. 11, ¶0061–0075: “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.”
Regarding claim 11, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
A computer program comprising program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according claim 1. (Duan Fig. 11, ¶0061–0075: “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.”
Regarding claim 12, Duan in view of Gu teaches all the limitations of claim 1.
Duan teaches:
computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according to claim 1. (Duan Fig. 11, ¶0061–0075: “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.”
Regarding claims 13–17, although varying in scope, the limitations of claims 13–17 are substantially the same as the limitations of claims 1–2 and 5–7, respectively. Thus, claims 13–17 are rejected using the same reasoning and analysis as claim 1–2 and 5–7 above.
Regarding claim 19, Duan in view of Gu teaches all the limitations of claim 13.
Duan teaches:
A computer program product comprising a non-transitory storage medium including program code to be executed by processing circuitry of a computing device, whereby execution of the program code causes the computing device to perform operations according to claim 13. (Duan Fig. 11, ¶0061–0075: “The present invention may be a system, a method, and/or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.”
Prior Art of Record
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Gendron-Bellemare et al., (“Training action selection neural networks”) discloses reinforcement learning methods including agent policies and actions “The policy neural network is used to select actions to be performed by an agent that interacts with an environment by receiving an observation characterizing a state of the environment and performing an action from a set of actions in response to the received observation. A trajectory is obtained from a replay memory, and a final update to current values of the policy network parameters is determined for each training observation in the trajectory. The final updates to the current values of the policy network parameters are determined from selected action updates and leave-one-out updates.” Gendron-Bellemare ¶Abstract.
Du et al., (“Stochastic Variance Reduction Methods for Policy Evaluation”) discloses stochastic reinforcement learning methods “Policy evaluation is concerned with estimating the value function that predicts long-term values of states under a given policy. It is a crucial step in many reinforcement-learning algorithms. In this paper, we focus on policy evaluation with linear function approximation over a fixed dataset. We first transform the empirical policy evaluation problem into a (quadratic) convex-concave saddle-point problem, and then present a primal-dual batch gradient method, as well as two stochastic variance reduction methods for solving the problem. These algorithms scale linearly in both sample size and feature dimension. Moreover, they achieve linear convergence even when the saddle-point problem has only strong concavity in the dual variables but no strong convexity in the primal variables. Numerical experiments on benchmark problems demonstrate the effectiveness of our methods.” Du Abstract.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NICHOLAS SHINE whose telephone number is (571)272-2512. The examiner can normally be reached M-F, 11a-7p ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached on (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.B.S./Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights