DETAILED ACTION
Claims 1-20 are presented for examination.
This office action is in response to submission of application on 05-JANUARY-2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01-FEBRUARY-2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendment
The amendment filed 05-JANUARY-2026 in response to the non-final office action mailed 03-SEPTEMBER-2025 has been entered. Claims 1-20 remain pending in the application.
With regards to the non-final office action’s rejection under 101, the amendments to the claims have overcome the original rejection with regards to the claims being directed towards non-statutory subject matter.
With regards to the non-final office action’s rejections under 103, the amendments to the claims necessitated a new consideration of the art. After this consideration, the examiner respectfully disagrees with the applicant’s arguments that the art referenced in the previous office action does not teach the amendment claim limitations. A new 103 rejection over the prior art has been provided.
Regarding the amended limitation of claim 1, the applicant argues that Hu does not teach “providing, to each of the plurality of simulated local agents, information representing a global state of the agent network, wherein the information representing a global state of the agent network indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from the global goal.” The applicant argues that Hu instead “describes an ‘updated global policy’ ‘to which each agent adheres when each RL agent interacts in the environment’ […] rather than providing, to each of the plurality of simulated local agents, information that indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from a global goal” (Applicant Arguments, page 2).
However, Hu teaches that there may be discrepancies between an expected global policy output and a local experience output (Paragraph 49). The expected global policy output is an indication of a deviation of the local policy of the simulated agent from a global goal as the global policy output is information representing a global state of the agent network by way of its current policy outputs, and represents the global goal and allows for comparison of the local experience output to it. This comparison provides a deviation of the local policy from a global goal which is used to assist in training both the local and global models.
Analogous limitations are amended to claims 15 and 18. For the same reasoning, Hu in view of Mankowitz is believed to discloses those limitations as well, and claims 15 and 18 are rejected under similar rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-13, and 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over Hu et al. (Pub. No. US 20230214725 A1, filed December 31st 2021, hereinafter Hu) in view of Mankowitz et al. (Pub. No. WO 2020254400 A1, filed June 17th 2020, hereinafter Mankowitz).
Regarding claim 1:
Claim 1 recites:
A computer-implemented method comprising: generating, for each of a plurality of simulated local agents in an agent network in which the plurality of simulated local agents share resources, information, or both, a plurality of experience tuples comprising a state for the simulated local agent, an action taken by the simulated local agent, and a local result for the action taken; updating each local policy of each simulated local agent according to the respective local result generated from the action taken by the simulated local agent; providing, to each of the plurality of simulated local agents, the information representing a global state of the agent network, wherein the information representing a global state of the agent network indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from a global goal; and updating each local policy of each simulated local agent according to the global state of the agent network.
Hu discloses generating, for each of a plurality of simulated local agents in an agent network in which the plurality of simulated local agents share resources, information, or both:
Hu teaches individual agents in an environment that must communicate with each or with a central entity their individual observations (Paragraph 6). The plurality of agents would be the simulated local agents in an agent network as the central entity that they are all connected to would be the agent network. Furthermore, their observations (observed states) would be information that is shared through communication between them.
Hu discloses providing, to each of the plurality of simulated local agents, information representing a global state of the agent network:
Hu teaches that a global policy is distributed to agents (Paragraph 17). A global policy is information representing a global state of the agent network as it describes the overall current goals and reasoning in light of the current environment.
Hu discloses updating each local policy of each simulated local agent according to the global state of the agent network:
Hu teaches that based on the global policy as described above, each of the agent behave at least in part based on the updated global policy (Paragraph 17). This would indicate that the local policy of each simulated agent is determined by the global state of the agent network.
Hu discloses wherein the information representing a global state of the agent network indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from a global goal:
Hu teaches that there may be discrepancies between an expected global policy output and a local experience output (Paragraph 49). The expected global policy output is an indication of a deviation of the local policy of the simulated agent from a global goal as the global policy output is information representing a global state of the agent network by way of its current policy outputs, and represents the global goal and allows for comparison of the local experience output to it. This comparison provides a deviation of the local policy from a global goal which is used to assist in training both the local and global models.
Mankowitz discloses a plurality of experience tuples comprising a state for the simulated local agent, an action taken by the simulated local agent, and a local result for the action taken:
Mankowitz in the same field of endeavor of machine learning teaches experience tuples that comprise an observation, an action, and a reward (which is generated based on the result) (Page 4, lines 5-10). These would correspond respectively to a state for the simulated local agent as an observation may describe the agent or the environment around it, the action taken, and a local result as the reward would be a result of the action taken as it directly corresponds to the action being performed.
Mankowitz and the present application are analogous art because they are in the same field of endeavor of machine learning.
Mankowitz discloses updating each local policy of each simulated local agent according to the respective local result generated from the action taken by the simulated local agent:
Mankowitz teaches that from perturbations of the states of the first environment (wherein the perturbation may be a respective local result generated from the action taken by the simulated local agent, (Page 4, lines 15-25)) the current values of Q network parameters may be updated, wherein the Q network is used to determine an update to the policy network for the local agent (Page 4, lines 5-15). Since the Q network which uses the local result updates each local policy through the policy network, then the update is according to the local result.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 2, which depends upon claim 1:
Claim 2 recites:
The computer-implemented method of claim 1, wherein generating the plurality of experience tuples comprises varying amounts of information provided to each of the plurality of simulated local agents
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 2 depends. However, Hu does not teach the limitation of claim 2:
Mankowitz teaches an observation characterizing a current state of the environment to be include in the experience tuple (Page 3, line 25 – Page 4, line 5). An observation is not a set amount of information, and as such the amount of information provided to each of the plurality of simulated local agents may vary.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 3, which depends upon claim 1:
Claim 3 recites:
The computer-implemented method of claim 1, wherein generating the plurality of experience tuples comprises varying the actions taken between one or more of the plurality of simulated local agents in the agent network
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 3 depends. Furthermore, Mankowitz discloses the limitations of claim 3:
Mankowitz teaches generating perturbed states based on possible actions (Page 19, lines 15-20) wherein the possible actions would be actions taken between one or more of the plurality of simulated local agents in the agent network. As the actions are possible, this indicates the ability to vary actions rather than having a single set action.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 4, which depends upon claim 1:
Claim 4 recites:
The computer-implemented method of claim 1, further comprising receiving, from a global critic network and for each of the plurality of simulated local agents in the agent network, local-agent-specific information about an action that should have been taken by the simulated local agent
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 4 depends. Furthermore, Mankowitz discloses the limitations of claim 4:
Mankowitz teaches using critic reinforcement learning to determine the optimal action, which would be an action that should have been taken by the simulated local agent. Furthermore, this critic network is the Q-value neural network, wherein it is used to update the policy neural network which distributes updates to the agents (Page 11, lines 5-20). Therefore, the Q-value neural network would be the global critic network that the policy network uses to determine for each of the plurality of simulated local agents in the agent network, local-agent-specific information about the optimal action.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 5, which depends upon claim 4:
Claim 5 recites:
The computer-implemented method of claim 4, further comprising: providing, to each of the plurality of simulated local agents, the local-agent- specific information; and updating each local policy of each simulated local agent according to the local- agent-specific information
Hu in view of Mankowitz disclose the method of claim 4 upon which claim 5 depends. Furthermore, Mankowitz discloses the limitations of claim 5:
Mankowitz teaches that the policy neural network provides the agent with a policy, or an action to take (Page 7, lines 10-20) which would be local-agent-specific information as the action is for that agent. Furthermore, the policy neural network that provides the local policy is updated by the optimal action, as describes above in claim 4 (Page 11, lines 5-20).
Mankowitz does not teach that this is done with a plurality of simulated local agents. However, Hu teaches multi-agent reinforcement learning (Paragraph 5), and in combination discloses the limitation of claim 5.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 6, which depends upon claim 1:
Claim 6 recites:
The computer-implemented method of claim 1, wherein each of the plurality of simulated local agents comprises a global state estimator, the simulated local agent configured to: periodically receive the information representing the global state of the agent network; process, by the global state estimator, the received information to determine global state information; and update the local policy of the simulated local agent according to the global state information of the global state estimator
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 6 depends. Furthermore, Mankowitz discloses wherein each of the plurality of simulated local agents comprises a global state estimator, the simulated local agent configured to: periodically receive the information representing the global state of the agent network:
Mankowitz teaches that observations may include the global pose of one or more part of an agent (Page 7, lines 25-30). As observations are periodically received by the local agent, this would be periodically received information representing the global state of the agent network as pose would be a state.
Mankowitz discloses process, by the global state estimator, the received information to determine global state information:
Mankowitz teaches determining the global pose of one or more part of an agent as the observation (Page 7, lines 25-30), wherein the pose would be global state information and the process of observation would be an estimation of the agent’s surrounding and environment, or a state estimation.
Finally, regarding the limitation update the local policy of the simulated local agent according to the global state information of the global state estimator:
Mankowitz teaches that the above-described observation that is the global state information of the global state estimator may be used as an input into a policy neural network which provides the agent with a policy to perform a further action (Page 7, lines 15-20). A further action after the initial one would result from an updated policy (as opposed to an initial one).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 7, which depends upon claim 1:
Claim 7 recites:
The computer-implemented method of claim 1, wherein the action taken is a quantity of resources that the simulated local agent has available
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 7 depends. Furthermore, Mankowitz discloses the limitations of claim 7:
Mankowitz teaches that when the agent is a mechanical agent, the actions can include force/torque/acceleration for one or more joints of the agent (Page 9, lines 5-10). These would be descriptions of the quantity of resources that the local agent has available in the form of actual power that can be exerted.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Regarding claim 9, which depends upon claim 1:
Claim 9 recites:
The computer-implemented method of claim 1, wherein the action taken is information shared by the simulated local agent with at least one of the plurality of simulated local agents in the agent network
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 9 depends. Furthermore, Hu discloses the limitations of claim 9:
Hu teaches that a shared experience replay buffer that stored experience tuples is shared by multiple agents throughout an episode, wherein an episode would include multiple steps including actions (Paragraph 10). The stored experience tuples in the replay buffer would be a form of information that is shared by the simulated local agents with at least one of the plurality of simulated local agents in the agent network as the experience tuples come from individual agents, and the replay buffer is shared with every agent.
Regarding claim 10, which depends upon claim 9:
Claim 10 recites:
The computer-implemented method of claim 9, wherein the information is shared locally with a subset of simulated local agents in the plurality of simulated local agents
Hu in view of Mankowitz disclose the method of claim 9 upon which claim 10 depends. Furthermore, Hu discloses the limitations of claim 10:
Hu teaches that a shared experience replay buffer that stored experience tuples is shared by multiple agents throughout an episode, wherein an episode would include multiple steps including actions (Paragraph 10). The stored experience tuples in the replay buffer would be a form of information that is shared by the simulated local agents. Furthermore, this would be a subset of simulated local agents as the information that is shared by an agent would be shared with every agent that is not itself, and therefore the set of agents that has received information from that agent is a subset of the plurality.
Regarding claim 11, which depends upon claim 9:
Claim 11 recites:
The computer-implemented method of claim 9, wherein the information is shared globally with the plurality of simulated local agents in the agent network
Hu in view of Mankowitz disclose the method of claim 9 upon which claim 11 depends. Furthermore, Hu discloses the limitations of claim 11:
Hu teaches that the plurality of agents may receive a global policy distributed to agents (Paragraph 89), and as a global policy is a form of information, this would be information shared globally with the plurality of simulated local agents in the agent network.
Regarding claim 12, which depends upon claim 9:
Claim 12 recites:
The computer-implemented method of claim 9, wherein the information includes at least one of (i) a type of resource-related information of the simulated local agent and (ii) a threshold quantity of the resource-related information of the simulated local agent.
Hu in view of Mankowitz disclose the method of claim 9 upon which claim 12 depends. Furthermore, Hu discloses the limitations of claim 12:
Hu teaches that the agents may be used in management of network congestion Paragraph 80) wherein information by the agents would include control of transmission rates, wherein the resource-related information of an agent would be the transmission rate and the threshold quantity would be the total rate available (Paragraph 82).
Regarding claim 13, which depends upon claim 1:
Claim 13 recites:
The computer-implemented method of claim 1, further comprising: determining, for each of the plurality of simulated local agents in the agent network, local-agent-specific information about an action that should have been taken by the simulated local agent; providing, to each of the plurality of simulated local agents, the local-agent- specific information; and updating each local policy of each simulated local agent according to the local- agent-specific information and the global state of the agent network
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 13 depends. Furthermore, Mankowitz discloses determining, for each of the plurality of simulated local agents in the agent network, local-agent-specific information about an action that should have been taken by the simulated local agent:
Mankowitz teaches using critic reinforcement learning to determine the optimal action, which would be an action that should have been taken by the simulated local agent. Furthermore, this critic network is the Q-value neural network, wherein it is used to update the policy neural network which distributes updates to the agents (Page 11, lines 5-20). Therefore, the Q-value neural network would determine for each of the plurality of simulated local agents in the agent network, local-agent-specific information about the optimal action.
Mankowitz discloses providing, to each of the plurality of simulated local agents, the local-agent- specific information; and updating each local policy of each simulated local agent according to the local- agent-specific information and the global state of the agent network
Mankowitz teaches that the policy neural network provides the agent with a policy, or an action to take (Page 7, lines 10-20) which would be local-agent-specific information as the action is for that agent. Furthermore, the policy neural network that provides the local policy is updated by the optimal action, as describes above in claim 4 (Page 11, lines 5-20).
Furthermore, Mankowitz teaches determining the global pose of one or more part of an agent as the observation (Page 7, lines 25-30), wherein the pose would be global state information, which would in turn be used as part of the update process as the observation is included information regarding an agent.
Mankowitz does not teach that this is done with a plurality of simulated local agents. However, Hu teaches multi-agent reinforcement learning (Paragraph 5), and in combination would teach the above limitation of claim 13.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and the teachings of Mankowitz. This would have provided the advantage of improving the agent policy with regards to environmental perturbations (Page 6, lines 1-5).
Claims 15, 16, and 17 recite system that parallels the method of claims 1, 6, and 9 respectively. Therefore, the analysis discussed above with respect to claims 1, 6, and 9 also applies to claims 15, 16, and 17 respectively. Accordingly, claims 15, 16, and 17 are rejected based on substantially the same rationale as set forth above with respect to claims 1, 6, and 9 respectively.
Claims 18, 19, and 20 recite system that parallels the method of claims 1, 6, and 9 respectively. Therefore, the analysis discussed above with respect to claims 1, 6, and 9 also applies to claims 18, 19, and 20 respectively. Accordingly, claims 18, 19, and 20 are rejected based on substantially the same rationale as set forth above with respect to claims 1, 6, and 9 respectively.
Claims 8 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Hu in view of Mankowitz further in view of Gupta et al. (Pub. No. US 20240029012 A1, filed July 20th 2022, hereinafter Gupta).
Regarding claim 8, which depends upon claim 1:
Claim 8 recites:
The computer-implemented method of claim 1, wherein the action taken is a quantity of goods to be transported in the agent network
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 8 depends. Furthermore, Gupta discloses the limitation of claim 8:
Gupta in the same field of endeavor of supply chain management teaches the use of moving agent in order to transport goods and generate a logistical supply chain (Paragraph, 2), which would involve the determination of a quantity of goods to be transported in the agent network.
Gupta and the present application are analogous art because they are in the same field of endeavor.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and Mankowitz and the teachings of Gupta. This would have provided the advantage of facilitating efficient and timely transportation of goods to various locations (Gupta, Paragraph 3), which would improve Hu and Mankowitz as the efficient and timely completion of tasks is relevant to their respective agents.
Regarding claim 14, which depends upon claim 1:
Claim 14 recites:
The computer-implemented method of claim 1, wherein the agent network is a supply chain
Hu in view of Mankowitz disclose the method of claim 1 upon which claim 14 depends. Furthermore, Gupta discloses the limitation of claim 14:
Gupta in the same field of endeavor of supply chain management teaches the use of moving agent in order to transport goods and generate a logistical supply chain (Paragraph, 2). The supply chain would be the agent network as it is the system that the agents are working within.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to implement a method that utilized the teachings of Hu and Mankowitz and the teachings of Gupta. This would have provided the advantage of facilitating efficient and timely transportation of goods to various locations (Gupta, Paragraph 3), which would improve Hu and Mankowitz as the efficient and timely completion of tasks is relevant to their respective agents.
Response to Arguments
Applicant’s arguments filed 05-JANUARY-2026 have been fully considered, but the examiner believes that not all are fully persuasive.
Regarding the applicant’s remarks on the non-final office action’s 103 rejection of the claims 1, 15, and 18, the applicant argues that Hu in view of Mankowitz do not teach the amended limitations of these claims. As such, the applicant argues that all claims dependent on the above would additionally not be obvious under 103. However, the examiner believes that Hu in view of Mankowitz does teach the amended limitations and respectfully requests applicant’s consideration of the following:
Regarding the amended limitation of claim 1, the applicant argues that Hu does not teach “providing, to each of the plurality of simulated local agents, information representing a global state of the agent network, wherein the information representing a global state of the agent network indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from the global goal.” The applicant argues that Hu instead “describes an ‘updated global policy’ ‘to which each agent adheres when each RL agent interacts in the environment’ […] rather than providing, to each of the plurality of simulated local agents, information that indicates, for each of the plurality of simulated local agents, a deviation of the local policy of the simulated local agent from a global goal” (Applicant Arguments, page 2).
However, Hu teaches that there may be discrepancies between an expected global policy output and a local experience output (Paragraph 49). The expected global policy output is an indication of a deviation of the local policy of the simulated agent from a global goal as the global policy output is information representing a global state of the agent network by way of its current policy outputs, and represents the global goal and allows for comparison of the local experience output to it. This comparison provides a deviation of the local policy from a global goal which is used to assist in training both the local and global models.
Analogous limitations are amended to claims 15 and 18. For the same reasoning, Hu in view of Mankowitz is believed to discloses those limitations as well, and claims 15 and 18 are rejected under similar rationale.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALEXANDRIA JOSEPHINE MILLER whose telephone number is (703)756-5684. The examiner can normally be reached Monday-Thursday: 7:30 - 5:00 pm, every other Friday 7:30 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.J.M./Examiner, Art Unit 2142
/Mariela Reyes/Supervisory Patent Examiner, Art Unit 2142