Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after January 29, 2024, is being examined under the first inventor to file provisions of the AIA
Status of Claims
The present application is being examined under the claims filed on January 29, 2024.
Claims 1-20 are rejected.
Claims 1-20 are pending
Specification
The specification filed on January 29, 2024 is acceptable for examination purposes
Drawings
The drawings filed on January 29, 2024 is acceptable for examination purposes.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 1 is method claim. Therefore, claim 1 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon ?
generating a control policy model comprising a plurality of nodes and a plurality of edges interconnecting the plurality of nodes, wherein the plurality of nodes represents a plurality of agents or subsystems and the plurality of edges represent information exchange between the plurality of agents or subsystems
“generating a control policy model comprising a plurality of nodes and a plurality of edges” can be understand as generating a graph with edges and nodes. This can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
encoding agent behavior control policy within the control policy model for executing to coordinate a plurality of actions of the plurality of agents or subsystems
“encoding agent behavior control policy within the control policy model” can be understand as evaluate a policy within a model. This can be performed in the human mind (including an observation, evaluation, judgment, opinion)
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea. Because the claim does not have any computer system or physical device to executing to coordinate a plurality of actions of the plurality of agents or subsystems. Therefore, there is no additional element to integration into a practical application.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
For the reasons above, claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-14. The additional limitations of the dependent claims are addressed below.
Regarding claim 2,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 2 is method claim. Therefore, claim 2 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein the plurality of nodes represents sensors and effectors and wherein the plurality of edges represents information exchange between the sensors and the effectors.
The claim just narrows down on what nodes can be represented, and nodes is the information exchange between them which is already recite in claim 1. These can be performed in the human mind (including an observation, evaluation, judgment, opinion)
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 3,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 3 is method claim. Therefore, claim 3 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein the control policy model comprises a Capability Graph Network (CGN).
The claim just gives an abstract idea from claim 1. This can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea.
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 4,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 4 is method claim. Therefore, claim 4 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
the CGN comprises a graph-based neural network and wherein the method further comprises: executing the agent behavior control policy using the graph-based neural network to coordinate actions of the plurality of agents or subsystems.
“the CGN comprises a graph-based neural network” can be understand as a series of mathematical operation that have number.
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional Elements: a graph-based neural network
The additional element only recites the idea of additional elements; the claim fails to recite details of how a solution to a problem is accomplished
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. So, the claim does not recite Additional Elements that integrate The Judicial Exception into A Practical Application See MPEP 2106.05(f).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
Additional Elements: a graph-based neural network
The additional element only recites the idea of additional elements; the claim fails to recite details of how a solution to a problem is accomplished
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(f).
Regarding claim 5,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 5 is method claim. Therefore, claim 5 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein sensor data and one or more observations about an environment surrounding the plurality of agents comprise input to the graph-based neural network.
“sensor data and one or more observations about an environment surrounding the plurality of agents” can be understand as sensor data and data from observation of an environment used for the input of the graph-based neural network. This can be performed by using mathematical formulas or equations, and mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional Elements: sensor data and one or more observations about an environment surrounding the plurality of agents comprise input to the graph-based neural network.
The additional elemment only recites the type of data it used, the claim failed to recite on how to collect the data.
Therefore, the claim recites additional elements is insignificant Extra-Solution Activity, that does not integrate The Judicial Exception into A Practical Application See MPEP 2106.05(g).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
Additional Elements: sensor data and one or more observations about an environment surrounding the plurality of agents comprise input to the graph-based neural network.
The additional element only recites the type of data it used, which is receiving or transmitting data over a network. This function is well-understood.
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(d), subsection II
Regarding claim 6,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 6 is method claim. Therefore, claim 6 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
dynamically reconfiguring the graph-based neural network, based on one or more changes in the environment, by adding and/or removing a subgraph of the graph-based neural network and by adding/removing one or more edges associated with added and/or removed subgraph.
“adding and/or removing" can be performed by using mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 7,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 7 is method claim. Therefore, claim 7 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein generating the graph-based neural network comprises: generating a plurality of graph-based neural networks, wherein each of the plurality of graph-based neural networks represents an individual agent of one or more pluralities of agents.
According to claim 4, graph-based neural networks are a mathematical concept. Generating a mathematical concept can be performed by mathematical formulas or equations, mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 8,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 8 is method claim. Therefore, claim 8 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein two or more of the teams of agents are split into adversarial teams.
“split into adversarial teams” can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea.
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 9,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 9 is method claim. Therefore, claim 9 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
organizing a plurality of tasks to be performed by the team of agents into a hierarchical structure having one or more lower levels and one or more higher levels.
“organizing a plurality of tasks to be performed by the team of agents into a hierarchical structure” can be understand as performing a mental process on a generic computer.
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 10,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 10 is method claim. Therefore, claim 10 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
summarizing information about an environment obtained by the one or more lower levels; and passing the summarized information up the hierarchical structure to the one or more higher levels.
“summarizing information” can be can be performed in the human mind (including an observation, evaluation, judgment, opinion)
Therefore, the claim recites an Abstract Idea, which is a Mental Process (see MPEP § 2106.04(a)(2), subsection III)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 11,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 11 is method claim. Therefore, claim 11 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
jointly training two or more layers of the graph-based neural network using hierarchical reinforcement learning
“jointly training” can be understand at training an machine learning model together. This is related to mathematical relationships, mathematical formulas or equations, and mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional Elements: jointly training two or more layers of the graph-based neural network using hierarchical reinforcement learning
The additional element only recites the name of the training method (i.e. hierarchical reinforcement learning), and the usage of the result (i.e. to refine coordination). The claim did not recite step by step of how to train the model
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. See MPEP 2106.05(f).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
jointly training two or more layers of the graph-based neural network using hierarchical reinforcement learning
The additional element only recites the name of the training method (i.e. hierarchical reinforcement learning), and the usage of the result (i.e. to refine coordination). The claim did not recite step by step of how to train the model
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(f).
Regarding claim 12,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 12 is method claim. Therefore, claim 12 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
generating, by the graph-based neural network, an output comprising at least one of: updated features associated with one or more of the plurality of nodes and one or more probabilities associated with one or more actions to be performed by the team of agents.
“updated features” and “one or more probabilities” is the result of mathematical formulas or equations, and mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional Elements: generating, by the graph-based neural network, an output
The claim recites generating an output; however, the claim did not recite the usage of that output.
Therefore, the claim recites additional elements is insignificant Extra-Solution Activity, that does not integrate The Judicial Exception into A Practical Application See MPEP 2106.05(g).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The additional element only recites the type of data it used, which is receiving or transmitting data over a network. This function is well-understood.
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(d), subsection II
Regarding claim 13,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 13 is method claim. Therefore, claim 13 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
wherein the agent behavior control policy comprises a decentralized control policy independently executed by the plurality of agents.
“decentralized control policy independently executed by the plurality of agents” can be understand as evaluating the policy independently executed by agents. This can be performed in the human mind (including an observation, evaluation, judgment, opinion)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional elements: independently executed by the plurality of agents
The additional elements recite (fthat the agent executed the control policy; however, the additional elements failed to recite how the agent was structured.
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. So, the claim does not recite Additional Elements that integrate The Judicial Exception into A Practical Application See MPEP 2106.05(f).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
Additional elements: independently executed by the plurality of agents
The additional elements recite that the agent executed the control policy; however, the additional elements failed to recite how the agent was structured.
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(f).
Regarding claim 14,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 14 is method claim. Therefore, claim 14 is directed to a process.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
modifying one or more properties of the one or more of the plurality of edges to represent communication restrictions between two or more of the plurality of nodes.
modifying one or more properties is an action that makes change in the structure. This can be performed by using mathematical formulas or equations, and mathematical calculations
Therefore, the claim recites an Abstract Idea, which is Mathematical Concepts (see MPEP § 2106.04(a)(2), subsection I)
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
The claim recites no additional element besides the abstract idea.
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
The claim recites no additional element besides the abstract idea. Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception.
Regarding claim 15,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 15 is system claim. Therefore, claim 14 is directed to a machine.
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
Similar reason as Step 2A Prong 1 as Claim 1
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional element:
processing circuitry in communication with storage media
the processing circuitry configured to
The additional elements only show a system to execute the abstract idea. They failed to show the particular machine or improve of computer functioning.
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. So, the claim does not recite Additional Elements that integrate The Judicial Exception into A Practical Application See MPEP 2106.05(f).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
Additional element:
processing circuitry in communication with storage media
the processing circuitry configured to
The additional elements only recite a system to execute the abstract idea. They failed to recite the particular machine or improve of computer functioning.
Therefore, the claim does not recite Additional Elements that amount to Significantly more than The Judicial Exception. See MPEP 2106.05(f).
For the reasons above, claim 15 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 16-19. The additional limitations of the dependent claims are addressed below.
Regarding claim 16, it is rejected for similar reasons as claim 2.
Regarding claim 17, it is rejected for similar reasons as claim 3.
Regarding claim 18, it is rejected for similar reasons as claim 4.
Regarding claim 16, it is rejected for similar reasons as claim 6.
Regarding claim 20,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Claim 20 is CRM claim. Therefore, claim 20 is directed to a manufactures .
Step 2A Prong 1: Does the claim recite an Abstract Idea, Law Of Nature or Natural Phenomenon?
Similar reason as Step 2A Prong 1 as Claim 1
Step 2A Prong 2: Does the claim recite Additional Elements that integrate The Judicial Exception into A Practical Application?
Additional elements:
Non-transitory computer-readable storage media
The additional element recites the media that store and execute the abstract idea.
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. So, the claim does not recite Additional Elements that integrate The Judicial Exception into A Practical Application See MPEP 2106.05(f).
Step 2B: Does the claim recite Additional Elements that amount to Significantly more than The Judicial Exception?
Additional elements:
Non-transitory computer-readable storage media
The additional element recites the media that store and execute the abstract idea.
Therefore, the claim merely reciting the words "apply it" (or an equivalent) with the judicial exception into A Practical Application. So, the claim does not recite Additional Elements that integrate The Judicial Exception into A Practical Application See MPEP 2106.05(f).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 6, 8, 9, 11, 12, 14 and 18 are rejected under 35 U.S.C. 112(b) because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 6,
The phrase “the environment” did not mention or define in claim 4 which claim 6 depend on.
However, in claim 5, the claim before claim 6 mention the phrase “an environment”
Thus, the claim lacks antecedent basis
Regarding claim 8,
The phrase “teams of agents” did not mention or define in claim 7 which claim 8 depend on.
Neither do claim 4 which claim 7 depend on mention or define the phrase “teams of agents”
Thus, the claim lacks antecedent basis
Regarding claim 9,
The phrase “teams of agents” did not mention or define in claim 4 which claim 9 depend on.
Thus, the claim lacks antecedent basis
Regarding claim 11,
The phrase “teams of agents” did not mention or define in claim 4 which claim 11 depend on.
Thus, the claim lacks antecedent basis
Regarding claim 12,
The phrase “teams of agents” did not mention or define in claim 4 which claim 12 depend on.
Thus, the claim lacks antecedent basis
Claim Objections
Claim 14 is objected to because of the following informalities
“one or more properties of the one or more of the plurality of edges” should read “one or more properties of one or more of the plurality of edges”
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 4, 5, 7, 8, 9, 10, 11, 12, 14, 15, 17, 18, 20 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US 20210192358 A1) in view of Chiu et al. (US 20230394294 A1)
Regarding claim 1,
A method for coordinating actions of a plurality of agents or subsystems, the method comprising (preamble)
Song teaches:
generating a control policy model comprising a plurality of nodes and a plurality of edges interconnecting the plurality of nodes
Song, paragraph [0016], “The reinforcement learning system may further comprise an action selection policy neural network to process the state data and reward data to select the actions. The action selection policy neural network may be configured to receive and process the representation data to select the actions.”
Song, paragraph [0021], “The method may comprise receiving agent data representing actions for each of multiple agents. The method may further comprise processing the agent data in conjunction with graph data to provide encoded graph data. The graph data may comprise data representing at least nodes and edges of a graph, wherein each of the agents is represented by a node… wherein the edges connect the agents to each other and to the non-agent entities”
Examiner note:
generating a control policy model (i.e. the action selection policy neural network that processes or receives the data)
comprising a plurality of nodes and a plurality of edges interconnecting the plurality of nodes(i.e. the edges connect the agents which each of the agents is represented by a node)
the plurality of nodes represents a plurality of agents or subsystems
Song, paragraph [0021], “each of the agents is represented by a node, wherein non-agent entities in the environment are each represented by a node”
Examiner note:
the plurality of nodes represents a plurality of agents or subsystems (i.e. each of the agents is represented by a node and non-agent entities… are each represented by a node)
the plurality of edges represent information exchange between the plurality of agents or subsystems
Song, paragraph [0012], “the edge attributes of the decoded graph may encode information which can be used to explain the behaviour of an agent,”
Examiner note:
the plurality of edges represent information exchange between the plurality of agents or subsystems (i.e. information which can be used to explain the behaviour of an agent)
encoding agent behavior control policy within the control policy model
Song, paragraph [0021], “each of the agents is represented by a node… wherein the nodes have node attributes for determining the actions of each agent”
Song, claim 1, “the encoded graph data and provide processed graph data comprising an updated version of the node attributes and edge attributes of the encoded graph data”
Song, paragraph [0016], “The action selection policy neural network may be configured to receive and process the representation data to select the actions.”
Examiner note:
encoding agent behavior control policy (i.e. the encoded graph data and provide processed graph data the encoded graph data and provide processed graph data)
within the control policy model (i.e. The action selection policy neural network that receives and process the representation data to select the actions.)
Song does not explicitly teach:
for executing to coordinate a plurality of actions of the plurality of agents or subsystems.
Chiu teaches:
for executing to coordinate a plurality of actions of the plurality of agents or subsystems.
Chiu, paragraph [0066], “θ is introduced as the set of all parameters describing the transfer functions of the GN. These functions determine what action the platforms can execute and, so, define a policy, π(θ).”
Chiu, paragraph [0006], “Embodiments of methods, apparatuses and systems for hierarchical, deep reinforcement learning (DRL) based planning and control for coordinating a team of multi-domain platforms/agents are disclosed herein.”
Examiner note:
for executing(i.e. these functions determine what action the platforms can execute)
to coordinate a plurality of actions of the plurality of agents or subsystems (i.e. planning and control for coordinating a team of multi-domain platforms/agents are disclosed herein.)
It would have been obvious to a person having ordinary skill in the art at a time before
the effective filing date of the application to have further modified the Song disclosures
and teachings by generating a control policy that have plurality of nodes represent agent or subsystem, plurality of edges represent information exchange between nodes, encoding agent behavior or data in the control policy and execute it to coordinate plurality agents or subsystem taught and suggested by Chiu. Such a person would have been motivated to do so with a reasonable expectation of success to allow generating control policy with nodes and edges, using those to encoding agent behavior or data in the control policy (Song, par [0021] and claim 1), and then executing them to coordinate a team of multi-domain platforms/agents. (Chiu, par [0006] and par [0066])
Regarding claim 3,
The combination of Song and Chiu teach:
The method of claim 1 (preamble)
the control policy model comprises a Capability Graph Network (CGN).
Song, paragraph [0059], “The agent incorporates the graph processing neural network system (Relational Forward Model, RFM) 106”
Song, paragraph [0061], “The representation data output 108 from the RFM 106 is provided to the RL system”
Song, paragraph [0016], “In some implementations the system may be included in a reinforcement learning system… The reinforcement learning system may further comprise an action selection policy neural network to process the state data and reward data to select the actions. The action selection policy neural network may be configured to receive and process the representation data to select the actions.”
Examiner note:
the control policy model (i.e. The action selection policy neural network receive and process the representation data to select the actions )
Capability Graph Network (CGN). (i.e. the graph processing neural network system provide data to RL system, which comprises action selection policy neural network)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 4,
The combination of Song and Chiu teach:
The method of claim 3 (preamble)
the CGN comprises a graph-based neural network
Song, paragraph [0042], “The neural network system 100 comprises a graph processing neural network system 106 including one or more graph neural network blocks 110 which process the input graph to provide data for an output 108,”
Examiner note:
the CGN (i.e. graph processing neural network system) comprises a graph-based neural network (i.e. 106 including one or more graph neural network blocks)
the method further comprises: executing the agent behavior control policy using the graph-based neural network
Song, paragraph [0016], “In some implementations the system may be included in a reinforcement learning system… The reinforcement learning system may further comprise an action selection policy neural network to process the state data and reward data to select the actions. The action selection policy neural network may be configured to receive and process the representation data to select the actions.”
Examiner note:
the method further comprises: executing the agent behavior control policy (i.e. The action selection policy neural network may be configured to receive and process the representation data to select the actions.)
to coordinate actions of the plurality of agents or subsystems.
Chiu, paragraph [0006], “Embodiments of methods, apparatuses and systems for hierarchical, deep reinforcement learning (DRL) based planning and control for coordinating a team of multi-domain platforms/agents are disclosed herein.”
Examiner note:
to coordinate (i.e. planning and control for coordinating) actions of the plurality of agents or subsystems (i.e. planning and control for coordinating a team of multi-domain platforms/agents)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 5,
The combination of Song and Chiu teach:
The method of claim 4 (preamble)
sensor data and one or more observations about an environment surrounding the plurality of agents
Song, paragraph [0013], “A neural network system as described above may be trained by supervised training, for example based on observations of the behaviour of the multiple agents in the shared environment.”
Song, paragraph [0016], “The reinforcement learning system may comprise an input to obtain state data representing a state of the shared environment”
Examiner note:
sensor data (i.e. obtain state data) and one or more observations about an environment surrounding the plurality of agents (i.e. on observations of the behaviour of the multiple agents in the shared environment)
input to the graph-based neural network.
Song, claim 16, “processing the encoded graph data using a recurrent graph neural network to provide processed graph data comprising an updated version of the node attributes and edge attributes of the encoded graph data;”
Examiner note:
input to the graph-based neural network (i.e. processing the encoded graph data using a recurrent graph neural network)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 7,
The combination of Song and Chiu teach:
The method of claim 4 (preamble)
generating the graph-based neural network comprises: generating a plurality of graph-based neural networks
Song, paragraph [0051], “FIG. 3 shows an implementation of the graph processing neural network system 106. This comprises an encoder graph neural network subsystem (GN encoder) 302 coupled to a recurrent graph neural network subsystem (e.g. Graph GRU) 304 coupled to a decoder graph neural network subsystem (GN decoder) 306.”
Examiner note:
Paragraph [0051] teaches that the graph processing neural network comprises an encoder graph neural network subsystem, a recurrent graph neural network subsystem, a decoder graph neural network subsystem coupled to each other. This mean that the graph processing neural network have three graph neural network (i.e. more than 2 or plurality)
each of the plurality of graph-based neural networks represents an individual agent of one or more pluralities of agents.
Chiu, paragraph [0061], “In the embodiment of FIG. 3 Graph nodes of the GN 332 represent capabilities associated with a platform, … each vertex, V, in the graph is associated with a respective capability of one of the platform node”
Examiner note:
PNG
media_image1.png
540
836
media_image1.png
Greyscale
each of the plurality of graph-based neural networks (i.e. 3 Graph nodes of the GN 332 ) represents an individual agent of one or more pluralities of agents (i.e. represent capabilities associated with Regarding claim 8,
The combination of Song and Chiu teach:
The method of claim 7,
two or more of the teams of agents are split into adversarial teams.
Chiu, paragraph [0080], “ In some embodiments, two approaches can be used by the Simulator 706 for modelling the adversarial policy parameter, θ.sup.adv: a Deep Neural Network approach and a Graph Network approach.
Examiner note:
two or more of the teams of agents(i.e. two approaches can be used by the Simulator 706 ) are split into adversarial teams. (i.e. modelling the adversarial policy parameter)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 9,
The method of claim 4,
organizing a plurality of tasks to be performed by the team of agents
Chiu, claim 1, “An artificial intelligence-based method for coordinating a team of platforms, the method comprising: implementing a global planning layer for; determining a collective goal for the team of the platforms; and determining, by applying at least one machine learning process, at least one respective platform goal to be achieved by at least one of the platforms to achieve the determined collective goal;”
Examiner note:
organizing a plurality of tasks (i.e. a collective goal or platform goal) to be performed by the team of agents (i.e. team of platforms)
into a hierarchical structure having one or more lower levels and one or more higher levels.
Chiu, paragraph [0034], “Some embodiments of the present principles implement deep reinforcement learning (DRL) and a hierarchical architecture to guide a team of heterogeneous platforms … each layer focuses on different domains (such as long-term mission/short-term task, team-level/unit-level) and provides decentralized hierarchical Deep Reinforcement Learning (DRL) and each layer can be trained separately. ”
Examiner note:
into a hierarchical structure (i.e. a hierarchical architecture) having one or more lower levels and one or more higher levels (i.e. each layer focuses on different domains or team-level/unit-level)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 10,
The combination of Song and Chiu teach:
The method of claim 9 (preamble)
summarizing information about an environment obtained by the one or more lower levels;
Chiu, paragraph [0084], “low-level Reactive Autonomy layer 310 (left) and the high-level Deliberative Autonomy layer 330 (right).”
Chiu, paragraph [0085], “As depicted in the embodiment of FIG. 8, the Perceptive Autonomy layer 320 can receive processed scene data, including but not limited to, detected object information 804, and semantic segmentation information 806 from the Reactive autonomy layer 310.”
Chiu, paragraph [0090], “As depicted in FIG. 8, the Reactive Autonomy layer 310 is not only responsible for low-level control and locomotion, but also processes low-level visual inputs (i.e., RGB camera images and depth sensing) to extrapolate to scene semantics”
Examiner note:
Paragraph [0084], [0085], [0090] teach that the Perceptive Autonomy layer 320 store the data or information that has been processed low-level Reactive Autonomy layer 310 (left)
summarizing information about an environment obtained (i.e. processed scene data or processes low-level visual inputs by the one or more lower levels (i.e. low-level Reactive Autonomy layer 310)
passing the summarized information up the hierarchical structure to the one or more higher levels.
Chiu, paragraph [0098], “the Perceptive Autonomy layer 320 can also communicate information about its status with regard to the overall mission, such as its accomplishment of short-term mission goals or information derived from an adversary model (described below), to the Deliberative Autonomy layer 330.”
Examiner note:
Paragraph [0098] teaches that the Perceptive Autonomy layer 320 pass the information it has to Deliberative Autonomy layer 330
PNG
media_image2.png
576
539
media_image2.png
Greyscale
passing the summarized information (i.e. information about its status) up the hierarchical structure to the one or more higher levels. (i.e. Deliberative Autonomy layer 330)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 11,
The combination of Song and Chiu teach:
The method of claim 4 (preamble)
jointly training two or more layers of the graph-based neural network
Son, paragraph [0019], “the two systems may be trained jointly”
Son, paragraph [0053], “The graph processing neural network system 106 may be trained by training the graph neural network subsystems jointly”
Son, paragraph [0069], “In implementations the training engine 120 may train both the RFM 106 and the RL system. They may be trained jointly”
Examiner note:
jointly training (i.e. trained jointly) two or more layers of the graph-based neural network (i.e. the two systems)
using hierarchical reinforcement learning to refine coordination within the team of agents.
Chiu, paragraph [0006], “Embodiments of methods, apparatuses and systems for hierarchical, deep reinforcement learning (DRL) based planning and control for coordinating a team of multi-domain platforms/agents are disclosed herein.”
Examiner note:
using hierarchical reinforcement learning to refine (i.e. hierarchical, deep reinforcement learning) coordination within the team of agents.(i.e. coordinating a team of multi-domain platforms/agents)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 12,
The combination of Song and Chiu teach:
The method of claim 4 (preamble)
generating, by the graph-based neural network, an output comprising at least one of: updated features associated with one or more of the plurality of nodes
Song, claim 1, “a decoder graph neural network subsystem to decode the processed graph data and provide decoded graph data comprising an updated version of the node attributes and edge attributes of the processed graph data”
Examiner note:
generating, by the graph-based neural network (i.e. a decoder graph neural network subsystem to decode the processed graph data)
an output comprising at least one of: updated features associated with one or more of the plurality of nodes (i.e. provide decoded graph data comprising an updated version of the node attributes of the processed graph data)
one or more probabilities associated with one or more actions to be performed by the team of agents.
Song, paragraph [0065], “In more detail, in one example, an action selection output of the action selection policy network 602 may include a respective numerical probability value for each action in a set of possible actions that can be performed by the agent. The RL system can select the action to be performed by the agent, e.g., by sampling an action in accordance with the probability values for the actions, or by selecting the action with the highest probability value. accordance with the probability values for the actions, or by selecting the action with the highest probability value.”
Examiner note:
one or more probabilities associated with (i.e. a respective numerical probability value for each action or probability values for the actions)
one or more actions to be performed by the team of agents.(i.e. an action selection output of the action selection policy network or select the action to be performed by the agent)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding claim 14,
The combination of Song and Chiu teach:
The method of claim 1 (preamble)
modifying one or more properties of the one or more of the plurality of edges
Chiu, paragraph [0064], “The updated edge parameters are provided to the vertex vector of the receiver platform… That is, in the Communications Edge update procedure…how the edges evolve as a function of the previous edge value and the values of source and receiver nodes.”
Examiner note:
modifying one or more properties of the one or more of the plurality of edges (i.e. updated edge parameters, Communications Edge update procedure)
to represent communication restrictions between two or more of the plurality of nodes.
Chiu, paragraph [0061], “Graph nodes of the GN 332 represent capabilities associated with a platform, while edges represent possible constraints associated with communications/data sharing links”
Examiner note:
to represent communication restrictions (i.e. possible constraints associated with communications/data sharing links) between two or more of the plurality of nodes. (i.e. Graph nodes of the GN 332 represent capabilities associated with a platform)
Regarding claim 15, it is rejected for similar reasons as claim 1.
A computing system to perform the steps recited in claim 1
Regarding claim 17, it is rejected for similar reasons as claim 3.
Regarding claim 18, it is rejected for similar reasons as claim 4.
Regarding claim 20, it is rejected for similar reasons as claim 1.
Non-transitory computer-readable storage media to perform the steps recited in claim 1 (Song’s claim 20)
Claims 2, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US 20210192358 A1) in view of Chiu et al. (US 20230394294 A1) as applied in claim 1, and further view of Dang et al (CN 118263893 A)
Regarding claim 2,
The combination of Song and Chiu teach:
The method of claim 1 (preamble)
wherein the plurality of edges represents information exchange between the sensors and the effectors.
Song, paragraph [0012], “the edge attributes of the decoded graph may encode information which can be used to explain the behaviour of an agent,”
Song, paragraph [0044], “edges may connect each agent to each other agent and to each non-agent entity”
Examiner note:
wherein the plurality of edges represents information exchange (i.e. “the edge attributes of the decoded graph may encode information) between the sensors and the effectors. (i.e. connect each agent to each other agent)
The combination of Song and Chiu does not explicitly teach:
plurality of nodes represents sensors and effectors
Dang teaches:
plurality of nodes represents sensors and effectors
Dang, Description, “In the embodiment, a sensor is arranged at each node…the sensor is connected with the edge calculating module for transmitting the collected information to the edge”
Examiner note:
plurality of nodes represents sensors and effectors (i.e. a sensor is arranged at each node)
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Song and Chiu disclosures and teachings by generating a control policy that have plurality of edges represent information exchange between plurality of nodes, and plurality of nodes represent sensor taught and suggested by Dang. Such a person would have been motivated to do so with a reasonable expectation of success to allow generating control policy with plurality of nodes and plurality of edges, where plurality of nodes represent sensor (Dang, Description), and plurality of edges represents information exchange between them. (Song, par [0012] and par [0044])
Regarding claim 16, it is rejected for similar reasons as claim 2.
Claims 6 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US 20210192358 A1) in view of Chiu et al. (US 20230394294 A1) as applied in claim 4, and further view of Siqi et al (Graphcomm: A Graph Neural Network Based Method for Multi-Agent Reinforcement Learning)
Regarding claim 6, 19
The combination of Song and Chiu teach:
The method of claim 4 (preamble)
The combination of Song and Chiu does not explicitly teach:
dynamically reconfiguring the graph-based neural network, based on one or more changes in the environment
adding and/or removing a subgraph of the graph-based neural network and by adding/removing one or more edges associated with added and/or removed subgraph.
Siqi teaches:
dynamically reconfiguring the graph-based neural network, based on one or more changes in the environment
Siqi, Page 1 - I. Introduction, “The implicit relationships are dynamic”
Siqi, Page 1 - I. Introduction, “Multi-Agent Reinforcement Learning (MARL)
Siqi, Page 1 – Abstract, “In this work, we propose GraphComm, a method makes use of the relationships among agents for MARL communication… GraphComm use Graph Neural Networks (GNNs) to model the relational information, and use GNNs to assist the learning of agent communication.”
Siqi, Page 3 - 4.2.2. Implicit Relations, “In this work, we consider a dynamic MARL that agents can disappear in the middle of the tasks”
Examiner note:
Siqui, introduction and abstract teach that MARL uses GNN. Siqui teach MARL can be dynamic based on the change in the environment. Change in the environment was taught by Siqi on 4.2.2 as the agent can disappear in the middle of the tasks.
Therefore, the limitation was taught by Siqi.
by adding and/or removing a subgraph of the graph-based neural network and by adding/removing one or more edges associated with added and/or removed subgraph.
Siqi, Page 3 - 4.2.2. Implicit Relations, “In this work, we consider a dynamic MARL that agents can disappear in the middle of the tasks”
Siqi, Page 3 - 4.2.2. Implicit Relations, “If an agent disappears, then all the edges connected with the agents are removed.”
Examiner note:
Siqi teaches that an agent can disappear in the middle of the task, which mean the amount of agent (i.e. subgraph) can be change during the task.
Siqi teaches that when agent (i.e. subgraph) disappears (i.e. changed in amount) the edges connected (i.e. associated) to it also be removed (i.e. changed in amount)
Therefore, the limitation was taught by Siqi.
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Song and Chiu disclosures and teachings by generating a control policy that comprises a graph base neural network (i.e. GNN) which is dynamically base on the changed in the environment, which are the change in amount of subgraph that also change the edges connect to it taught or suggested by Siqi. Such a person would have been motivated to do so with a reasonable expectation of success to allow generating control policy comprises a graph base neural network (i.e. GNN), that GNN can by dynamically configured base on the change of the environment (Siqi, I. Introduction), by change of amount in the subgraph(i.e. agent) that also change to the associated edges.
Regarding claim 19, it is rejected for similar reasons as claim 6
Claims 13 are rejected under 35 U.S.C. 103 as being unpatentable over Song et al. (US 20210192358 A1) in view of Chiu et al. (US 20230394294 A1) as applied in claim 1, and further view of Yeh et al (US 20220014963 A1)
Regarding claim 13,
The combination of Song and Chiu does not explicitly teach:
the agent behavior control policy comprises a decentralized control policy independently executed by the plurality of agents
Yeh teaches:
the agent behavior control policy comprises a decentralized control policy independently executed by the plurality of agents
Yeh, paragraph [0050], “In cases where the state is determined by individual multi-agents 101 so each multi-agent 101 can only determine the action based on its own observed state using the deployed model. Such architecture 200 allows centralized training and distributed inferences.”
Examiner note:
the agent behavior control policy comprises a decentralized control policy (i.e. architecture 200 allows centralized training and distributed inferences.)
executed by the plurality of agents (i.e. so each multi-agent 101 can only determine the action based on its own observed state using the deployed model.)
It would have been obvious to a person having ordinary skill in the art at a time before the effective filing date of the application to have further modified the Song and Chiu disclosures and teachings by encoding agent behavior control policy and the agent behavior control policy that have a decentralized control policy, independently executed by plurality of agents taught or suggested by Yeh. Such a person would have been motivated to do so with a reasonable expectation of success to allow encoding agent behavior control policy that have centralized training and distributed inferences, and each multi-agent can determine the action. (Yeh, par [0050]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAO T VONG whose telephone number is (571)270-7701. The examiner can normally be reached Monday - Friday (8:00 AM - 6:00 AM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker A Lamardo can be reached at (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAO THIEN VONG/Examiner, Art Unit 2147
/VIKER A LAMARDO/Supervisory Patent Examiner, Art Unit 2147
Prosecution Insights