Prosecution Insights
Last updated: November 20, 2026
Application No. 18/666,607

COLLABORATIVE EXPLORATION FOR REINFORCEMENT LEARNING

Non-Final OA §101§102§112
Filed
May 16, 2024
Priority
May 17, 2023 — FI 20235558
Examiner
BEJCEK II, ROBERT H
Art Unit
2148
Tech Center
2100 — Computer Architecture & Software
Assignee
Nokia Corporation
OA Round
1 (Non-Final)
63%
Grant Probability
Moderate
1-2
OA Rounds
1y 3m
Est. Remaining
85%
With Interview

Examiner Intelligence

Grants 63% of resolved cases
63%
Career Allowance Rate
162 granted / 258 resolved
+7.8% vs TC avg
Strong +22% interview lift
Without
With
+22.5%
Interview Lift
resolved cases with interview
Typical timeline
3y 9m
Avg Prosecution
23 currently pending
Career history
292
Total Applications
across all art units

Statute-Specific Performance

§101
22.6%
-17.4% vs TC avg
§103
41.1%
+1.1% vs TC avg
§102
10.5%
-29.5% vs TC avg
§112
22.4%
-17.6% vs TC avg
Black line = Tech Center average estimate • Based on career data from 258 resolved cases

Office Action

§101 §102 §112
DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Drawings The drawings are objected to because of the following informalities. The view numbers must be larger than the numbers used for reference characters. See 37 C.F.R. 1.84(u)(2). Corrected drawings in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. The replacement sheet(s) should be labeled “Replacement Sheet” in the page header (as per 37 CFR 1.84(c)) so as not to obstruct any portion of the drawing figures. If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. Title The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. Examiner believes that the title of the invention is imprecise. A descriptive title indicative of the invention will help in proper indexing, classifying, searching, etc. See MPEP 606.01. However, the title of the invention should be limited to 500 characters. Examiner suggests including the aspect(s) of the claims which Applicant believes to be novel or nonobvious over the prior art. Claim Objections The following claims are objected to because of the following informalities: Claim 1 recites for use by a reinforcement learning, RL, management function but for the sake of clarity should recite for use by a reinforcement learning (RL) management function. Claims 13 and 14 each start with dash marks before their first two limitations. For the sake of clarity, these dashes should be removed. Appropriate correction is required. Claim Rejections - 35 USC § 112(b) The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claims 1-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 1 recites the considered RL agent without proper antecedent basis. Claim 3 has two periods, one at the end of each limitation. A claim is a sentence and can only have one period. Claims 7 and 8 each recite processing the first exploration results and second exploration results but second exploration results were previously introduced in claim 1. It is unclear if this is the same element, a different element, or related elements. Claim 9 recites the third RL agent without proper antecedent basis. Claim 13 recites at least one RL agent, however the abbreviation RL was not previously defined. Additionally, claim 13 recites the considered RL agent without proper antecedent basis. Claim 14 recites the RL management function, however the abbreviation RL was not previously defined. For this reason, the above listed claims are rejected for containing this language or being dependent on a claim that contains this language. Claim Rejections - 35 USC § 101 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1: Claim 1 is a method claim. Claim 13 is a apparatus claim. Therefore, claims 1 and 13 are directed to either a process, machine, manufacture or composition of matter. With respect to Claim 1: Step 2A Prong 1: configuring, based on first information representative of exploration capabilities received from a first RL agent, the first RL agent with first exploration tasks of a first exploration process to be performed by the first RL agent to contribute to a collaborative RL (mental process – user can manually configure, based on first information representative of exploration capabilities received from a first RL agent, the first RL agent with first exploration tasks of a first exploration process to be performed by the first RL agent to contribute to a collaborative RL) processing the first exploration results and second exploration results of a second exploration process performed by a second RL agent to contribute to the collaborative RL (mental process – user can manually process the first exploration results and second exploration results of a second exploration process performed by a second RL agent to contribute to the collaborative RL) Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements: receiving, from at least one RL agent, information representative of exploration capabilities of the considered RL agent (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)) receiving, from the first RL agent, first exploration results of the first exploration process (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)) Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements: receiving, from at least one RL agent, information representative of exploration capabilities of the considered RL agent (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer) receiving, from the first RL agent, first exploration results of the first exploration process (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer) Conclusion: The claim is not patent eligible. Claim 13 is rejected on the same grounds as claim 1. Additionally for claim 13: Claim 13 has the additional elements of at least one processor and at least one memory storing instructions. These elements are mere instructions to apply the exception using a generic computer component under Step 2A prong 2 and Step 2B. Regarding Claim 2: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually wherein the first RL agent is configured with the first exploration tasks in response to a determination, based on the first information representative of exploration capabilities received from the first RL agent, that the first RL agent has capabilities to contribute to the collaborative RL. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 3: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually selecting, based on the received information representative of exploration capabilities, a RL strategy that defines an algorithm to determine one or more actions to be taken by an RL agent in the context of an exploration process. configuring the first RL agent with the selected RL strategy. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 4: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually wherein the second exploration process is executed by the second RL agent in the first node, the method comprising executing, by the second RL agent in the first node, the second exploration process to generate the second exploration results. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 5: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually wherein the second exploration process is executed by the second RL agent in a second node distinct from the first node, wherein the method comprises receiving the second exploration results from the second RL agent. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 6: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually processing the first exploration results and second exploration results is performed to configure further exploration tasks for the first RL agent. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 7: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually processing the first exploration results and second exploration results includes determining whether a convergence criterion is met for the collaborative RL;. wherein the method comprises configuring the first agent with further exploration tasks based on a determination that the convergence criterion is not met for the collaborative RL. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 8: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually processing the first exploration results and second exploration results includes aggregating the first exploration results and the second exploration results to generate aggregated exploration results; determining whether a convergence criterion is met for the collaborative RL based on the aggregated exploration results. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 9: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) includes the additional elements of sending the first, second or aggregated exploration results to the third RL agent in response to a determination, based on second information representative of exploration capabilities received from a third RL agent, that the third RL agent has not enough capabilities to contribute to the collaborative RL. These judicial exceptions are not integrated into a practical application. The additional element(s) of sending the first, second or aggregated exploration results to the third RL agent in response to a determination, based on second information representative of exploration capabilities received from a third RL agent, that the third RL agent has not enough capabilities to contribute to the collaborative RLrecite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of sending the first, second or aggregated exploration results to the third RL agent in response to a determination, based on second information representative of exploration capabilities received from a third RL agent, that the third RL agent has not enough capabilities to contribute to the collaborative RLrecite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible. Regarding Claim 10: The limitation(s), as drafted, are a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually configuring the first RL agent with at least one reporting rule for exploration results of the first exploration process performed by the first RL agent. These judicial exceptions are not integrated into a practical application. In particular, the claims do not recite any additional elements. Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, no additional elements are cited. Accordingly, the claim is not patent eligible. Regarding Claim 11: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually wherein the at least one RL agent includes a plurality of RL agents in respective distinct nodes, the method comprising: selecting, based on the information representative of exploration capabilities received from the plurality of network RL agents, RL agents having capabilities to contribute to the collaborative RL, the selected RL agents including the first RL agent and the second RL agent; configuring, based on the information representative of exploration capabilities received from the second RL agent, the second agent with second exploration tasks of the second exploration process to be performed by the second RL agent; receiving the second exploration results from the second RL agent; aggregating exploration results of exploration processes performed by the selected RL agents to generate aggregated exploration results, the aggregated exploration results including the first exploration results and the second exploration results. The limitation(s) includes the additional elements of receiving the second exploration results from the second RL agent. These judicial exceptions are not integrated into a practical application. The additional element(s) of receiving the second exploration results from the second RL agentrecite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of receiving the second exploration results from the second RL agentrecite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible. Regarding Claim 12: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) includes the additional elements of sending, to the at least one RL agent, a request for obtaining information representative of exploration capabilities of the concerned RL agent. These judicial exceptions are not integrated into a practical application. The additional element(s) of sending, to the at least one RL agent, a request for obtaining information representative of exploration capabilities of the concerned RL agentrecite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of sending, to the at least one RL agent, a request for obtaining information representative of exploration capabilities of the concerned RL agentrecite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible. With respect to Claim 14: Step 1: Claim 14 is a apparatus claim. Therefore, claim 14 is directed to either a process, machine, manufacture or composition of matter. Step 2A Prong 1: performing the exploration process based on the configuration information to generate first exploration results (mental process – user can manually perform the exploration process based on the configuration information to generate first exploration results) Step 2A Prong 2: This judicial exception is not integrated into a practical application. Additional elements: at least one processor (mere instructions to apply the exception using a generic computer component) at least one memory storing instructions (mere instructions to apply the exception using a generic computer component) sending, to a RL management function, information representative of exploration capabilities of the first RL agent (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g)) receiving configuration information for exploration tasks of an exploration process to be performed by the first RL agent to contribute to a collaborative RL (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))) sending, to at least one of the RL management function and a second RL agent, the first exploration results (Adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g))) Step 2B: The claim does not include additional elements considered individually and in combination that are sufficient to amount to significantly more than the judicial exception. Additional elements: at least one processor (mere instructions to apply the exception using a generic computer component) at least one memory storing instructions (mere instructions to apply the exception using a generic computer component) sending, to a RL management function, information representative of exploration capabilities of the first RL agent (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer) receiving configuration information for exploration tasks of an exploration process to be performed by the first RL agent to contribute to a collaborative RL (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer) sending, to at least one of the RL management function and a second RL agent, the first exploration results (MPEP 2106.05(d)(II) indicate that merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed step is well-understood, routine, conventional activity is supported under Berkheimer) Conclusion: The claim is not patent eligible. Regarding Claim 15: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) encompasses the user manually performing the exploration process using the RL strategy. The limitation(s) includes the additional elements of receiving, from the RL management function, a RL strategy that defines an algorithm to determine one or more actions to be taken by the first RL agent in the context of the first exploration process. These judicial exceptions are not integrated into a practical application. The additional element(s) of receiving, from the RL management function, a RL strategy that defines an algorithm to determine one or more actions to be taken by the first RL agent in the context of the first exploration processrecite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of receiving, from the RL management function, a RL strategy that defines an algorithm to determine one or more actions to be taken by the first RL agent in the context of the first exploration processrecite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible. Regarding Claim 16: The limitation(s), as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation(s) in the mind. That is, other than the additional elements, nothing in the claim limitation(s) precludes the step from practically being performed in the mind. The limitation(s) includes the additional elements of receiving, from the RL management function, at least one reporting rule for exploration results of the exploration process, wherein sending the first exploration results is performed according to the reporting rule. These judicial exceptions are not integrated into a practical application. The additional element(s) of receiving, from the RL management function, at least one reporting rule for exploration results of the exploration process, wherein sending the first exploration results is performed according to the reporting rulerecite adding insignificant extra-solution activity to the judicial exception - see MPEP 2106.05(g). Accordingly, this does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of receiving, from the RL management function, at least one reporting rule for exploration results of the exploration process, wherein sending the first exploration results is performed according to the reporting rulerecite merely “storing and retrieving information in memory” or “receiving or transmitting data over a network” is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim) (MPEP 2106.05(d)(II)). Thereby, a conclusion that the claimed storing step is well-understood, routine, conventional activity is supported under Berkheimer. Accordingly, the claims are not patent eligible. Claim Rejections - 35 USC § 102 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. Claim(s) 1-16 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Liu et al. (hereinafter Liu), Multi-Agent Reinforcement Learning-Based Coordinated Dynamic Task Allocation for Heterogenous UAVs. Regarding Claim 1, Liu discloses a method for use by a reinforcement learning, RL, management function in a first node, the method comprising: receiving, from at least one RL agent, information representative of exploration capabilities of the considered RL agent [“heterogeneous UAVs may have different capabilities; different tasks performed by UAVs will have different benefits or costs” §I ¶1; “If (state(Responser)=idle) Then the Responser UAVs send the types and the capability it contains, the distance cost and the Q value” §IV ¶4; “Proposer receives the ResponserMessage” §IV ¶6]; configuring, based on first information representative of exploration capabilities received from a first RL agent, the first RL agent with first exploration tasks of a first exploration process to be performed by the first RL agent to contribute to a collaborative RL [“If(the new task source requirement is satisfied with the UAVs that are in idle state) Then the Proposer will select the UAVs with higher Q value to participate in the task” §IV ¶6; “The Proposer sends a contract to the selected UAVs. The state of the selected UAVs will be changed to commited.” Algorithm 1 line 15]; receiving, from the first RL agent, first exploration results of the first exploration process [“Receive reward of the Proposer and Responser: Rp, Rr and observe new state: st+1p , st+1r” Algorithm 1 line 16]; processing the first exploration results and second exploration results of a second exploration process performed by a second RL agent to contribute to the collaborative RL [“for agent i = 1 to Number of agents (N) do” Algorithm 1 lines 19-35]. Regarding Claim 2, Liu discloses the method of claim 1. Liu further discloses wherein the first RL agent is configured with the first exploration tasks in response to a determination, based on the first information representative of exploration capabilities received from the first RL agent, that the first RL agent has capabilities to contribute to the collaborative RL [“If (state(Responser)=idle) Then the Responser UAVs send the types and the capability it contains, the distance cost and the Q value” §IV ¶4]. Regarding Claim 3, Liu discloses the method of claim 1. Liu further discloses comprising: selecting, based on the received information representative of exploration capabilities, a RL strategy that defines an algorithm to determine one or more actions to be taken by an RL agent in the context of an exploration process [“If(the new task source requirement is satisfied with the UAVs that are in idle state) Then the Proposer will select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2).” §IV ¶7]. configuring the first RL agent with the selected RL strategy [“The Proposer sends a contract to the selected UAVs, which will participate in the new task. The state of the selected UAVs will be changed to committed.” §IV ¶9]. Regarding Claim 4, Liu discloses the method of claim 1. Liu further discloses wherein the second exploration process is executed by the second RL agent in the first node [“tasks may require coordination among several UAVs” §I ¶1], the method comprising executing, by the second RL agent in the first node [“tasks may require coordination among several UAVs” §I ¶1], the second exploration process to generate the second exploration results [“for agent i = 1 to Number of agents (N) do” Algorithm 1 lines 19-35]. Regarding Claim 5, Liu discloses the method of claim 1. Liu further discloses wherein the second exploration process is executed by the second RL agent in a second node distinct from the first node [“a set of UAVs performing specific tasks” §II.A ¶1; Fig. 1], wherein the method comprises receiving the second exploration results from the second RL agent [“Receive reward of the Proposer and Responser: Rp, Rr and observe new state: st+1p , st+1r” Algorithm 1 line 16]. Regarding Claim 6, Liu discloses the method of claim 1. Liu further discloses wherein: processing the first exploration results and second exploration results is performed to configure further exploration tasks for the first RL agent [“Update target network parameters for each agent i:” Algorithm 1 line 34]. Regarding Claim 7, Liu discloses the method of claim 1. Liu further discloses wherein: processing the first exploration results and second exploration results includes determining whether a convergence criterion is met for the collaborative RL [“select corresponding UAVs with higher Q-value on the new task, then recalculate whether the original task meets the requirements.” §IV ¶8; “Convergence: The key point for the convergence of the Propose and Response network is whether their critic network (Q network) could approximate the optimal action-value function,” §III.D.3 ¶1];. wherein the method comprises configuring the first agent with further exploration tasks based on a determination that the convergence criterion is not met for the collaborative RL [“After recalculating the original task requirement, whether there are new tasks appears is classified. This process will continue until the requirement of the new tasks is satisfied.” §IV ¶10 ]. Regarding Claim 8, Liu discloses the method of claim 1. Liu further discloses wherein: processing the first exploration results and second exploration results includes aggregating the first exploration results and the second exploration results to generate aggregated exploration results [“In implementing of DQN for multi-UAV task allocation, the learning process is accelerated by sharing network parameters among UAVs. In the distributed execution, the UAV evaluates the reward of the new task through its neural network and responds to the Proposer. Although the UAVs learn from one network and the parameters of the decision-making network are the same, they can behave differently because of the different state of each UAV. To update the network parameters efficiently, the gradient descent method is used to train the network and use the dual network structure to improve the convergence performance of the algorithm.” §III.C ¶3]; determining whether a convergence criterion is met for the collaborative RL based on the aggregated exploration results [“to improve the convergence performance of the algorithm.” §III.C ¶3]. Regarding Claim 9, Liu discloses the method of claim 5. Liu further discloses comprising: sending the first, second or aggregated exploration results to the third RL agent in response to a determination, based on second information representative of exploration capabilities received from a third RL agent, that the third RL agent has not enough capabilities to contribute to the collaborative RL [“The Proposer receives the ResponserMessage. If(the new task source requirement is satisfied with the UAVs that are in idle state) Then the Proposer will select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2). Else: According to (1) and (2), it will select corresponding UAVs with higher Q-value on the new task, then recalculate whether the original task meets the requirements.” §IV ¶¶6-8; “The Proposer receives the ResponserMessage, and select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2).” Algorithm 1 line 14]. Regarding Claim 10, Liu discloses the method of claim 1. Liu further discloses comprising: configuring the first RL agent with at least one reporting rule for exploration results of the first exploration process performed by the first RL agent [“If (state(Responser)=idle) Then the UAVs Responser send the types and the capability it contains, the distance cost and the Q value” §IV ¶4]. Regarding Claim 11, Liu discloses the method of claim 1. Liu further discloses wherein the at least one RL agent includes a plurality of RL agents in respective distinct nodes [“a set of UAVs performing specific tasks” §II.A ¶1; Fig. 1], the method comprising: selecting, based on the information representative of exploration capabilities received from the plurality of network RL agents, RL agents having capabilities to contribute to the collaborative RL, the selected RL agents including the first RL agent and the second RL agent [“The Proposer receives the ResponserMessage. If(the new task source requirement is satisfied with the UAVs that are in idle state) Then the Proposer will select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2).” §IV ¶¶6-7]; configuring, based on the information representative of exploration capabilities received from the second RL agent, the second agent with second exploration tasks of the second exploration process to be performed by the second RL agent [“The Proposer sends a contract to the selected UAVs, which will participate in the new task.” §IV ¶9; “The Proposer sends a contract to the selected UAVs. The state of the selected UAVs will be changed to commited.” Algorithm 1 line 15]; receiving the second exploration results from the second RL agent [“Receive reward of the Proposer and Responser: Rp, Rr and observe new state: st+1p , st+1r” Algorithm 1 line 16]; aggregating exploration results of exploration processes performed by the selected RL agents to generate aggregated exploration results, the aggregated exploration results including the first exploration results and the second exploration results [“In implementing of DQN for multi-UAV task allocation, the learning process is accelerated by sharing network parameters among UAVs. In the distributed execution, the UAV evaluates the reward of the new task through its neural network and responds to the Proposer. Although the UAVs learn from one network and the parameters of the decision-making network are the same, they can behave differently because of the different state of each UAV. To update the network parameters efficiently, the gradient descent method is used to train the network and use the dual network structure to improve the convergence performance of the algorithm.” §III.C ¶3]. Regarding Claim 12, Liu discloses the method of claim 1. Liu further discloses comprising: sending, to the at least one RL agent, a request for obtaining information representative of exploration capabilities of the concerned RL agent [“When the UAV denoted as Proposer finds a new task, it broadcast the task announcement to the Responsers.” §IV ¶2]. Claim 13 is rejected on the same grounds as claim 1. Lui further discloses at least one processor [“a computer with Intel i5-8500 and 24 GB of memory” §V.A ¶2]; and at least one memory storing instructions [“a computer with Intel i5-8500 and 24 GB of memory” §V.A ¶2]. Regarding Claim 14, Liu discloses an apparatus comprising - at least one processor [“a computer with Intel i5-8500 and 24 GB of memory” §V.A ¶2]; - at least one memory storing instructions [“a computer with Intel i5-8500 and 24 GB of memory” §V.A ¶2] that, when executed by the at least one processor, cause the apparatus to perform: sending, to a RL management function, information representative of exploration capabilities of the first RL agent [“If (state(Responser)=idle) Then the UAVs Responser send the types and the capability it contains, the distance cost and the Q value” §IV ¶4]; receiving configuration information for exploration tasks of an exploration process to be performed by the first RL agent to contribute to a collaborative RL [“The Proposer sends a contract to the selected UAVs, which will participate in the new task.” §IV ¶9]; performing the exploration process based on the configuration information to generate first exploration results [“Receive reward of the Proposer and Responser: Rp, Rr and observe new state: st+1p , st+1r” Algorithm 1 line 16, 19-33]; sending, to at least one of the RL management function and a second RL agent, the first exploration results [“The Proposer receives the ResponserMessage. If(the new task source requirement is satisfied with the UAVs that are in idle state) Then the Proposer will select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2). Else: According to (1) and (2), it will select corresponding UAVs with higher Q-value on the new task, then recalculate whether the original task meets the requirements.” §IV ¶¶6-8; “The Proposer receives the ResponserMessage, and select the UAVs with higher Q value to participate in the task under the constraints of (1) and (2).” Algorithm 1 line 14]. Regarding Claim 15, Liu discloses the apparatus of claim 14. Liu further discloses wherein the apparatus is further caused to perform: receiving, from the RL management function, a RL strategy that defines an algorithm to determine one or more actions to be taken by the first RL agent in the context of the first exploration process [“sends the task information to the response UAV” §II.A ¶2; Fig. 1]; performing the exploration process using the RL strategy [“The response UAVs make their own corresponding evaluations based on the importance, urgency of the new task and the distance from the task, then send the evaluation results and resource information to the request UAV” §II.A ¶3; Fig. 1]. Regarding Claim 16, Liu discloses the apparatus of claim 14. Liu further discloses wherein the apparatus is further caused to perform: receiving, from the RL management function, at least one reporting rule for exploration results of the exploration process, wherein sending the first exploration results is performed according to the reporting rule [“sends the task information to the response UAV” §II.A ¶2; “The response UAVs make their own corresponding evaluations based on the importance, urgency of the new task and the distance from the task, then send the evaluation results and resource information to the request UAV” §II.A ¶3; Fig. 1]. Examiner’s Note The Examiner respectfully requests of the Applicant in preparing responses, to fully consider the entirety of the reference(s) as potentially teaching all or part of the claimed invention. It is noted, REFERENCES ARE RELEVANT AS PRIOR ART FOR ALL THEY CONTAIN. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (CCPA 1968)). A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art, including non-preferred embodiments (see MPEP 2123). The Examiner has cited particular locations in the reference(s) as applied to the claim(s) above for the convenience of the Applicant. Although the specified citations are representative of the teachings of the art and are applied to the specific limitations within the individual claim(s), typically other passages and figures will apply as well. Additionally, any claim amendments for any reason should include remarks indicating clear support in the originally filed specification. Conclusion Any prior art made of record and not relied upon is considered pertinent to Applicant's disclosure. Applicant is reminded that in amending in response to a rejection of claims, the patentable novelty must be clearly shown in view of the state of the art disclosed by the references cited and the objections made. Applicant must also show how the amendments avoid such references and objections. See 37 CFR §1.111(c). Additionally when amending, in their remarks Applicant should particularly cite to the supporting paragraphs in the original disclosure for the amendments. The following references were found during the examination of this patent application and were found to be relevant to patentability. Applicant is advised to review these references prior to responding to this Office action. Hall et al. (Adaptive Average Exploration in Multi-Agent Reinforcement Learning) discloses improving Multi-Agent Reinforcement Learning performance in the StarCraft II environment with respect to faster training times, greater stability, and higher win ratios. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT H BEJCEK II whose telephone number is (571)270-3610. The examiner can normally be reached Monday - Friday: 9:00am - 5:00pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /R.B./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/ Supervisory Patent Examiner, Art Unit 2148
Read full office action

Prosecution Timeline

May 16, 2024
Application Filed
Oct 19, 2026
Non-Final Rejection mailed — §101, §102, §112 (current)

Precedent Cases

Applications granted by this same examiner with similar technology

Patent 12737689
Scale-Permuted Machine Learning Architecture
2y 1m to grant Granted Sep 15, 2026
Patent 12737592
PARTIAL ACTIVATION OF MULTIPLE PATHWAYS IN NEURAL NETWORKS
1y 10m to grant Granted Sep 15, 2026
Patent 12651203
SYSTEM AND METHOD FOR TRAINING TEACHER MODEL BASED ON STRUCTURE OF STUDENT MODEL
4y 9m to grant Granted Jun 09, 2026
Patent 12639600
DATABASE UTILIZING SPATIAL PROBABILITY MODELS FOR DATA COMPRESSION
3y 7m to grant Granted May 26, 2026
Patent 12619909
STUDENT PERFORMANCE EVALUATION METHOD AND SYSTEM BASED ON AI IDENTIFICATION DATA
4y 2m to grant Granted May 05, 2026
Study what changed to get past this examiner. Based on 5 most recent grants.

Strategy Recommendation AI-generated — please review before filing

Get a prosecution strategy drawn from examiner precedents, rejection analysis, and claim mapping.
Typically takes 5-10 seconds — AI-generated, attorney review required before filing

Prosecution Projections

1-2
Expected OA Rounds
63%
Grant Probability
85%
With Interview (+22.5%)
3y 9m (~1y 3m remaining)
Median Time to Grant
Low
PTA Risk
Based on 258 resolved cases by this examiner. Grant probability derived from career allowance rate.

Sign in with your work email

Enter your email to receive a magic link. No password needed.

Personal email addresses (Gmail, Yahoo, etc.) are not accepted.

Free tier: 3 strategy analyses per month