Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2 and 4-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining a control value such that at least one limit value of a measurement variable of the supply network is violated in a time range, wherein the time range is determined in such a way that protective devices of the supply network do not trip”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“wherein the neural network uses measured values associated with a supply network with network to determine a plurality of control values for controlling the supply network”
“training the neural network using the calculated control value and the acquired associated measured value”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“acquiring at least one measured value associated with the control value”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” and “insignificant extra-solution activity”. Specifically, the acquiring limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the supply network comprises an electricity network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein learning takes place during operation of the supply network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the time range is less than or equal to one minute”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“wherein avoided violations of limit values are used as quality parameters of the reinforced learning”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 2.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the measured values comprise active powers, reactive powers, angles and/or currents of the respective phase at respective network nodes of the electricity network and/or in the respective lines of the electricity network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 2.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“feeding changes in active powers and/or reactive powers into the electricity network and/or out due to the control values”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 8.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“wherein the control values are transmitted using a ripple control signal and/or telecontrol signal to a smart meter and/or to a controllable mains transformer and/or to converters of photovoltaic systems and/or to charging stations”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the transmission limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein
PNG
media_image1.png
30
398
media_image1.png
Greyscale
is used as the reward function, where Gk indicates a measurement variable and Gkmax indicates its associated limit value, ΔP indicates a change in the active power, ΔQ indicates a change in the reactive power and st = (P1,t, P2,t, …, PN,t, Q1,t, Q2,t, …, QN,t)T indicates a state of the electricity network at the time t.
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 2.
Step 2B Analysis: See corresponding analysis of claim 2.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the learning takes place in such a way that the reward function r(st) is maximized”
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 10.
Step 2B Analysis: See corresponding analysis of claim 10.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“wherein the neural network determines the vector at = (ΔP1,t, …, ΔPF,t, ΔQ1,t, …, ΔQF,t)T as control values”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measures values are calculated by means of a state estimation”
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 2.
Step 2B Analysis: See corresponding analysis of claim 2.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2 and 4-9 are rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (U.S. Patent Publication No. 2020/0327411) (“Shi”) in view of Radhakrishnan et al. (Improving primary frequency response in networked microgrid operations using multilayer perceptron-driven reinforcement learning) (“Radhakrishnan”).
Regarding claim 1, Shi teaches a method for the reinforced learning of an artificial neural network (Shi [0007] “FIG. 1 shows an exemplary Voltage profile zone definition for training DRL agents.”; [0008] “FIG. 2 shows an exemplary overview of training and implementation of the DRL agents.”; [0067] “DRL is essentially developed from the classic reinforcement learning technique when combining with deep neural network (DNN) that consists of many layers.” Shi provides training and implementing DRL agents, which combine deep neural networks with reinforcement learning techniques, corresponding to the reinforced learning of an artificial neural network.), wherein the neural network uses measured values associated with a supply network to determine a plurality of control values for controlling the supply network (Shi [0020] “This system provides for training DRL agents for providing data-driven, real-time and autonomous control strategies for regulating voltage profiles in a power grid..”; [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures [control values] once voltage violations are detected... FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.”; [0039] “For the purpose of coordinated voltage control, states [neural network inputs] are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches [measured values associated with a supply network] that can be directly provided by EMS or WAMS systems.” Shi provides training a DRL agent corresponding to the neural network, which uses state values comprising voltage values of a power grid to determine corrective control measures once voltage violations are detected, wherein state vectors comprising voltage values are used an input to the neural network, as shown in Figure 3..), the method comprising: determining a control value (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures [determined control values] once voltage violations are detected.”; [0033] “However, due to variations in system loads, renewable generation and contingencies, once voltage issues occur, the DRL agent starts to take actions selected from an action space in order to fix the voltage issues. For each iteration of applied control actions, the control performance is calculated in terms of reward values.” Shi provides the DRL agent determining control actions regarding voltage levels in the power grid, as shown in Figure 2, corresponding to determining a control value.) such that at least one limit value of a measurement variable of the supply network is violated in a time range (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected. It is worth mentioning voltage limits considered can be adjusted/narrowed to make the proposed framework work for preventive control. Constraints considered in this study include full AC power flow equations, generation limits and voltage limits [limit value of a measurement variable of the supply network]. FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.”; Figure 1 “Violation Zone”; [0061] “The formulation of DRL for voltage control is flexible as it can intake multiple control objectives and consider various security constraints, especially time-series constraints [time range].” Shi provides a voltage violation zone for training of the DRL agent neural network, as shown by the “Violation zone” in Figure 1, wherein the voltage value corresponds to the measurement variable of the supply network which is violated in a time range, and wherein the time-series constraints corresponds to the time range.) …acquiring at least one measured value associated with the control value (Shi Figures 1 and 2; [0023] “For the problem of coordinated voltage control, a 4-tuple can be used to formulate the MDP, (S, A, P.sub.a, R.sub.a), where S is a vector of system states, including voltage magnitudes and phase angles across the system or areas of interest”; [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems. To maintain consistency of different inputs and outputs with various units when training DRL agents, the batch normalization technique is applied.” Shi provided the DRL agent acquiring voltage magnitude of the power grid to provide control signals, corresponding to acquiring at least one measured value associated with the control value.); and training the neural network using the calculated control value and the acquired associated measured value (Shi [0031] “Without loss of generality, this system trains effective DRL agents [training the neural network] for providing prompt corrective control measures once voltage violations are detected… Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected.”; [0033] “To train effective agents, massive representative operating conditions need to be collected or created, including random load changes, variations in renewable generation, generation dispatch patterns, major topology changes due to maintenance and contingencies.”; [0067] “Through such massive interactions, the DRL agent keeps optimizing its policy to maximize the accumulated rewards. In this way, the DRL agent will gradually master the control problem after a certain period of training.” Shi provides training the DRL agent neural network using the acquired voltage values and the control values through policy optimization, corresponding to training the neural network with the acquired measured voltage values and calculated control actions.).
Shi fails to explicitly teach …wherein the time range is determined in such a way that protective devices of the supply network do not trip.
However, Radhakrishnan teaches …wherein the time range is determined in such a way that protective devices of the supply network do not trip (Radhakrishnan 4.4 Performance of the RL-based controller “In Scenario 10, shown in Fig. 5, the system contains grid friendly appliance controllers [12] that stop operations when the frequency drops below 59 Hz. In the scenario, without CVR controllers, a communication delay causes the GFAs to be tripped as the PV Microgrid 2 (G3) output is not increased in time to avoid this. Later, when the PV in Microgrid 2 increases the output, an over-frequency event is caused at 6.6 s [time range]. On the other hand, the RL-based CVR controllers greatly improve the frequency response, maintaining the frequency closer to the desired 60 Hz, completely avoiding the GFA-level load shedding, and causing no over-frequency event when the PV in Microgrid 2 increases its output. Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off [protective devices of the supply network do not trip] due to IEEE Std. 1547. The large voltage deviation is also avoided as the CVR controller triggers a drop in terminal voltage at generatorG4 temporarily reducing load on the system till the under frequency event is stabilized.” Radhakrishnan provides the Reinforcement learning controller recovering the system fast enough to avoid inverter trip-off when the frequency drops below 59.5 Hz at 6.6s, as shown in Figure 5, corresponding determining a time range in such a way that protective devices do not trip.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 2, Shi in view of Radhakrishnan teaches wherein the supply network comprises an electricity network (Shi [0031] “FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.” Shi provides a power grid and corresponding bus voltages, corresponding to an electricity network.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 4, Shi in view of Radhakrishnan teaches wherein learning takes place during operation of the supply network (Shi [0067] “A general interaction process between the agent and the environment in DRL is presented in FIG. 3. After receiving the current states from the environment, a DRL agent generates a corresponding action using its policy; then, the environment provides the next state (s′) and the corresponding reward (r′) for the executed action. Through such massive interactions, the DRL agent keeps optimizing its policy to maximize the accumulated rewards [learning during operation]. In this way, the DRL agent will gradually master the control problem after a certain period of training.”; [0068] “FIG. 3 illustrates a sample interaction between Agent and Environment in reinforcement learning [learning during operation], while the main flowchart for training DRL Agents for coordinated voltage control is shown in FIG. 4, consisting of four major steps:” Shi provides the Agent interacting with the environment during learning, as shown in Figure 3, wherein learning takes place during operation of the electric supply network, as shown in Figure 3, which uses power values from the environment during the reinforcement training/learning to optimize policy and maximize accumulated rewards.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 5, Shi in view of Radhakrishnan teaches wherein the time range is less than or equal to one minute (Radhakrishnan “In Scenario 10, shown in Fig. 5, the system contains grid friendly appliance controllers [[12]] that stop operations when the frequency drops below 59 Hz. In the scenario, without CVR controllers, a communication delay causes the GFAs to be tripped as the PV Microgrid 2 (G3) output is not increased in time to avoid this. Later, when the PV in Microgrid 2 increases the output, an over-frequency event is caused at 6.6 s [time range is less than or equal to one minute]… Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off due to IEEE Std. 1547.” Radhakrishnan provides the trip avoidance of Scenario 10, as shown in Fig. 5, which has a time range of 8 seconds on the x-ais of the figure, and wherein the signal response is recorded at the 6.6s time range.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 6, Shi in view of Radhakrishnan teaches wherein avoided violations of limit values are used as quality parameters of the reinforced learning (Radhakrishnan 3.2 Problem formulation “The implementation of the RL starts with the definition of the MDP, with the four parameters… The reward function R is Equation (5), where the first term signifies the tracking performance [quality parameters] (maintain frequency at 60Hz at each discrete time) and the second term is to reduce oscillations during state transitions.”; 4.4 Performance of the RL-based controller “On the other hand, the RL-based CVR controllers greatly improve the frequency response, maintaining the frequency closer to the desired 60 Hz, completely avoiding the GFA-level load shedding, and causing no over-frequency event when the PV in Microgrid 2 increases its output. Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off [avoided violations of limit values] due to IEEE Std. 1547.” Radhakrishnan provides avoiding violations of limit values by avoiding inverter trip-off and causing no over-frequency event based on frequency values, wherein the frequency values are used in the reward function to track performance of the reinforcement learning algorithm, wherein the frequency values in the reward function for tracking performance correspond to the quality parameters.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 7, Shi in view of Radhakrishnan teaches wherein the measured values comprise active powers, reactive powers, angles and/or currents of the respective phase at respective network nodes of the electricity network and/or in the respective lines of the electricity network (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected. It is worth mentioning voltage limits considered can be adjusted/narrowed to make the proposed framework work for preventive control. Constraints considered in this study include full AC power flow equations, generation limits and voltage limits. FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus [lines of the electricity network] voltages of a power grid within a predefined zone before and after a disturbance.”; [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems.” Shi provides phase angles, and active and reactive power flows and voltages from the power grid, wherein the power grid includes bus voltages that correspond to the respective lines of the electricity network).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 2.
Regarding claim 8, Shi in view of Radhakrishnan teaches further comprising feeding changes in active power and/or reactive powers into the electricity network and/or out due to the control values (Shi [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems.”; [0078] “FIG. 6 shows an exemplary power grid control system. The system includes a power grid measurement system with SCADA and WAMS. The states are provided to a power grid controller with DQN/DDPG agent and a prioritized replay buffer. The control signals are then provided as control variables for generator setting, transformer tap setting, shunt switching setting, topology adjustments, among others.” Shi provides active and reactive power flows as states, which are provided to a power grid controller to send control signals as control variables for generator setting, transformer tap setting, shunt switching setting, topology adjustments, corresponding to feeding changes in active and reactive powers in our out of the electricity network due to the control values.)
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 2.
Regarding claim 9, Shi in view of Radhakrishnan teaches wherein the control values are transmitted using a ripple control signal and/or telecontrol signal to a smart meter and/or to a controllable mains transformer and/or to converters of photovoltaic systems and/or to charging stations (Shi [0078] “FIG. 6 shows an exemplary power grid control system. The system includes a power grid measurement system with SCADA and WAMS. The states are provided to a power grid controller with DQN/DDPG agent and a prioritized replay buffer. The control signals [telecontrol signal] are then provided as control variables for generator setting, transformer tap setting [a controllable mains transformer], shunt switching setting, topology adjustments, among others.” Shi provides transmitting control settings for transformer tap setting, corresponding to transmitting control values to a controllable mains transformer using a telecontrol signal.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 8.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (U.S. Patent Publication No. 2020/0327411) (“Shi”) in view of Radhakrishnan et al. (Improving primary frequency response in networked microgrid operations using multilayer perceptron-driven reinforcement learning) (“Radhakrishnan”) in further view of Shi et al. (U.S. Patent Publication No. 2020/0119556) (“Shi 2”).
Regarding claim 13, Shi in view of Radhakrishnan teaches the method as claimed in claim 2, but fails to teach wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measured values are calculated by means of a state estimation.
However, Shi 2 teaches wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measured values are calculated by means of a state estimation (Shi 2 [0033] “An autonomous voltage control schema for grid operation using deep reinforcement learning (DRL) is detailed next.”; [0065] “To mimic real power system environment, a commercial power grid simulator is adopted, which is equipped with function modules such as power flow, dynamic simulation, contingency analysis, state estimation and so on. In this embodiment, only the AC power flow module, as environment, is applied to interact with the DRL agent.” Shi 2 provides deep reinforcement learning including mimicking real power systems with a simulated power grid including state estimation.).
Shi, Radhakrishnan, and Shi 2 are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to electric supply networks. Therefore it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan with the above teachings of Shi 2. Doing so would allow for algorithms that are proven to be effective in various real-world control problems in highly dynamic and stochastic environments. (Shi 2 [0033] " In one embodiment, an innovative and promising approach of training DRL agents with improved RL algorithms provides data-driven, real-time and autonomous control strategies by coordinating and optimizing available controllers to regulate voltage profiles in a power grid, where the AVC problem is formulated as Markov decision process (MDP) so that it can take full advantages of state-of-the-art reinforcement learning (RL) algorithms that are proven to be effective in various real-world control problems in highly dynamic and stochastic environments.”).
Response to Arguments
Regarding the objection to the claims and the rejection of the claims under 35 U.S.C. 112, Applicant’s cancellation of claim 3 overcomes the objection and 112b rejection previously introduced in the Non-Final Rejection, dated June 17, 2026.
Regarding the rejection of the claims under 35 U.S.C. 101, Applicant asserts that the pending claims are directed to a concrete technical process for reinforced learning of an artificial neural network that controls a supply network, and not to a mental process or mathematical concept (“Remarks”, Pg. 5). Applicant further asserts that the “determining a control value…” limitation is not practically performed in the human mind, and is instead directed to control of physical network behavior, constrained by the operation of protective devices (“Remarks”, Pgs. 5-6).
However, under the broadest reasonable interpretation, the “determining…” limitation does correspond to a mental process. For example, a “determination” of a value is mentally performable, by simply making an estimation of a value that meets the claimed constraints. The claimed step of making the determination itself does not explicitly require control of physical network behavior constrained by the operation of protective devices, as a determination of a value for a control system can still be determined mentally. Therefore, as written, the claimed limitation does correspond to a mental process.
Applicant further asserts that the claimed steps use the physical response of the supply network to improve training of the neural network that controls that supply network, which integrates the claims into a practical application (“Remarks”, Pg. 6). Applicant further asserts that as an ordered combination, the claim is tied to the technical operation of the supply network (“Remarks”, Pg. 7).
Regarding the asserted improvement, even if the claims did recite an improvement, it would be in the abstract idea of “determining a control value…”. As recited in the MPEP, an improvement in the abstract idea itself is not an improvement in technology. MPEP 2106.05(a). Therefore, even if the claims did recite an improvement, it would be an improvement in the abstract idea of determining a control value.
Regarding the claim as an ordered whole, the claim as an ordered whole is directed to the mentally performable process of determining a control value. Nothing in the claim recites significantly more than this. Therefore, the claims as written, remain rejected under 35 U.S.C. 101.
Regarding the rejection of the claims under 35 U.S.C. 103, Applicant firstly asserts that the applied reference Shi does not disclose the ordered protocol disclosed in claim 1, and specifically that Shi does not teach determining a control value for the purpose of intentionally causing a short-term limit value-value violation, keeping that violation within a trip-free time range, acquiring the measured response cause by the deliberate intervention, and training the neural network using the resulting paired data (“Remarks”, Pgs. 9-10).
However, the claim does not recite “determining a control value for the purpose of intentionally causing a short-term limit value-value violation, keeping that violation within a trip-free time range, acquiring the measured response cause by the deliberate intervention, and training the neural network using the resulting paired data”, rather, the claim recites “determining a control value such that at least one limit value of a measurement variable of the supply network is violated in a time range”, which is taught by Shi in paragraphs [0031] and [0033]. For example, Shi teaches a “Violation zone”, as shown in Figure 1, which shows violated voltage values, including “voltage limits” corresponding to a limit value of a measurement variable violated in a time range, as recited in the independent claim. Further, Shi teaches determining control measures therefrom for training reinforcement learning agents, including determining control actions regarding voltage levels in the power grid, corresponding to the determined control values. Therefore, Shi does teach “determining a control value such that at least one limit value of a measurement variable of the supply network is violated in a time range”.
Further, Shi is not relied upon to teach “wherein the time range is determined in such a way that protective devices of the supply network do not trip”, rather, the applied reference Radhakrishnan is relied upon, which teaches the limitation in Section 4.4. For example, Radhakrishnan teaches in that section the learning controller recovers the system fast enough to avoid inverter trip-off when the frequency drops below 59.5 Hz at the 6.6s mark, as shown in Figure 5, thus corresponding to the time range determined in such a way that the protective device (i.e., the inverter), does not trip. Therefore Radhakrishnan teaches the claimed limitation.
Applicant further asserts that the applied reference Radhakrishnan does not teach “deliberately selecting a control value to create a controlled limit-value violation as a training intervention, then using the measured response associated what that deliberate intervention to train the neural network” (“Remarks”, Pg. 10).
Applicant further asserts that the reasons to combine the applied references Shi and Radhakrishnan are insufficient (“Remarks”, Pgs. 10-11).
However, the claim does not recite “deliberately selecting a control value to create a controlled limit-value violation as a training intervention, then using the measured response associated what that deliberate intervention to train the neural network”, rather, the claim recites “wherein the time range is determined in such a way that protective devices of the supply network do not trip”, which as discussed above, is taught by the applied references.
Further, the combination of references is proper, as both are in the same field of artificial intelligence and more specifically applied to power grid operations or electric supply networks. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan "Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.").
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KURT NICHOLAS PRESSLY whose telephone number is (703)756-4639. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KURT NICHOLAS PRESSLY/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights