Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDSs) submitted on December 14, 2023 and March 29, 2024 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Objections
Claim 3 is objected to because of the following informalities: “wherein the supply network is comprises as a medium-voltage and/or low voltage network” should read “wherein the supply network comprises a medium-voltage and/or low voltage network”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 3 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “medium-voltage network and/or low-voltage network” in claim 3 is a relative term which renders the claim indefinite. The term “medium-voltage network and/or low-voltage network” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. The “voltage network” has been rendered indefinite by the use of the terms “medium” and “low”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 1 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“determining a control value such that at least one limit value of a measurement variable of the supply network is violated in a time range, wherein the time range is determined in such a way that protective devices of the supply network do not trip”
As drafted, under their broadest reasonable interpretations, cover mental processes, i.e., concepts performed in the human mind (including an observation, evaluation, judgement, opinion). The above limitations in the context of this claim correspond to mental processes, e.g., evaluation and judgement with assistance of pen and paper.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)) and insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“wherein the neural network uses measured values associated with a supply network with network to determine a plurality of control values for controlling the supply network”
“training the neural network using the calculated control value and the acquired associated measured value”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
The limitations:
“acquiring at least one measured value associated with the control value”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply” and “insignificant extra-solution activity”. Specifically, the acquiring limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Mere instructions to apply and insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 2,
Claim 2 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 2 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the supply network comprises an electricity network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 3,
Claim 3 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 3 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 2.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the supply network is comprises as a medium-voltage and/or low voltage network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 4,
Claim 4 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 4 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein learning takes place during operation of the supply network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 5,
Claim 5 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 5 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the time range is less than or equal to one minute”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 6,
Claim 6 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 6 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 1.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“wherein avoided violations of limit values are used as quality parameters of the reinforced learning”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 7,
Claim 7 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 7 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 2.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that do not apply the exception in a meaningful way (See MPEP 2106.05(e)).
The limitations:
“wherein the measured values comprise active powers, reactive powers, angles and/or currents of the respective phase at respective network nodes of the electricity network and/or in the respective lines of the electricity network”
As drafted, is an additional element that does not apply an exception for the abstract ideas in a meaningful way. See MPEP 2106.05(e).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements do not apply the exception in a meaningful way. The claim is not patent eligible.
Regarding Claim 8,
Claim 8 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 8 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 2.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“feeding changes in active powers and/or reactive powers into the electricity network and/or out due to the control values”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 9,
Claim 9 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 9 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 8.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recites additional elements that are insignificant extra-solution activity (See MPEP 2106.05(g)).
The limitations:
“wherein the control values are transmitted using a ripple control signal and/or telecontrol signal to a smart meter and/or to a controllable mains transformer and/or to converters of photovoltaic systems and/or to charging stations”
As drafted, are additional elements that amount to no more than insignificant extra-solution activity. See MPEP 2106.05(g).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “insignificant extra-solution activity”. Specifically, the transmission limitation recites the well-understood, routine, and conventional activity of receiving and transmitting data over a network. MPEP 2106.05(d)(II); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network). Insignificant extra-solution activity cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 10,
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 10 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein
PNG
media_image1.png
30
398
media_image1.png
Greyscale
is used as the reward function, where Gk indicates a measurement variable and Gkmax indicates its associated limit value, ΔP indicates a change in the active power, ΔQ indicates a change in the reactive power and st = (P1,t, P2,t, …, PN,t, Q1,t, Q2,t, …, QN,t)T indicates a state of the electricity network at the time t.
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 2.
Step 2B Analysis: See corresponding analysis of claim 2.
Regarding Claim 11,
Claim 11 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 11 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the learning takes place in such a way that the reward function r(st) is maximized”
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 10.
Step 2B Analysis: See corresponding analysis of claim 10.
Regarding Claim 12,
Claim 12 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 12 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: See corresponding analysis of claim 10.
Step 2A Prong Two Analysis: The judicial exceptions are not integrated into a practical application. In particular, the claim recited additional elements that are mere instructions to apply an exception (See MPEP 2106.05(f)).
The limitations:
“wherein the neural network determines the vector at = (ΔP1,t, …, ΔPF,t, ΔQ1,t, …, ΔQF,t)T as control values”
As drafted, are additional elements that amount to no more than mere instructions to apply an exception for the abstract ideas. See MPEP 2106.05(f).
Therefore, the additional elements do not integrate the abstract ideas into a practical application.
Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract ideas into a practical application, all of the additional elements are “mere instructions to apply”. Mere instructions to apply an exception cannot provide an inventive concept. The claim is not patent eligible.
Regarding Claim 13,
Claim 13 is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1 Analysis: Claim 13 is directed to a method for the reinforced learning of an artificial neural network, which is directed to a process, one of the statutory categories.
Step 2A Prong One Analysis: The limitations:
“wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measures values are calculated by means of a state estimation”
As drafted, under their broadest reasonable interpretations, cover mathematical
concepts, i.e., mathematical relationships, mathematical formulas or equations, and
mathematical calculations. The above limitations in the context of this claim correspond
to mathematical formulas and/or equations.
Step 2A Prong Two Analysis: See corresponding analysis of claim 2.
Step 2B Analysis: See corresponding analysis of claim 2.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2 and 4-9 are rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (U.S. Patent Publication No. 2020/0327411) (“Shi”) in view of Radhakrishnan et al. (Improving primary frequency response in networked microgrid operations using multilayer perceptron-driven reinforcement learning) (“Radhakrishnan”).
Regarding claim 1, Shi teaches a method for the reinforced learning of an artificial neural network (Shi [0007] “FIG. 1 shows an exemplary Voltage profile zone definition for training DRL agents.”; [0008] “FIG. 2 shows an exemplary overview of training and implementation of the DRL agents.”; [0067] “DRL is essentially developed from the classic reinforcement learning technique when combining with deep neural network (DNN) that consists of many layers.” Shi provides training and implementing DRL agents, which combine deep neural networks with reinforcement learning techniques, corresponding to the reinforced learning of an artificial neural network.), wherein the neural network uses measured values associated with a supply network to determine a plurality of control values for controlling the supply network (Shi [0020] “This system provides for training DRL agents for providing data-driven, real-time and autonomous control strategies for regulating voltage profiles in a power grid..”; [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures [control values] once voltage violations are detected... FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.”; [0039] “For the purpose of coordinated voltage control, states [neural network inputs] are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches [measured values associated with a supply network] that can be directly provided by EMS or WAMS systems.” Shi provides training a DRL agent corresponding to the neural network, which uses state values comprising voltage values of a power grid to determine corrective control measures once voltage violations are detected, wherein state vectors comprising voltage values are used an input to the neural network, as shown in Figure 3..), the method comprising: determining a control value (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures [determined control values] once voltage violations are detected.”; [0033] “However, due to variations in system loads, renewable generation and contingencies, once voltage issues occur, the DRL agent starts to take actions selected from an action space in order to fix the voltage issues. For each iteration of applied control actions, the control performance is calculated in terms of reward values.” Shi provides the DRL agent determining control actions regarding voltage levels in the power grid, as shown in Figure 2, corresponding to determining a control value.) such that at least one limit value of a measurement variable of the supply network is violated in a time range (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected. It is worth mentioning voltage limits considered can be adjusted/narrowed to make the proposed framework work for preventive control. Constraints considered in this study include full AC power flow equations, generation limits and voltage limits [limit value of a measurement variable of the supply network]. FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.”; Figure 1 “Violation Zone”; [0061] “The formulation of DRL for voltage control is flexible as it can intake multiple control objectives and consider various security constraints, especially time-series constraints [time range].” Shi provides a voltage violation zone for training of the DRL agent neural network, as shown by the “Violation zone” in Figure 1, wherein the voltage value corresponds to the measurement variable of the supply network which is violated in a time range, and wherein the time-series constraints corresponds to the time range.) …acquiring at least one measured value associated with the control value (Shi Figures 1 and 2; [0023] “For the problem of coordinated voltage control, a 4-tuple can be used to formulate the MDP, (S, A, P.sub.a, R.sub.a), where S is a vector of system states, including voltage magnitudes and phase angles across the system or areas of interest”; [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems. To maintain consistency of different inputs and outputs with various units when training DRL agents, the batch normalization technique is applied.” Shi provided the DRL agent acquiring voltage magnitude of the power grid to provide control signals, corresponding to acquiring at least one measured value associated with the control value.); and training the neural network using the calculated control value and the acquired associated measured value (Shi [0031] “Without loss of generality, this system trains effective DRL agents [training the neural network] for providing prompt corrective control measures once voltage violations are detected… Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected.”; [0033] “To train effective agents, massive representative operating conditions need to be collected or created, including random load changes, variations in renewable generation, generation dispatch patterns, major topology changes due to maintenance and contingencies.”; [0067] “Through such massive interactions, the DRL agent keeps optimizing its policy to maximize the accumulated rewards. In this way, the DRL agent will gradually master the control problem after a certain period of training.” Shi provides training the DRL agent neural network using the acquired voltage values and the control values through policy optimization, corresponding to training the neural network with the acquired measured voltage values and calculated control actions.).
Shi fails to explicitly teach …wherein the time range is determined in such a way that protective devices of the supply network do not trip.
However, Radhakrishnan teaches …wherein the time range is determined in such a way that protective devices of the supply network do not trip (Radhakrishnan 4.4 Performance of the RL-based controller “In Scenario 10, shown in Fig. 5, the system contains grid friendly appliance controllers [12] that stop operations when the frequency drops below 59 Hz. In the scenario, without CVR controllers, a communication delay causes the GFAs to be tripped as the PV Microgrid 2 (G3) output is not increased in time to avoid this. Later, when the PV in Microgrid 2 increases the output, an over-frequency event is caused at 6.6 s [time range]. On the other hand, the RL-based CVR controllers greatly improve the frequency response, maintaining the frequency closer to the desired 60 Hz, completely avoiding the GFA-level load shedding, and causing no over-frequency event when the PV in Microgrid 2 increases its output. Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off [protective devices of the supply network do not trip] due to IEEE Std. 1547. The large voltage deviation is also avoided as the CVR controller triggers a drop in terminal voltage at generatorG4 temporarily reducing load on the system till the under frequency event is stabilized.” Radhakrishnan provides the Reinforcement learning controller recovering the system fast enough to avoid inverter trip-off when the frequency drops below 59.5 Hz at 6.6s, as shown in Figure 5, corresponding determining a time range in such a way that protective devices do not trip.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 2, Shi in view of Radhakrishnan teaches wherein the supply network comprises an electricity network (Shi [0031] “FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus voltages of a power grid within a predefined zone before and after a disturbance.” Shi provides a power grid and corresponding bus voltages, corresponding to an electricity network.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 4, Shi in view of Radhakrishnan teaches wherein learning takes place during operation of the supply network (Shi [0067] “A general interaction process between the agent and the environment in DRL is presented in FIG. 3. After receiving the current states from the environment, a DRL agent generates a corresponding action using its policy; then, the environment provides the next state (s′) and the corresponding reward (r′) for the executed action. Through such massive interactions, the DRL agent keeps optimizing its policy to maximize the accumulated rewards [learning during operation]. In this way, the DRL agent will gradually master the control problem after a certain period of training.”; [0068] “FIG. 3 illustrates a sample interaction between Agent and Environment in reinforcement learning [learning during operation], while the main flowchart for training DRL Agents for coordinated voltage control is shown in FIG. 4, consisting of four major steps:” Shi provides the Agent interacting with the environment during learning, as shown in Figure 3, wherein learning takes place during operation of the electric supply network, as shown in Figure 3, which uses power values from the environment during the reinforcement training/learning to optimize policy and maximize accumulated rewards.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 1.
Regarding claim 5, Shi in view of Radhakrishnan teaches wherein the time range is less than or equal to one minute (Radhakrishnan “In Scenario 10, shown in Fig. 5, the system contains grid friendly appliance controllers [[12]] that stop operations when the frequency drops below 59 Hz. In the scenario, without CVR controllers, a communication delay causes the GFAs to be tripped as the PV Microgrid 2 (G3) output is not increased in time to avoid this. Later, when the PV in Microgrid 2 increases the output, an over-frequency event is caused at 6.6 s [time range is less than or equal to one minute]… Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off due to IEEE Std. 1547.” Radhakrishnan provides the trip avoidance of Scenario 10, as shown in Fig. 5, which has a time range of 8 seconds on the x-ais of the figure, and wherein the signal response is recorded at the 6.6s time range.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 6, Shi in view of Radhakrishnan teaches wherein avoided violations of limit values are used as quality parameters of the reinforced learning (Radhakrishnan 3.2 Problem formulation “The implementation of the RL starts with the definition of the MDP, with the four parameters… The reward function R is Equation (5), where the first term signifies the tracking performance [quality parameters] (maintain frequency at 60Hz at each discrete time) and the second term is to reduce oscillations during state transitions.”; 4.4 Performance of the RL-based controller “On the other hand, the RL-based CVR controllers greatly improve the frequency response, maintaining the frequency closer to the desired 60 Hz, completely avoiding the GFA-level load shedding, and causing no over-frequency event when the PV in Microgrid 2 increases its output. Though frequency drops below 59.5 Hz, the system recovers fast enough to avoid inverter trip-off [avoided violations of limit values] due to IEEE Std. 1547.” Radhakrishnan provides avoiding violations of limit values by avoiding inverter trip-off and causing no over-frequency event based on frequency values, wherein the frequency values are used in the reward function to track performance of the reinforcement learning algorithm, wherein the frequency values in the reward function for tracking performance correspond to the quality parameters.).
Shi and Radhakrishnan are both considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to power grid operations. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi with the above teachings of Radhakrishnan. Doing so would improve the resilience of the power distribution network beyond isolated microgrids (Radhakrishnan “Networked microgrid operations improve the resilience of the power distribution network beyond isolated microgrids.”).
Regarding claim 7, Shi in view of Radhakrishnan teaches wherein the measured values comprise active powers, reactive powers, angles and/or currents of the respective phase at respective network nodes of the electricity network and/or in the respective lines of the electricity network (Shi [0031] “Without loss of generality, this system trains effective DRL agents for providing prompt corrective control measures once voltage violations are detected. It is worth mentioning voltage limits considered can be adjusted/narrowed to make the proposed framework work for preventive control. Constraints considered in this study include full AC power flow equations, generation limits and voltage limits. FIG. 1 illustrates the desired control objective of training a DRL-agent, which is to regulate the bus [lines of the electricity network] voltages of a power grid within a predefined zone before and after a disturbance.”; [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems.” Shi provides phase angles, and active and reactive power flows and voltages from the power grid, wherein the power grid includes bus voltages that correspond to the respective lines of the electricity network).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 2.
Regarding claim 8, Shi in view of Radhakrishnan teaches further comprising feeding changes in active power and/or reactive powers into the electricity network and/or out due to the control values (Shi [0039] “For the purpose of coordinated voltage control, states are defined as a vector of voltage magnitudes, phase angles, and active and reactive power flows on branches that can be directly provided by EMS or WAMS systems.”; [0078] “FIG. 6 shows an exemplary power grid control system. The system includes a power grid measurement system with SCADA and WAMS. The states are provided to a power grid controller with DQN/DDPG agent and a prioritized replay buffer. The control signals are then provided as control variables for generator setting, transformer tap setting, shunt switching setting, topology adjustments, among others.” Shi provides active and reactive power flows as states, which are provided to a power grid controller to send control signals as control variables for generator setting, transformer tap setting, shunt switching setting, topology adjustments, corresponding to feeding changes in active and reactive powers in our out of the electricity network due to the control values.)
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 2.
Regarding claim 9, Shi in view of Radhakrishnan teaches wherein the control values are transmitted using a ripple control signal and/or telecontrol signal to a smart meter and/or to a controllable mains transformer and/or to converters of photovoltaic systems and/or to charging stations (Shi [0078] “FIG. 6 shows an exemplary power grid control system. The system includes a power grid measurement system with SCADA and WAMS. The states are provided to a power grid controller with DQN/DDPG agent and a prioritized replay buffer. The control signals [telecontrol signal] are then provided as control variables for generator setting, transformer tap setting [a controllable mains transformer], shunt switching setting, topology adjustments, among others.” Shi provides transmitting control settings for transformer tap setting, corresponding to transmitting control values to a controllable mains transformer using a telecontrol signal.).
It would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan for the same reasons disclosed above in the rejection of claim 8.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (U.S. Patent Publication No. 2020/0327411) (“Shi”) in view of Radhakrishnan et al. (Improving primary frequency response in networked microgrid operations using multilayer perceptron-driven reinforcement learning) (“Radhakrishnan”) in further view of Froehner et al. (U.S. Patent Publication No. 2019/0288508) (“Froehner”).
Regarding claim 3, Shi in view of Radhakrishnan teaches the method as claimed in claim 2, but fails to explicitly teach wherein the supply network is comprises as a medium-voltage network and/or low-voltage network.
However, Froehner teaches wherein the supply network is comprises as a medium-voltage network and/or low-voltage network (Froehner [0049] “Referring now to the figures of the drawings in detail and first, particularly to FIG. 1 thereof, there is shown a schematic view of an energy supply network 10, indicated purely by way of example, which may, for example, be a medium-voltage network. The energy supply network is connected at a connection station 11, for example a local network station, to a lower-level distribution network 12.” Froehner provides a medium-voltage network and a lower-level distribution network, corresponding to a low-voltage network.).
Shi, Radhakrishnan, and Froehner are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to electric supply networks. Therefore it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan with the above teachings of Froehner. Doing so would contribute to the stability of the operation of the energy supply network and the distribution network (Froehner [0052] “An increasing number of active devices (PV installations, wind power installations, storage devices, etc.) in the distribution networks increases the need to provide system services of this type at least partially in the distribution network also and therefore to contribute to the stability of the operation of the energy supply network and the distribution network.”).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (U.S. Patent Publication No. 2020/0327411) (“Shi”) in view of Radhakrishnan et al. (Improving primary frequency response in networked microgrid operations using multilayer perceptron-driven reinforcement learning) (“Radhakrishnan”) in further view of Shi et al. (U.S. Patent Publication No. 2020/0119556) (“Shi 2”).
Regarding claim 13, Shi in view of Radhakrishnan teaches the method as claimed in claim 2, but fails to teach wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measured values are calculated by means of a state estimation.
However, Shi 2 teaches wherein the learning additionally takes place with synthetic measured values, wherein the synthetic measured values are calculated by means of a state estimation (Shi 2 [0033] “An autonomous voltage control schema for grid operation using deep reinforcement learning (DRL) is detailed next.”; [0065] “To mimic real power system environment, a commercial power grid simulator is adopted, which is equipped with function modules such as power flow, dynamic simulation, contingency analysis, state estimation and so on. In this embodiment, only the AC power flow module, as environment, is applied to interact with the DRL agent.” Shi 2 provides deep reinforcement learning including mimicking real power systems with a simulated power grid including state estimation.).
Shi, Radhakrishnan, and Shi 2 are all considered to be analogous to the claimed invention because they are in the same field of artificial intelligence and more specifically applied to electric supply networks. Therefore it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Shi in view of Radhakrishnan with the above teachings of Shi 2. Doing so would allow for algorithms that are proven to be effective in various real-world control problems in highly dynamic and stochastic environments. (Shi 2 [0033] " In one embodiment, an innovative and promising approach of training DRL agents with improved RL algorithms provides data-driven, real-time and autonomous control strategies by coordinating and optimizing available controllers to regulate voltage profiles in a power grid, where the AVC problem is formulated as Markov decision process (MDP) so that it can take full advantages of state-of-the-art reinforcement learning (RL) algorithms that are proven to be effective in various real-world control problems in highly dynamic and stochastic environments.”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KURT NICHOLAS PRESSLY whose telephone number is (703)756-4639. The examiner can normally be reached M-F 8-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KURT NICHOLAS PRESSLY/Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights