DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the Application filed on March 9, 2023. Claims 1-3 are pending in the case. Claim 1 is the independent claim.
This action is non-final.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-3 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors, as provided in further detail below.
With respect to claim 1:
On lines 7-8, the claims recite “a traditional deep neural network.” It is unclear whether the limitation “traditional” is intended to convey any particular meaning or structure which would further limit the deep neural network. Further, the term “traditional” may be interpreted as a term of relative degree which renders the limitation indefinite. For example, the term may be intended to convey that the neural network is one which would be widely known in the art, or has a set of relatively well-known, long-utilized, etc. features, with respect to a particular point in time, but the claim and the specification provide no additional definition or limitation indicating the scope of this limitation such that one of ordinary skill in the art would understand what neural network examples are, and are not, “traditional.”
On lines 11-13, the claim recites, “Actor network structure, comprising an input layer, a hidden layer and an output layer, wherein the hidden layer…normalizes the output of a previous layer and simultaneously inputs an action value.” The limitation “the output of a previous layer” lacks antecedent basis. Further, the claim appears to recite that the network structure comprises an input layer, a hidden layer and an output layer, and that the hidden layer normalizes the output of a previous layer. Given that the other recited layers besides the hidden layer comprise an input layer and an output layer it is unclear whether the limitation “previous layer” is intended to refer to one of these, or if it is intended to refer to yet another unrecited layer of the network structure. Moreover, the claim recites that the hidden layer “simultaneously inputs an action value.” First, it is unclear what the input of the action value occurs simultaneous to. Second, it is unclear whether the inputting of the action value by the hidden layer is intended to recite an input to the hidden layer, or an input by the hidden layer to some other layer such as the output layer. To the extent that this input occurs simultaneous to one other operations, such as other operations by the hidden layer, it is unclear whether the simultaneous input of the action value is dependent upon or otherwise relevant to any of the other processes recited as being performed by the hidden layer (i.e. mapping a state to a feature, normalizing the output of a previous layer, etc.).
On line 14, the claim recites “the Critic network.” This limitation lacks antecedent basis. Although the claims previously recite “an Actor-Critic network model,” the claims do not appear to necessarily recite that the Critic network is, or is not, related to the Actor-Critic network model. The claim also recites “the performing quality of the action.” The limitation “the performing quality” lacks antecedent basis. To the extent that the claim recites “generating actions by an Actor network,” the limitation “the action” is interpreted as referring to at least one of the actions generated by the Actor network.
On line 15, the claim recites “the deep neural network.” This limitation lacks antecedent basis. Although the claim does recite “a traditional deep neural network,” this is recited as a component of the Actor network, while the instant limitation (on line 15) is apparently a component of the Critic network. Therefore, it is further unclear whether the recitation of “the deep neural network” with respect to the critic network is intended to mean that the Critic network has the same deep neural network as the Actor network, or a neural network having a same structure, or if this limitation is intended to recite a different deep neural network altogether.
On line 17, the claim recites “designing a Critic network structure, and adding the hidden layer after the input state s.” The only previous recitation of a “hidden layer” is the recitation of a hidden layer as a component of the Actor network structure. Here, the hidden layer appears to be recited as a possible component of the Critic network. Therefore, it is unclear whether this limitation is intended to recite a same hidden layer in both the Actor and Critic networks, or separate hidden layers which potentially have the same structure, or if the limitation is intended to recite an additional hidden layer within the Critic network which is distinct from the hidden layer of the Actor network. If the claim is intended to recite a distinct hidden layer, then the recitation “the hidden layer” lacks antecedent basis. In addition, the limitation “the input state s” also lacks antecedent basis.
On line 18, the claim recites “meanwhile, because the input of the Critic network should have an action a.” It is unclear whether the limitation “meanwhile” is intended to require some specific timing or not. The limitation “should” appears to indicate that the network having the action a may be optional; it is unclear whether the network having the action a is, or is not, optional.
On line 19, the claim recites “the features of the state s.” The limitation “the features” lacks antecedent basis.
On line 20, the claim recites “the performing quality of the action.” The limitation “the performing quality” lacks antecedent basis, as discussed above. It is unclear whether the limitation “the action” is intended to refer to “an action a,” recited within the same clause, or to “the action” recited on line 14, or one of the actions generated by an Actor network.
On line 21, the claim recites “the deep neural network.” It is unclear whether this limitation is intended to refer to “the traditional deep neural network” of the Actor network, to “the deep neural network” which is associated with the Critic network, or to some other “deep neural network” which is configured specifically for use as a “function fitter.”
On lines 24-25, the claim recites “simultaneously solving the problems of high-dimensional state space and continuous action output which cannot be solved by the traditional DQN algorithm.” The limitation “the problems” and the limitation “the traditional DQN algorithm” each lack antecedent basis. With respect to the limitation “simultaneously” it is unclear what other act the solving must occur simultaneous to. The limitation “the problems…which cannot be solved…” appears to provide for an open-ended, undefined set of problems having a potentially unlimited size/amount of problems. Therefore, one of ordinary skill in the art could not ascertain the proper scope of this limitation, including what type of problems are required to be simultaneously solved, what they are required to be simultaneously solved with, whether all such problems must be solved, or just some subset, etc.
On line 26, the claim recites “the correlation between samples.” This limitation lacks antecedent basis.
On lines 27-28, the claim recites “the weight parameters of the network.” The limitations “the weight parameters” and “the network” lacks antecedent basis. To the extent that “the network” is intended to refer to one of the multiple previously-recited deep neural networks (as discussed above), it is unclear which of these the limitation “the network” is intended to refer to.
On lines 28-29, the claim recites “the stability of network training.” This limitation lacks antecedent basis.
On line 29, the claim recites “deterministic behavior policies make the output of each step computable.” The limitation “the output of each step” lacks antecedent basis. Moreover, this limitation reads as generally descriptive of the “deterministic behavior policies,” but it is unclear whether such policies are actually required or utilized anywhere within the claimed invention. Further, the limitation “computable” appears to possibly recite a relative term, as it generally recites that a result that steps are computable, but provides no limitation or definition regarding which types of steps are, or are not, considered computable within the intended meaning of the claim. Finally, it is unclear whether the limitation “each step” refers to each step of the recited method as a whole, or each step of some other portion of the method, such as each step of a learning/training process, etc.
On lines 30-33, the claim recites “the core problem…is to process a training objective…therefore, an appropriate reward function should be set to make the network select an optimal policy” As discussed above, this limitation is generally narrative and indefinite. The limitation “the core problem” lacks antecedent basis. The limitation “an appropriate award function” appears to recite a relative term (i.e. what is “appropriate” or not may vary based on a variety of different conditions and circumstances) such that one of ordinary skill in the art could not ascertain the scope of the limitation. The limitation “should” appears to render the set of associated limitations as potentially optional (i.e. the appropriate reward function should be set, but is not necessarily required to be set, etc.). It is unclear which of the various recited neural networks the limitation “the network” is intended to refer to.
On line 34, the claim recites “the target requirements of the transient state, the objective function….” The limitations “the target requirements,” “the transient state,” and “the objective function each lack antecedent basis. Moreover, it cannot be determined whether any of these limitations have any relation to other recited limitation. For example, it is unclear what the transient state is a state of, it is unclear what the target requirements are target requirements of, and it is unclear what entity (e.g. one of the various recited networks, models, etc.) the objective function is relevant to, if any.
On lines 35-37, the claim recites “the process of learning and exploration in continuous space can be independent of the learning algorithm; therefore, it is necessary to add noise to the output of the Actor network policy….” As discussed above, this limitation is generally narrative and indefinite. The limitation “the process of learning and exploration…” lacks antecedent basis. The limitation “the learning algorithm” appears to lack antecedent basis. Although the claim does recite, prior to this, “a reinforcement learning algorithm,” it is unclear whether this limitation is intended to refer to the reinforcement learning algorithm or some other learning algorithm. The limitation “can be independent” appears to indicate that the recited independence is optional; therefore, it cannot be determined whether the recited process is independent, or not, of the learning algorithm. The limitations “the output” and “the Actor network policy” both appear to lack antecedent basis. Although the claim does recite, prior to this, “the output behavior” and “the output of a previous layer” in association with the Actor network, it is unclear whether “the output” is intended to refer to one of these, or to some other output. Although the claim does recite, prior to this, “a deterministic policy function” in association with the Actor network, it is unclear whether “the Actor network policy” is intended to refer to this policy function, or some other policy, such as the “optimal policy” recited elsewhere in the claim.
On line 40, the claim recites “the model.” This limitation lacks antecedent basis. Prior to this, the claim recites an “engine model,” “a real-time model,” and “an Actor-Critic network model.” It is unclear which of these the limitation “the model” refers to, or if it refers to a different model altogether.
On line 43, the claim recites “the existing requirements.” This limitation lacks antecedent basis.
On line 44, the claim recites “the engine model.” This limitation appears to lack antecedent basis. Although the claim also recites “a twin-spool turbo-fan engine model” which is adjusted as “a model for invoking a reinforcement learning algorithm,” it is unclear whether “the engine model” is intended to refer to these. Moreover, to the extent that “the engine model” is intended to refer to these, it is unclear whether it is intended to refer to the “twin-spool turbo-fan engine model” prior to the adjusting, or if it is intended to refer to “the model for invoking the reinforcement learning algorithm” which apparently results from the adjustment.
On line 45, the claim recites the limitation “the range of a target reward value.” This limitation lacks antecedent basis.
On lines 46-47, the claim recites the limitation “the corresponding requirements.” This limitation lacks antecedent basis.
On line 48, the claim recites “the policy.” This limitation lacks antecedent basis. Although the claims do recite, prior to this, “a deterministic policy function,” an “optimal policy,” and a “new exploration policy,” it is unclear whether the limitation “the policy” refers to one of these, or to some other policy.
On lines 50-51, the claim recites “the target speed,” and “the premise.” These limitations lack antecedent basis.
On line 52, the claim recites “the training.” This limitation lacks antecedent basis. Although the claim recites, prior to this, “network training,” “training the model,” and “batch training,” it is unclear whether the limitation “the training” refers to one of these or a different act of training.
On lines 53-55, the claim recites “the control law of engine acceleration transition,” “the above training process,” “the method,” and “the engine acceleration process.” Each of these limitations lack antecedent basis.
On line 57, the claim recites “the training.” It cannot be determined whether this limitation is intended to refer to any of the previously-recited “network training,” “training of the model,” “batch training,” and “the training,” or to some other act of training.
On lines 58-59, the claim recites “at this time.” It is unclear whether this limitation is intended to impose a temporal/timing requirement, or how the limitation is intended to further specify or limit the associated limitations (e.g. does the limitation intend to require that at the time that operating conditions correspond to controller parameters, the controller input is a target speed value and the output is a fuel flow, or does it mean that these limitations are the case at the time after the training and when the controller parameters are obtained? In addition, is the limitation “at this time” intended to convey that the input and output are variable and may be some other value at some other time?). Therefore, this limitation is indefinite. The claim further recites “the controller input” and “the output.” These limitations lack antecedent basis. Although the claim recites multiple different outputs, i.e. “continuous action output of a real-time model,” “the output behavior,” “the output of a previous layer,” “an output is a Q value function,” “a final output result is a Q value,” “outputting an action,” “continuous action output,” “the output of each step,” “the output of the Actor network policy,” “optimal control quantity output,” etc., it is unclear whether “the output refers to one of these or some other output.
On lines 60-62, the claim recites “directly giving the control law by the model under the current operating condition, and controlling the transient state of the engine acceleration process by directly communicating the output of the model with the input of the engine.” With respect to the limitation “the control law,” this limitation appears to lack antecedent basis. Although the claim recites, prior to this, “the control law of engine acceleration transition,” which is obtained from a training process, it is unclear whether that control law is the same as the control law recited at this location, which is not explicitly recited as being the control law of engine acceleration transition, and is apparently a given control law by the model under current operating conditions and therefore potentially different from a control law obtained from a training process. The limitation “the model” also lacks antecedent basis. Prior to this, the claim recites an “engine model,” “a real-time model,” and “an Actor-Critic network model.” It is unclear which of these the limitation “the model” refers to, or if it refers to a different model altogether. The limitations “the current operating condition,” “the transient state of the engine acceleration process,” “the output of the model,” and “the input of the engine,” also each lack antecedent basis. As noted above, the claim recites multiple different outputs, but it is not clear whether “the output of the model” refers to one of these or to a different output. The claim recites multiple inputs, such as “an input state s,” input of an action value, “input thereof” with respect to the Critic network and “the input of the Critic network,” and “the controller input”; however, none of these appear to be the same as “the input of the engine.” With respect to the collective limitations “giving the control law by the model under the current operating condition,” it is unclear whether this is intended to an output of the control law by the model, where the model is under the current operating condition, where the control law itself is under the current operating condition, or if some other configuration is contemplated; that is, it is unclear what relationships each of “the control law,” “the model,” and “the current operating condition” have to the recited act of giving. Finally, with respect to the collective limitations “directly giving…by the model” and “directly communicating the output of the model with the input of the engine…” it is unclear whether the “direct giving” and the “direct communicating” both have a same recipient, such as “the input of the engine,” or if only the “directly communicating” is “with the input of the engine” and the “directly giving…by the model” actually has some other recipient; in addition, while the act of directly giving is recited as being done by the model, it is not clear if the model is also performing the act of “directly communicating” or some other component performs this function.
With respect to claims 2 and 3, these claims depend upon claim 1 and inherit the deficiencies identified above with respect to claim 1. Therefore, claims 2 and 3 are rejected on the same basis as discussed above with respect to claim 1.
Claim 2 recites, on line 4, “the engine transient state.” Prior to this, the claim recites “the aero-engine transient state.” In addition, claim 1 also recites “the aero-engine transient state,” “the transient state,” and “the transient state of the engine acceleration process.” It cannot be determined which of these the limitation “the engine transient state” is intended to refer to.
Claim 3 recites, on lines 2-3, “the activation function. This limitation lacks antecedent basis.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain,” In re Heck, 699 F.2d 1331, 1332-33, 216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting in re Lemelson, 397 F.2d 1006, 1009, 158 USPQ 275, 277 (GCPA 1968)). Further, a reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co, v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert, denied, 493 U.S. 975 (1989). See also Upsher-Smith Labs. v. Pamlab, LLC, 412 F,3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir, 2005): Celeritas Technologies Ltd. v. Rockwell International Corp., 150 F.3d 1354, 1361, 47 USPQ2d 1516, 1522-23 (Fed. Cir. 1998).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEREMY L STANLEY whose telephone number is (469)295-9105. The examiner can normally be reached on Monday-Friday from 9:00 AM to 5:00 PM CST.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abdullah Al Kawsar, can be reached at telephone number (571) 270-3169. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR for authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/JEREMY L STANLEY/
Primary Examiner, Art Unit 2127
Prosecution Insights