DETAILED ACTION
The present application, filed on 11/13/2023 is being examined under the AIA first inventor to file provisions.
The following is a FINAL Office Action in response to Applicant’s amendments filed on 8/4/2026.
a. Claims 1, 6-8 are amended
Overall, claims 1-8 are pending and have been considered below.
Claim Rejections - 35 USC § 101
35 USC 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8 are rejected under 35 USC 101 because the claimed invention is not directed to patent eligible subject matter. The claimed matter is directed to a judicial exception, i.e. an abstract idea, not integrated into a practical application, and without significantly more.
Per Step 1 of the multi-step eligibility analysis, claims 1-5 are directed to a computer implemented method, claim 6 are directed to a computer implemented method, claims 7, and claims 8 are directed to a are directed to a system computer executable instruction stored on a non-transitory storage medium.
Thus, on its face, each independent claim and the associated dependent claims are directed to a statutory category of invention.
[INDEPENDENT CLAIMS]
Per Step 2A.1. Independent claim 1, (which is representative of independent claims 6-8) is rejected under 35 USC 101 because the independent claim is directed to an abstract idea, a judicial exception, without reciting additional elements that integrate the judicial exception into a practical application.
The limitations of the independent claim 1 (which is representative of independent claims 6-8) recite an abstract idea, shown in bold below:
[A] A method for controlling a technical system, comprising the following steps: training a control policy including:
[B] estimating a variance of a value function which associates: (i) a state with a value of the state, or (ii) a pair of state and action with a value of the pair, by solving a Bellman uncertainty equation,
[C] wherein, for each of multiple states, a reward function of the Bellman uncertainty equation is set to a difference of a total uncertainty about a mean of a value of a subsequent state following the state and an average aleatoric uncertainty of the value of the subsequent state
[D] biasing the control policy in training towards regions for which the estimation gives a higher variance of the value function than for other regions; and
[E] controlling the technical system according to the trained control policy,
[F] wherein the technical system is a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.
Independent claim 1 (which is representative of independent claims 6-8) recites: estimating a value function variance ([B]); biasing the control policy ([D]) and controlling the technical system ([E]), which, based on the claim language and in view of the application disclosure, represents a process aimed at: procedure for controlling a technical system (e.g. robot).
This is a combination that, under its broadest reasonable interpretation, covers performance of limitations expressing mathematical concepts like mathematical relationships, mathematical formulas or equations, mathematical calculations. These fall under the Mathematical Concepts. i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations grouping of abstract ideas (see MPEP 2106.04(a)(2) I).
Accordingly, it is concluded that independent claim 1 (which is representative of independent claims 1, 6-8) recites an abstract idea that corresponds to a judicial exception.
In addition, or alternatively, this is a combination that, under its broadest reasonable interpretation, covers reasonable performance of limitations expressing observation, evaluation, in the human mind. Nothing in the claim elements precludes the steps from being practically performed in the human mind. For example, the step “estimating a variance of a value function which associates: (i) a state with a value of the state, or (ii) a pair of state and action with a value of the pair, by solving a Bellman uncertainty equation,”, as drafted in the context of this claim, encompasses the user manually or mentally making an estimation, without physical aid. Further, the step “biasing the control policy in training towards regions for which the estimation gives a higher variance of the value function than for other regions”, as drafted in the context of this claim, encompasses the user manually or mentally biasing (e.g., squeeing) a policy, without physical aid. Further, the step “controlling the technical system according to the trained control policy.”, as drafted in the context of this claim, encompasses the user manually or mentally controlling the system, without physical aid. These limitations fall under the Mental Processes, i.e., Concepts Performed in the Human Mind grouping of abstract ideas (see MPEP 2106.04(a)(2)). The use of a physical aid would not negate the mental nature of this limitation (see MPEP 2106.04(a)(2) iii B)
Accordingly, it is concluded that independent claim 1 (which is representative of independent claims 6-8) recites an abstract idea that corresponds to a judicial exception.
[INDEPENDENT CLAIMS – Additional Elements]
Per Step 2A.2. The identified abstract idea is not integrated into a practical application because the additional elements in the independent claims only amount to instructions to apply the judicial exception to a computer, or are a general link to a technological environment (see MPEP 2106.05(f); MPEP 2106.05(h)).
For example, the added elements “computer,” “computer readable medium,” recite computing elements at a high level of generality, generally linking the use of a judicial exception to a particular technological environment (see MPEP 2106.05(h)), or merely using a computer as a tool to perform an abstract idea (MPEP 2106.05(f)). Further, the additional elements “wherein, for each of multiple states, a reward function of the Bellman uncertainty equation is set to a difference of a total uncertainty about a mean of a value of a subsequent state following the state and an average aleatoric uncertainty of the value of the subsequent state,” “wherein the technical system is a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.” as applied to the bellman uncertainty equation, are nothing more than (a) descriptive limitations of claim elements, such as describing the nature, structure and/or content of other claim elements, or (b) general links to the computing environment, which amount to instructions to “apply it,” or equivalent (MPEP 2106.05(f)).
These additional elements of the independent claims do not preclude from carrying out the identified abstract idea procedure for controlling a technical system (e.g. robot), and do not serve to integrate the identified abstract idea into a practical application.
Per Step 2B. Independent claim 1 (which is representative of claims independent 6-8) does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when the independent claim is reevaluated as a whole, as an ordered combination under the considerations of Step 2B, the outcome is the same like under Step 2A.2.
Overall, it is concluded that independent claims 1, 6-8 are deemed ineligible.
[DEPENDENT CLAIMS]
Dependent claim 3 recites:
setting an uncertainty about the mean of the value of the subsequent state following the state to an estimate of the variance of the mean of the value of the subsequent state, and
setting the average aleatoric uncertainty to the mean of an estimate of the variance of the value of the subsequent state.
When considered individually, these added claim elements further elaborate on the abstract idea identified in the independent claims, because the dependent claim continues to recite the identified abstract idea: procedure for controlling a technical system (e.g. robot). The elements in this dependent claim are comparable to limitations expressing mathematical concepts like mathematical relationships, mathematical formulas or equations, mathematical calculations. These fall under the Mathematical Concepts. i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations grouping of abstract ideas (see MPEP 2106.04(a)(2) I).. Thus, the dependent claim elements are not directed to any specific improvements of the independent claims and do not practically or significantly alter how the identified abstract idea would be performed.
Therefore, dependent claim 3 is deemed ineligible.
Dependent claim 5 recites:
solving the Bellman uncertainty equation using a neural network trained to predict a solution of the Bellman uncertainty equation in response to an input of a state or pair of state and action value.
When considered individually, these added claim elements further elaborate on the abstract idea identified in the independent claims, because the dependent claim continues to recite the identified abstract idea: procedure for controlling a technical system (e.g. robot). The elements in this dependent claim are comparable to limitations expressing mathematical concepts like mathematical relationships, mathematical formulas or equations, mathematical calculations. These fall under the Mathematical Concepts. i.e., mathematical relationships, mathematical formulas or equations, or mathematical calculations grouping of abstract ideas (see MPEP 2106.04(a)(2) I).. Thus, the dependent claim elements are not directed to any specific improvements of the independent claims and do not practically or significantly alter how the identified abstract idea would be performed.
Therefore, dependent claim 5 is deemed ineligible.
Dependent claims 2, 4, respectively, recite:
wherein: (i) the value function is a state value function, and the control policy is biased in training towards regions of a state space for which the estimation gives a higher variance of values of states than for other regions of the state space, or (ii) the value function is a state-action value function and the control policy is biased in training towards regions of a space of state-action pairs for which the estimation gives a higher variance of a value of pairs of states and actions than for other regions of the space of state-action pairs.
wherein the estimating of the variance of the value function includes selecting one of multiple neural networks,
wherein each of the neural networks is trained to output information about a probability distribution of a subsequent state following a state input to the neural network and of a reward obtained from a state transition and determining the value function from outputs of the selected neural network for a sequence of states.
These further elements in the dependent claims do not perform any claimed method steps. They describe the nature, structure and/or content of other claim elements – the value function; the estimating of the variance; the neural networks – and as such, cannot change the nature of the identified abstract idea (procedure for controlling a technical system (e.g. robot)), from a judicial exception into eligible subject matter, because they do not represent significantly more (see MPEP 2106.07). The nature, form or structure of the other claim elements themselves do not practically or significantly alter how the identified abstract idea would be performed and do not provide more than a general link to a technological environment.
Therefore, dependent claims 2. 4 are deemed ineligible.
When the dependent claims are considered as a whole, as an ordered combination, the claim elements noted above appear to merely apply the abstract concept to a technical environment in a very general sense. The most significant elements, which form the abstract concept, are set forth in the independent claims. The fact that the computing devices and the dependent claims are facilitating the abstract concept is not enough to confer statutory subject matter eligibility, since their individual and combined significance do not transform the identified abstract concept at the core of the claimed invention into eligible subject matter. Therefore, it is concluded that the dependent claims of the instant application, considered individually, or as a as a whole, as an ordered combination, do not amount to significantly more (see MPEP 2106.07(a)II).
In sum, claims 1-8 are rejected under 35 USC 101 as being directed to non-statutory subject matter.
The prior art made of record and not relied upon which, however, is considered pertinent to applicant's disclosure:
US 20120065746 A1 Wintrich; Franz et al. CONTROL SYSTEM A control system (1) for a complex process, particularly for controlling a combustion process in a power plant, a waste incinerator plant, or a cement plant, has a controlled system (14) and at least one controller (36), wherein the control system (1) is divided hierarchically into various levels (10, 20, 30, 40). The first level (10) represents the complex, real process to be controlled and is implemented by the controlled system (14). The second level (20) represents an interface to the process and is implemented by a process control system. The third level (30) represents the control of the process and is implemented by the at least one active controller (36). The fourth level (40) represents a superordinate overview and is implemented by a principal controller (44).
US 20090271340 A1 Schneegass; Daniel et al. Method for the computer-aided learning of a control or adjustment of a technical system A method for the computer-aided learning of a control of a technical system is provided. An operation of the technical system is characterized by states which the technical system can assume during operation. Actions are executed during the operation and convert a relevant state into a subsequent state. The method is characterized in that, when learning the control, suitable consideration is given to the statistical uncertainty of the training data. This is achieved in that the statistical uncertainty of a quality function which models an optimal operation of the technical system is specified by an uncertainty propagation and is incorporated into an action selection rule when learning. By a correspondingly selectable certainty parameter, the learning method can be adapted to different application scenarios which vary in statistical requirements. The method can be used for learning the control of an operation of a turbine, in particular a gas turbine.
US 20130131839 A1 Washington; Rodney B. et al. Dynamically Adapting to Changes in Control System Topology An apparatus and method for adapting to changes in the control topology of a cooperative control system including a plurality of controllers are disclosed. The method is implemented by an actuator or sensor, and includes the steps of selecting one of the controllers as the master controller for one or more state variables of an actuator or sensor, detecting a change in the control topology of the cooperative control system, and reselecting a master controller for the one or more state variables responsive to the change in the control topology.
US 20190318051 A1 OSWALD; Mario et al. METHOD FOR SIMULATION-BASED ANALYSIS OF A MOTOR VEHICLE The invention relates to a method for simulation-based analysis and/or optimization of a motor vehicle, preferably having the following working steps: simulating (SIOI) a driving operation of the motor vehicle (I) on the basis of a model (M) with at least one manipulated variable for acquiring values of at least one simulated variable which is suitable for characterizing an overall vehicle behaviour, in particular a driving capability, of the motor vehicle (I), wherein the model has at least one partial model, in particular a torque model, and wherein the at least one partial model is based on a function and preferably characterizes the operation of at least one component, in particular of an internal combustion engine of the motor vehicle (I); and—outputting (S I03) the values of the at least one simulated variable.
US 20200150672 A1 NAGHSHVAR; Mohammad et al. HYBRID REINFORCEMENT LEARNING FOR AUTONOMOUS DRIVING A method includes determining a current state of an environment of an autonomous agent, such as a vehicle. The method also includes determining, via a first neural network, a set of actions based on the current state. The method further includes determining whether further analysis of the set of actions is desired. The method selects an action from the set of actions using a model-based solution based on a reward and a risk of the action when further analysis is desired. The method also includes selecting the action from the set of actions according to a metric when further analysis is not desired. The method controls the autonomous agent to perform the selected action.
US 20100257866 A1 Schneegass; Daniel et al. METHOD FOR COMPUTER-SUPPORTED CONTROL AND/OR REGULATION OF A TECHNICAL SYSTEM A method for computer-supported control and/or regulation of a technical system is provided. In the method a reinforcing learning method and an artificial neuronal network are used. In a preferred embodiment, parallel feed-forward networks are connected together such that the global architecture meets an optimal criterion. The network thus approximates the observed benefits as predictor for the expected benefits. In this manner, actual observations are used in an optimal manner to determine a quality function. The quality function obtained intrinsically from the network provides the optimal action selection rule for the given control problem. The method may be applied to any technical system for regulation or control. A preferred field of application is the regulation or control of turbines, in particular a gas turbine.
US 11699062 B2 Isele; David Francis System and method for implementing reward based strategies for promoting exploration A system and method for implementing reward based strategies for promoting exploration that include receiving data associated with an agent environment of an ego agent and a target agent and receiving data associated with a dynamic operation of the ego agent and the target agent within the agent environment. The system and method also include implementing a reward function that is associated with exploration of at least one agent state within the agent environment. The system and method further include training a neural network with a novel unexplored agent state.
US 11989658 B2 Kim; Hyunseok et al. Method and apparatus for reinforcement machine learning A method and an apparatus for exclusive reinforcement learning are provided, comprising: collecting information of states of an environment through the communication interface and performing a statistical analysis on the states using the collected information; determining a first state value of a first state among the states in a training phase and a second state value of a second state among the states in an inference phase based on analysis results of the statistical analysis; performing reinforcement learning by using one reinforcement learning unit of a plurality of reinforcement learning unit which performs reinforcement learnings from different perspectives according to the first state value; and selecting one of actions determined by the plurality of reinforcement learning unit based on the second state value and applying selected action to the environment.
US 6208981 B1 Graf; Friedrich et al. Circuit configuration for controlling a running-gear or drive system in a motor vehicle control signals for a system device of the motor vehicle--for example an automatic transmission, active suspension, speed stabilization, power-steering assistance, or traction control. The fuzzy system is connected to a neural network, which evaluates the sensor signals and reference data from a recording of driving data of the motor vehicle. The neural network optimizes the rule base of the fuzzy system. During a driving operation, the fuzzy system generates on-line signals categorizing the respective driving situation, and thus makes possible intelligent, time-adaptive, driving-situation-dependent control. The fuzzy system and the neural network each contain a classification system which can be reciprocally converted by a correspondence-maintaining bidirectional transformation.
US 12154029 B2 Schaul; Tom et al. Continual reinforcement learning with a multi-task agent A method of training an action selection neural network for controlling an agent interacting with an environment to perform different tasks is described. The method includes obtaining a first trajectory of transitions generated while the agent was performing an episode of the first task from multiple tasks; and training the action selection neural network on the first trajectory to adjust the control policies for the multiple tasks. The training includes, for each transition in the first trajectory: generating respective policy outputs for the initial observation in the transition for each task in a subset of tasks that includes the first task and one other task; generating respective target policy outputs for each task using the reward in the transition, and determining an update to the current parameter values based on, for each task, a gradient of a loss between the policy output and the target policy output for the task.
Response to Amendments/Arguments
Applicant’s submitted remarks and arguments have been fully considered.
Applicant disagrees with the Office Action conclusions and asserts that the presented claims fully comply with the requirements of 35 U.S.C. § 101 regrading judicial exceptions. Further, Applicant is of the opinion that the prior art fails to teach Applicant’s invention.
Examiner respectfully disagrees with the former.
With respect to Applicant’s Remarks as to the claims being rejected under 35 USC § 101.
Applicant submits:
a. The pending claims are not directed to an abstract idea.
b. The identified abstract idea is integrated into a practical application.
c. The pending claims amount to significantly more.
Furthermore, Applicant asserts that the Office has failed to meet its burden to identify the abstract idea and to establish that the identified abstract idea is not integrated into a practical application and that the pending claims do not amount to significantly more.
Examiner responds – The arguments have been considered in light of Applicants’ amendments to the claims. The arguments ARE NOT PERSUASIVE. Therefore, the rejection is maintained.
The pending claims, as a whole, are directed to an abstract idea not integrated into a practical application. This is because (1) they do not effect improvements to the functioning of a computer, or to any other technology or technical field (see MPEP 2106.05 (a)); (2) they do not apply or use the abstract idea to effect a particular treatment or prophylaxis for a disease or a medical condition (see the Vanda memo); (3) they do not apply the abstract idea with, or by use of, a particular machine (see MPEP 2106.05 (b)); (4) they do not effect a transformation or reduction of a particular article to a different state or thing (see MPEP 2106.05 (c)); (5) they do not apply or use the abstract idea in some other meaningful way beyond generally linking the use of the identified abstract idea to a particular technological environment, such that the claim as a whole is more than a drafting effort designated to monopolize the exception (see MPEP 2106.05 (e) and the Vanda memo).
In addition, the pending claims do not amount to significantly more than the abstract idea itself.
As such, the pending claims, when considered as a whole, are directed to an abstract idea not integrated into a practical application and not amounting to significantly more.
More specific:
Applicant submits “Contrary to the Action's assertion, the human mind is not equipped to practically control a technical system (e.g., a robot, a vehicle, a power tool, etc.).”
Examiner has carefully considered, but doesn’t find Applicant’s arguments persuasive.
The eligibility analysis in the instant office action does not make such an allegation.
Thus, the rejection is proper and has been maintained.
Applicant submits “… the Office has failed to articulate how the human mind can control a technical system with mere thought.”
Examiner has carefully considered, but doesn’t find Applicant’s arguments persuasive.
The eligibility analysis in the instant office action concludes that the step can be executed manually or mentally. It stands to reason that a system can be controlled manually, while the metal part can be used as a tool to prepare the manual intervention. One or more manual interventions is herewith considered to form the controlling action. By way of example – all road vehicles are, to this day, controlled manually, even if this might change in the future.
Thus, the rejection is proper and has been maintained.
Applicant submits “Indeed, the amended claims integrate any alleged abstract ideas into a practical application of controlling a technical system.”
Examiner has carefully considered, but doesn’t find Applicant’s arguments persuasive.
MPEP 2106.04(d)(1) discloses:
An important consideration to evaluate when determining whether the claim as a whole integrates a judicial exception into a practical application is whether the claimed invention improves the functioning of a computer or other technology .... In short, first the specification should be evaluated to determine if the disclosure provides sufficient details such that one of ordinary skill in the art would recognize the claimed invention as providing an improvement. The specification need not explicitly set forth the improvement, but it must describe the invention such that the improvement would be apparent to one of ordinary skill in the art .... Second, if the specification sets forth an improvement in technology. the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement. (Emphasis added)
That is, the claimed invention may integrate the judicial exception into a practical application by demonstrating that it improves the relevant existing technology although it may not be an improvement over well-understood, routine, conventional activity. (Emphasis added)
Thus, the rejection is proper and has been maintained.
Applicant submits “Therefore, the specific combination of amended claim 1 integrates any alleged abstract ideas into a practical application of controlling a technical system. Accordingly, claim 1 recites additional elements that amount to a practical application of any alleged abstract ideas under Step 2A, Prong two of the two-step inquiry.”
Examiner has carefully considered, but doesn’t find Applicant’s arguments persuasive.
See response immediately above.
Thus, the rejection is proper and has been maintained.
It follows from the above that there are no meaningful limitations in the claims that transform the judicial exception into a patent eligible application such that the claims amount to significantly more than the judicial exception itself. Therefore, the rejection under 35 U.S.C. § 101 is maintained.
With respect to Applicant’s Remarks as to the claims being rejected under 35 USC § 103.
After further considerations, the rejection is withdrawn. The prior art of record does not disclose at least:
wherein, for each of multiple states, a reward function of the Bellman uncertainty equation is set to a difference of a total uncertainty about a mean of a value of a subsequent state following the state and an average aleatoric uncertainty of the value of the subsequent state
Examiner has reviewed and considered all of Applicant’s remarks. The rejection is maintained, necessitated by the fact that the rejection of the claims under 35 USC § 101 has not been overcome.
Conclusion
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Radu Andrei whose telephone number is 313.446.4948. The examiner can normally be reached on Monday – Friday 8:30am – 5pm EST. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Hayes can be reached at 571.272.6708. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http:/www.uspto.gov/interviewpractice.
As disclosed in MPEP 502.03, communications via Internet e-mail are at the discretion of the applicant. Without a written authorization by applicant in place, the USPTO will not respond via Internet e-mail to any Internet correspondence which contains information subject to the confidentiality requirement as set forth in 35 U.S.C. 122. A paper copy of such correspondence will be placed in the appropriate patent application. The following is a sample authorization form which may be used by applicant:
“Recognizing that Internet communications are not secure, I hereby authorize the USPTO to communicate with me concerning any subject matter of this application by electronic mail. I understand that a copy of these communications will be made of record in the application file.”
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center information webpage. Status information for unpublished applications is available to registered users through Patent Center information webpage only.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (in USA or CANADA) or 571-272-1000.
Any response to this action should be mailed to:
Commissioner of Patents and Trademarks
P.O. Box 1450
Alexandria, VA 22313-1450
or faxed to 571-273-8300
/Radu Andrei/
Primary Examiner, AU 3697
Prosecution Insights