DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner notes
Examiner cites particular columns, paragraphs, figures and line numbers in the references as applied to the claims below for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested that, in preparing responses, the applicant fully consider the references in their entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage as taught by the prior art or disclosed by the examiner. The entire reference is considered to provide disclosure relating to the claimed invention. The claims & only the claims form the metes & bounds of the invention. Office personnel are to give the claims their broadest reasonable interpretation in light of the supporting disclosure. Unclaimed limitations appearing in the specification are not read into the claim. Prior art was referenced using terminology familiar to one of ordinary skill in the art. Such an approach is broad in concept and can be either explicit or implicit in meaning. Examiner's Notes are provided with the cited references to assist the applicant to better understand how the examiner interprets the applied prior art. Such comments are entirely consistent with the intent & spirit of compact prosecution.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 7, 8, 9-11, 17 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 7 recites the limitation "or the fourth category ". There is insufficient antecedent basis for this limitation in the claim. Suggested correction, either add the fourth category definition to claim 7 or restructure the dependency so the fourth category is inherited from claim 4.
Claim 8 recites the limitation “the trained neural network-based machine learning model.” There is insufficient antecedent basis for this limitation in the claim. Suggested correction, restructure the dependency to depend on claim 6.
Claim 17 recites the limitation “a server node according to claim 1” but claim 1 claims “A method of operating a server node”. Claim 17 is therefore rejected as being indefinite. Suggested correction, change verbiage to “a server node configured to perform the operations of claim 1, or a server node actually performing the method of claim 1.
Claims 9-11 are rejected for their dependency on either a rejected base claim or on claim dependent on a rejected base claim.
Correction is therefore required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefore, subject to the conditions and requirements of this title.
Claims 1-11, 13, and 17 are rejected under 35 U.S.C. 101 because the claimed invention is directed towards an abstract idea without significantly more.
Claim 1.
A method of operating a server node implementing a Lightweight Machine-to-Machine, LWM2M protocol, the method comprising: obtaining sensor data comprising values of a metric measured in an environment by a client node implementing the LWM2M protocol, wherein the sensor data further comprises a metric identifier; based on the metric identifier, determining a controllability parameter value representing an extent of controllability of the metric by a reinforcement learning agent operating on the environment; annotating the sensor data with the determined controllability parameter value; and providing the annotated sensor data for training the machine learning model simulating the environment.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards an abstract idea. The claim recites the following limitations: (Highlighted portions of the claim in bold above constitute an abstract idea; the remaining limitations are "additional elements")
Determining and annotating limitations are directed towards the abstract idea of a mental process, or a concept performed in the human mind, including observation, evaluation, judgement or opinion (see MPEP § 2106.04(a)(2)(III). The courts do not distinguish between mental processes that are performed entirely in the human mind and mental processes that require a human to use a physical aid (e.g., pen and paper or a slide rule) to perform the claim limitation. This limitation could be performed in the human mind or with the aid of pen and paper.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The obtaining sensor data limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)).
The providing the annotated sensor limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The obtaining sensor data limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)).
The providing the annotated sensor limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log.
Claim 2.
A method according to claim 1: storing a list of metric identifiers and respective controllability parameter values; wherein the controllability parameter value is determined based on a comparison of the metric identifier with the metric identifiers in the list.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The storing a list limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log.
The controllability parameter determination is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The storing a list limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log.
The controllability parameter determination is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
Claim 3.
A method according to claim 2, wherein each metric identifier corresponds to a LwM2M Object stored in a LwM2M registry, and wherein each controllability parameter value corresponds to a LwM2M Resource stored in the LwM2M registry for the respective LwM2M Object.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 2 which itself depends on the abstract idea of claim1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The metric identifier corresponding limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The metric identifier corresponding limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not amount to significantly more than the judicial exception.
Claim 4.
A method according to claim 2, wherein the respective controllability parameter values are selected from a group comprising: a first category indicating that the associated metric is indirectly controllable by the reinforcement learning agent; a second category indicating that the associated metric is not controllable by the reinforcement learning agent; a third category indicating that the associated metric is directly controllable by the reinforcement learning agent; a fourth category indicating that the values of the associated metric are measured at a first time instance, wherein the first time instance is preceding a second time instance at which values of the metric associated with the first category were measured.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 2 which itself depends on the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The grouping of controllability parameter limitations are directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)). This limitation is just checking information in the registry.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The grouping of controllability parameter limitations are directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)). This limitation is just checking information in the registry.
Claim 5.
A method according to claim 1, wherein providing the annotated sensor data for training the machine learning model comprises: sending the annotated sensor data to another node implementing the machine learning model.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
Sending annotated data is directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
Sending annotated data is directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
Claim 6.
A method according to claim 1, wherein providing the annotated sensor data for training the machine learning model comprises: training a neural network-based machine learning model using the annotated sensor data.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The training a neural network limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations. Neural networks are mathematical correlations that achieve the highest probability.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The training a neural network limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations. Neural networks are mathematical correlations that achieve the highest probability.
Claim 7.
A method according to claim 6, further comprising storing a list of metric identifiers and respective controllability parameter values; wherein the controllability parameter value is determined based on a comparison of the metric identifier with the metric identifiers in the list; wherein the respective controllability parameter values are selected from a group comprising: a first category indicating that the associated metric is indirectly controllable by the reinforcement learning agent; a second category indicating that the associated metric is not controllable by the reinforcement learning agent; a third category indicating that the associated metric is directly controllable by the reinforcement learning agent; wherein the annotated sensor data comprises values of a metric of the first category and values of a metric of the at least one of the second category, third category or the fourth category, and wherein the trained neural net- work-based machine learning model is configured to predict values of a metric of the first category.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The storing a list limitation, and the controllability parameter grouping limitations are directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log
The controllability parameter limitation is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
The is configured to predict limitation is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The storing a list limitation, and the controllability parameter grouping limitations are directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (v.) Consulting and updating an activity log
The controllability parameter limitation is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
The is configured to predict limitation is directed towards the abstract concept of mathematical calculations. (See MPEP § 2106.04(a)(2)(C)).
Claim 8.
A method according to claim 4, further comprising: training the reinforcement learning agent by interacting with the trained neural network-based machine learning model.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The training the reinforcement learning agent limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations. Training a learning agent is based on mathematical correlations.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The training the reinforcement learning agent limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations. Training a learning agent is based on mathematical correlations.
Claim 9.
A method according to claim 8, wherein training the reinforcement learning agent comprises: taking an action on the trained neural network-based machine learning model using the reinforcement learning agent, wherein the action is performed on a metric of the third category; generating a reward value using the machine learning model responsive to taking the action, wherein the reward value is based on a metric of the first category; updating a policy of the reinforcement learning agent based on the reward value.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The taking an action and updating a policy limitations are directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (IV) organizing information and manipulating information through mathematical correlations.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The taking an action and updating a policy limitations are directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (IV) organizing information and manipulating information through mathematical correlations.
Claim 10.
A method according to claim 9, further comprising controlling the environment using the trained reinforcement learning agent based on the updated policy.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The controlling the environment limitation is directed towards abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (IV) organizing information and manipulating information through mathematical correlations.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The controlling the environment limitation is directed towards abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (IV) organizing information and manipulating information through mathematical correlations.
Claim 11.
A method according to claim 10, wherein the environment comprises a dynamic system, wherein the dynamic system comprises one or more sensors or actuators operating in a communications network.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
The wherein the environment comprises limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
The wherein the environment comprises limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
Claim 12. (canceled)
Claim 13.
A server node implementing a Lightweight Machine to- Machine, LWM2M protocol, the server node comprising a processing circuit; and a memory coupled to the processing circuit wherein the memory comprises computer readable program instructions that, when executed by the processing circuit, cause the server node to perform operations according to claim 1.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
Memory and processors are generic computer components; generic computer components provide instructions to invoke computers as a tool in its ordinary capacity (MPEP 2106.05(f)) Step 2B – The claims as a whole do not integrate the exception into a practical application.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
Memory and processors are generic computer components; generic computer components provide instructions to invoke computers as a tool in their ordinary capacity (MPEP 2106.05(f).
Claim 14.-16. (canceled)
Claim 17.
A system comprising: a server node according to claim 1; a machine learning model simulating the environment, wherein the machine learning model is communicatively couplable with the server node; a reinforcement learning agent communicatively couplable with the machine learning model and the environment; wherein the server node provides the annotated sensor data for training the machine learning model, and wherein the reinforcement learning agent is configured to control the environment based on the trained machine learning model.
Step 1 – The claim is directed towards a method, one of the four statutory categories.
Step 2A Prong 1 – The claim is directed towards the abstract idea of claim 1.
Step 2A Prong 2 – The claim does not recite any additional elements which integrate the abstract idea into a practical application.
A machine learning model simulating the environment and the reinforcement learning agent limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
The provides the annotated sensor data limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)).
The configured to control the environment limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations.
Step 2B – The claims as a whole do not amount to significantly more than the judicial exception.
A machine learning model simulating the environment and the reinforcement learning agent limitations are directed towards an insignificant extra-solution activity of whether the limitation is significant. (2106.05(g)(2)). This limitation is recited at a high level of generality; it does not bring this out of the realm of insignificant extra solution activity.
The provides the annotated sensor data limitation is directed towards an insignificant extra-solution activity of mere data gathering, or outputting in particular section (iv) Obtaining information about. . . (See MPEP §2106.05(g)(3)(iv)).
The configured to control the environment limitation is directed towards the abstract concept of mathematical relationships. (See MPEP § 2106.04(a)(2)(I)(A)). In particular section (iv). organizing information and manipulating information through mathematical correlations.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-11, 13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Ahn et al., US 20210049460 (Ahn) in view of Govindaraju et al., US 10212261 (Govindaraju) in further view of Tabatabaei et al., US 20170094592 (Tabatabaei).
Claim 1.
Ahn teaches based on the metric identifier, determining a controllability parameter value representing an extent of controllability of the metric by a reinforcement learning agent operating on the environment; (Ahn 0026) “such as action inputs (controllable inputs to be planned under constraints condition inputs (uncontrollable or exogenous inputs given by the process context and previous action selections).” {EXAMINERS NOTE: Applicants spec pg. 3 lines 30 treats the controllability parameter as “representing an extent of controllability of the metric by a reinforcement learning agent operating on the environment”, therefore Ahn with its directly uncontrollable and controllable action inputs are considered parameter values of control.}
and providing the annotated sensor data for training the machine learning model simulating the environment. (Ahn 0189-0191) “one or more of actions 1303, conditions 1304, targets 1305, sensors 1306, and states 1307. . . With the training data 1312 and the identified features 1302, the machine-learning tool is trained at operation 1314.”
Ahn does not teach, but Govindaraju teaches A method of operating a server node implementing a Lightweight Machine-to-Machine, LWM2M protocol, the method comprising: obtaining sensor data comprising values of a metric measured in an environment by a client node implementing the LWM2M protocol, (Govindaraju col 5 Lines 38-40) “Sensor 105 measures and collects surrounding physical and/or environmental conditions (generally referred to as sensor data).”
wherein the sensor data further comprises a metric identifier; (Govindaraju col 13 Lines 7-8) “and each Resource 510 is identified by a Resource ID”
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju to more “reliably predictive simulation model using the historical process data that companies have accumulated over time.” (Ahn 0003)
Modified Ahn with Govindaraju do not teach, but Tabatabaei teaches annotating the sensor data with the determined controllability parameter value; (Tabatabaei 0010) “The sensory data in LSM is annotated and transformed into RDF triples.”
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju and the of Tabatabaei to minimize traffic load as (0003) of Tabatabaei suggests.
Claim 2.
Modified Ahn with Govindaraju teaches A method according to claim 1: storing a list of metric identifiers and respective controllability parameter values; (Govindaraju col 13 Lines 3-8) “LWM2M defines an Object Registry and Resource Registry . . . and each Resource 510 is identified by a Resource ID.”
wherein the (Govindaraju Col 13 Lines 41-43) “LWM2M Server 502 manipulates Resources 510 on LWM2M Client 504 using commands, such as read, write, execute, create, delete, write attributes, and/or discover.” {EXAMINERS NOTE: Under BRI, an identifier based registry lookup involves matching/comparing the incoming identifier with stored identifiers. The controllability parameter aspect comes from Ahn.}
Claim 3.
Modified Ahn with Govindaraju teaches A method according to claim 2, wherein each metric identifier corresponds to a LwM2M Object stored in a LwM2M registry, and wherein each controllability parameter value corresponds to a LwM2M Resource stored in the LwM2M registry for the respective LwM2M Object. (Govindaraju col 13 Lines 3-9) “LWM2M defines an Object Registry and Resource Registry . . . Each Object 512 defines a group of Resources 510, and each Resource 510 is identified by a Resource ID. Resources 510 are registered as specific to each Object 512.” (col 13 Lines 40-41) “LWM2M also enables other Objects and/or Resources to be defined to support desired M2M services” {EXAMINERS NOTE: Metric identifier corresponds to the object ID identifying the particular LwM2M object associated with the metric and is stored in a registry.}
Claim 4.
Modified Ahn teaches A method according to claim 2, wherein the respective controllability parameter values are selected from a group comprising: a first category indicating that the associated metric is indirectly controllable by the reinforcement learning agent; (Ahn 0026) “target outputs (utility observations to be predicted by the process model and optimized by the action policy).” (0029) “These targets are optimized using the controllable actions of battery discharging and recharging.” (0049) “The decision engine 204 provides the reinforcement learning based on predictive simulations in model-based reinforcement learning (RL) or model-predictive control (MPC) approaches.”
a second category indicating that the associated metric is not controllable by the reinforcement learning agent; (Ahn 0026) “condition inputs (uncontrollable or exogenous inputs given by the process context and previous action selections),” (0224) Conditions ( c ): Uncontrollable/Exogenous features. (0065) “and conditions c, (uncontrollable and exogenously given inputs) are effective right after timestep t,”
a third category indicating that the associated metric is directly controllable by the reinforcement learning agent; (Ahn 0026) “action inputs (controllable inputs to be planned under constraints),” (0065) “It is assumed that actions a, ( controllable inputs from the agent's decision making) and conditions c, (uncontrollable and exogenously given inputs) are effective right after timestep t, and they are applied over the timestep t+1.”
a fourth category indicating that the values of the associated metric are measured at a first time instance, wherein the first time instance is preceding a second time instance at which values of the metric associated with the first category were measured. (Ahn 0064) “vector-valued sensor measurements x, are observed at timestep t and vector valued target measurements y, at each timestep t.” (0428) In one example, the method 4500 further comprises predicting a sequence of states of the real process following a current state based on previous values of actions, conditions, sensors, and targets. (0235) “Each array has a size equal to the number of past observations used plus the number of future observations that are going to be predicted.”
Claim 5.
Modified Ahn teaches A method according to claim 1, wherein providing the annotated sensor data for training the machine learning model comprises: sending the annotated sensor data to another node implementing the machine learning model. (Ahn 0304) “The FRAI backend includes an internal database for the DPDM system, an application server that interfaces with the FRAI app, and the DPDM model engine. A secure connection is used for communications between the business process systems and the FRAI backend.”
Claim 6.
Modified Ahn teaches A method according to claim 1, wherein providing the annotated sensor data for training the machine learning model comprises: training a neural network-based machine learning model using the annotated sensor data. (Ahn 0216) “This application focuses on training a model that can simulate the plant state and KPI measurements given the set of future action and conditions.” (0217) “In many enterprise processes such as rubber extrusion, the agent is not allowed to learn and optimize its decision-making or control policy through a lot of trials in the real environment. Generating the real-world experience data from executing numerous actions and conditions is very time-consuming, expensive, and even dangerous when the input ranges happen not to be properly set. For this reason, it is very desirable to build a predictive simulation model using the historical process data that companies have accumulated. Then, simulated experiences for hypothetical actions and conditions enable the agent to make optimal decisions.”
Claim 7.
Modified Ahn with Govindaraju teaches A method according to claim 6, further comprising storing a list of metric identifiers and respective controllability parameter values; wherein the (Govindaraju col 13 Lines 3-8) “LWM2M defines an Object Registry and Resource Registry . . . and each Resource 510 is identified by a Resource ID.” (Col 13 Lines 41-43) “LWM2M Server 502 manipulates Resources 510 on LWM2M Client 504 using commands, such as read, write, execute, create, delete, write attributes, and/or discover.” {EXAMINERS NOTE: Under BRI, an identifier based registry lookup involves matching/comparing the incoming identifier with stored identifiers. The controllability parameter aspect comes from Ahn.}
wherein the respective controllability parameter values are selected from a group comprising: a first category indicating that the associated metric is indirectly controllable by the reinforcement learning agent; (Ahn 0026) “target outputs (utility observations to be predicted by the process model and optimized by the action policy).” (0221) “Profile Measurements (y): Utilities/Targets to be predicted and optimized;”
a second category indicating that the associated metric is not controllable by the reinforcement learning agent; (Ahn 0026) “condition inputs (uncontrollable or exogenous inputs given by the process context and previous action selections.)” (0224) “Conditions ( c ): Uncontrollable/Exogenous features.”
a third category indicating that the associated metric is directly controllable by the reinforcement learning agent; (Ahn 0065) “It is assumed that actions a, ( controllable inputs from the agent's decision making)” (0023) “Actions (a): Controllable inputs to be planned.”
wherein the annotated sensor data comprises values of a metric of the first category and values of a metric of the at least one of the second category, third category or the fourth category, (Ahn 0189) “the features 1302 may be of different types and may include one or more of actions 1303, conditions 1304, targets 1305, sensors 1306, and states 1307.” (0191) “With the training data 1312 and the identified features 1302, the machine-learning tool is trained at operation 1314. The machine-learning tool appraises the value of the features 1302 as they correlate to the outcomes in the training data 1312. The result of the training is the trained machine-learning program 1316.”
and wherein the trained neural net-work-based machine learning model is configured to predict values of a metric of the first category. (Ahn 0026) “target outputs (utility observations to be predicted by the process model and optimized by the action policy).”
Claim 8.
Modified Ahn teaches A method according to claim 4, further comprising: training the reinforcement learning agent by interacting with the trained neural network-based machine learning model. (Ahn 0120) “whose mean and standard deviation are multi-layer neural network functions of the last posterior state”
Claim 9.
Modified Ahn teaches A method according to claim 8, wherein training the reinforcement learning agent comprises: taking an action on the trained neural network-based machine learning model using the reinforcement learning agent, wherein the action is performed on a metric of the third category; (Ahn 0065) “It is assumed that actions a, (controllable inputs from the agent's decision making) and conditions c, (uncontrollable and exogenously given inputs) are effective right after timestep t.” (0159) “With model-based reinforcement learning, the agent's actions at” (0160) “An RL approach using the process simulation model can make the optimal action policy to maximize a given long-term utility goal.”
generating a reward value using the machine learning model responsive to taking the action, wherein the reward value is based on a metric of the first category; (Ahn 0219) “The deep probabilistic (DP) dynamic model tells us how the state of the system will evolve as a result of the actions taken based on observed sequences ( conditions, actions, observations over time).” (0162) “Assuming that the utility function r is given as a function of the predicted target yt.”
updating a policy of the reinforcement learning agent based on the reward value. (Ahn 0172) In reinforcement learning the averaged R can be used to optimize the policy parameters 0 via the stochastic gradient ascent.
Claim 10.
Modified Ahn teaches A method according to claim 9, further comprising controlling the environment using the trained reinforcement learning agent based on the updated policy. (Ahn 0041) “the action policy or decision-making controller is trained for selecting optimal actions to achieve the maximum KPI for future conditions.” (0172) “the averaged R can be used to optimize the policy parameters 0.”
Claim 11.
Modified Ahn with Govindaraju teaches A method according to claim 10, wherein the environment comprises a dynamic system, (Ahn 0028) “an industrial dynamic process. . . ” (0200) “This is a highly complex problem to solve as it is a dynamic process that involves a complex system with multiple intercorrelated features.”
wherein the dynamic system comprises one or more sensors or actuators operating in a communications network. (Govindaraju col 3 Lines 44-46) “Wireless sensor network system 10 includes various wireless sensor network nodes that cooperatively monitor, measure, and collect surrounding physical and/or environmental conditions.”
12. (canceled)
Claim 13.
Modified Ahn with Govindaraju and Tabatabaei teaches A server node implementing a Lightweight Machine to- Machine, (Govindaraju col 28 Lines 13-15) “A wireless sensor network node for network connectivity in a wireless sensor network system having con-strained leaf nodes which do not implement a network layer. . . (LWM2M) application layer gateway is configured for:”
LWM2M protocol, the server node comprising a processing circuit; (Govindaraju col 28 Line 23) “a processor operable to execute the instructions”
and a memory coupled to the processing circuit wherein the memory comprises computer readable program instructions that, (Govindaraju col 29 Line 8) “a memory for storing data” (col 29 Lines 10-11) “the processor and the memory 10 cooperate. . .”
when executed by the processing circuit, cause the server node to perform operations according to claim 1. (Govindaraju col 24 Lines 7-8) “executed by a processor to carry out the activities described herein.”
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju to more “reliably predictive simulation model using the historical process data that companies have accumulated over time.” (Ahn 0003)
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju and the of Tabatabaei to minimize traffic load as (0003) of Tabatabaei suggests.
14.-16. ( canceled)
Claim 17.
Modified Ahn with Govindaraju and Tabatabaei teaches A system comprising: a server node according to claim 1; a machine learning model simulating the environment, (Ahn 0056) “DPDM includes the decision-making controller 110 that generates simulated actions and the deep probabilistic (DP) simulated process 112.”
wherein the machine learning model is communicatively couplable with the server node; (Ahn 0057) “Further, the DP simulated process 112 receives the actual observations from the real processes 104, and the actual observations are used to update the simulation process.”
a reinforcement learning agent communicatively couplable with the machine learning model and the environment; (Ahn 0049) “The decision engine 204 provides the reinforcement learning based on predictive simulations in model-based reinforcement learning (RL) or model-predictive control (MPC) approaches.”
wherein the server node provides the annotated sensor data for training the machine learning model, (Tabatabaei 0010) “The sensory data in LSM is annotated and transformed into RDF triples.”
and wherein the reinforcement learning agent is configured to control the environment based on the trained machine learning model. (Ahn 0055) “DPDM 302 interacts with the real processes 104 (having current state 106) by setting actual actions for the real process 104, and by receiving actual observations from the real processes 104.”
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju to more “reliably predictive simulation model using the historical process data that companies have accumulated over time.” (Ahn 0003)
are analogous to the claimed invention because Ahn is from the same field of endeavor of
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art, having the teachings of Ahn and Govindaraju before him or her, to modify the controllability parameters of Ahn with sensor data of Govindaraju and the of Tabatabaei to minimize traffic load as (0003) of Tabatabaei suggests.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN DAVID HAGLER whose telephone number is (703)756-1339. The examiner can normally be reached Monday - Friday 10am- 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rehana Perveen can be reached at 5712723676. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN DAVID HAGLER/ Examiner, Art Unit 2189
/REHANA PERVEEN/ Supervisory Patent Examiner, Art Unit 2189
Prosecution Insights