DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the applicant’s communication filed on 9/5/2024
Claims 1-6 are pending.
Specification
Applicant is reminded of the proper content of an abstract of the disclosure.
A patent abstract is a concise statement of the technical disclosure of the patent and should include that which is new in the art to which the invention pertains. The abstract should not refer to purported merits or speculative applications of the invention and should not compare the invention with the prior art.
If the patent is of a basic nature, the entire technical disclosure may be new in the art, and the abstract should be directed to the entire disclosure. If the patent is in the nature of an improvement in an old apparatus, process, product, or composition, the abstract should include the technical disclosure of the improvement. The abstract should also mention by way of example any preferred modifications or alternatives.
Where applicable, the abstract should include the following: (1) if a machine or apparatus, its organization and operation; (2) if an article, its method of making; (3) if a chemical compound, its identity and use; (4) if a mixture, its ingredients; (5) if a process, the steps.
Extensive mechanical and design details of an apparatus should not be included in the abstract. The abstract should be in narrative form and generally limited to a single paragraph within the range of 50 to 150 words in length.
See MPEP § 608.01(b) for guidelines for the preparation of patent abstracts.
Claim Objections
Claim 1 objected to because of the following informalities: “configured to: wherein” in lines 4-6 is grammatically incorrect. Appropriate correction is required.
Claim 5 objected to because of the following informalities: “execute processing of: wherein” in lines 3-4 is grammatically incorrect. Appropriate correction is required.
Claim 6 objected to because of the following informalities: “execute processing of: wherein” in lines 4-5 is grammatically incorrect. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1, 5, and 6 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the setting in the reinforcement learning" in line 8 and “the setting of the agent model” in line 17. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 recites the limitation "the setting in the reinforcement learning" in line 6 and “the setting of the agent model” in line 11. There is insufficient antecedent basis for this limitation in the claim.
Claim 6 recites the limitation "the setting in the reinforcement learning" in line 7 and “the setting of the agent model” in line 12. There is insufficient antecedent basis for this limitation in the claim.
The dependent claims are also rejected under 35 U.S.C. § 112 as they inherit all of the characteristics of the claim from which they depend and none of the dependent claims provide a cure for the indefiniteness of the parent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-6 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding independent claim 1, at step 1, the claim recites a reinforcement learning device comprising a memory and at least one processor coupled to the memory, and therefore is a machine, which is a statutory category of invention.
At step 2A, prong one, the claim recites “predetermined settings for simulation and an agent model are stored”; “in the simulation, based on the setting in the reinforcement learning, a state in a next trial, a reward according to the state, and a flag indicating whether simulation execution has ended are acquired with a behavior defined in advance as an input”; “inputs the state acquired by the simulation to the agent model and acquires a measure”; “calculates the behavior based on the measure and a search amount defined in advance”; “updates the agent model according to the setting of the agent model based on the state, the reward, the flag, and the behavior”; “updates the search amount based on a prediction reward obtained for the reward and the search amount in a previous trial”, and “the calculation of the behavior, the update of the agent model, and the update of the search amount are repeated until a predetermined condition according to the flag and the setting is satisfied”.
The above limitations, under their broadest reasonable interpretation, recite mathematical concepts and/or mental processes. For example, the claim recites reinforcement-learning data processing including calculating a behavior, updating an agent model, updating a search amount, and repeating the calculations/updates based on data and predetermined conditions. These limitations amount to applying reinforcement-learning rules, mathematical relationships, and data evaluations to update model parameters and determine behavior. Thus, the claim recites an abstract idea, namely mathematical concepts and/or mental processes of performing reinforcement learning calculations for determining a behavior and updating an agent model and search amount based on data. See MPEP 2106.04(a).
At step 2A, prong two, this judicial exception is not integrated into a practical application. The additional elements, including the reinforcement learning device, memory, and at least one processor, merely apply the abstract idea using generic computer technology. The recited simulation, agent model, state, reward, flag, behavior, measure, search amount, prediction reward, and predetermined condition are used as part of the abstract reinforcement-learning calculation and do not impose meaningful limits on the abstract idea. The claim does not recite an improvement to the functioning of the computer itself, an improvement to another technology or technical field, or any particular machine that applies or uses the calculated behavior to effect a real-world technological change. Rather, the claim is directed to calculating and updating reinforcement-learning information using generic computer components. Accordingly, the claim does not integrate the abstract idea into a practical application. See MPEP 2106.04(d) and 2106.05(f).
At step 2B, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of a reinforcement learning device, memory, and at least one processor amount to no more than instructions to apply the abstract idea using generic computer components. Such generic computer implementation does not provide an inventive concept. The remaining limitations are part of the abstract idea itself because they recite the reinforcement-learning data, calculations, model update, search amount update, and repeated execution of the abstract reinforcement-learning process. Considering the additional elements individually and in combination and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Thus, claim 1 is not patent eligible.
Regarding Independent Claim 5, the claim recites substantively the same abstract idea identified in claim 1 above; and recites substantively similar additional elements (a method for performing the abstract idea) and is ineligible for the same reasons as those indicated in the analysis of claim 1 above.
Regarding Independent Claim 6, the claim recites substantively the same abstract idea identified in claim 1 above; and recites substantively similar additional elements (a non-transitory computer readable medium storing a program executable by a computer to perform the abstract idea) and is ineligible for the same reasons as those indicated in the analysis of claim 1 above.
Regarding dependent claim 2, the additional limitations of “calculates the prediction reward based on a parameter of a learning rate of the prediction reward determined in the setting and the reward,” and “updates a search amount based on the calculated prediction reward and a parameter for estimating the search amount in the setting” merely further specify the reinforcement-learning data processing/calculations identified above because they recite calculating reward-related information and updating search amount information based on parameters. These limitations are part of the abstract idea and do not integrate the abstract idea into a practical application. Thus, claim 2 does not add significantly more than the abstract idea.
Regarding dependent claim 3, the additional limitations of “wherein the behavior determined by the behavior determination unit is probabilistically determined according to a probability density function using a random variable and an average and a variance of a normal distribution represented by the measure” merely recites mathematical concepts because the claim expressly recites a probability density function, random variable, average, variance, and normal distribution. These limitations are part of the abstract idea and do not integrate the abstract idea into a practical application. Thus, claim 3 does not add significantly more than the abstract idea.
Regarding dependent claim 4, the additional limitation of “wherein the behavior is an air conditioning control method.” merely links the abstract reinforcement-learning calculation to a particular field of use. Thus, claim 4 does not integrate the abstract idea into a practical application and does not add significantly more than the abstract idea. See MPEP 2106.05(h).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-3, and 5-6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Macglashan USPGPUB 2020/0302323 A1 (hereinafter Macglashan) in view of Khamassi et al. (Active exploration in parameterized reinforcement learning, 2016) (hereinafter Khamassi).
Macglashan teaches a reinforcement learning device that performs reinforcement learning for a continuous behavior space (Par. [0043], “the DAC algorithm can optimize reinforcement learning problems with discrete and continuous action spaces” – continuous action space is interpreted as continuous behavior space), comprising:
a memory (Par. [0011], “non-transitory computer-readable storage medium with an executable program stored thereon”); and
at least one processor coupled to the memory (Par. [0030], “Typically, a processor (e.g., a micro-processor) will receive instructions from a memory or like device, and execute those instructions, thereby performing a process defined by those instructions.”), the at least one processor being configured to:
wherein
predetermined settings for simulation and an agent model are stored (Par. [0009], “method of training a policy model and an action-value model of an agent,”; Par. [0082], “The online algorithm takes the same hyperparameters as the offline algorithm, except instead of a dataset, it receives a reference to an environment with which it can interact.”; Par. [0010], “γ is a discount factor in a domain [0, 1) that defines how valued future rewards are to more immediate rewards”; Par. [0068], “t is a "temperature" hyperparameter that defines how greedy the target distribution is towards the highest scoring Q-value”; Par. [0011], “non-transitory computer-readable storage medium with an executable program stored thereon” – γ and t correspond to predetermined settings because they are defined parameters used by the reinforcement-learning algorithm. Stored executable programs on a storage medium teaches that the reinforcement-learning program/settings are stored),
in the simulation based on the setting in the reinforcement learning, a state in a next trial (Par. [0081], Algorithm 2 Online DAC, “s’, r <- env.execute(a)”; Par. [0082], “it observes the current state; selects and executes an action using the actor; observes the resulting state and reward” - s’ corresponds to the next state after executing action a. Because Macglashan repeats the environment interaction loop, the resulting state s’ corresponds to the state in the next trial.), a reward according to the state (Par. [0082], “observes the resulting state and reward” – r corresponds to the reward associated with the resulting state), and a flag indicating whether simulation execution has ended (Par. [0081], Algorithm 2 Online DAC, “loop … end loop” – Macglashan teaches repeated environment execution through the environment loop that determines whether the simulation should continue or end. It would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether simulation execution has ended.) are acquired with a behavior defined in advance as an input (Par. [0081], Algorithm 2 Online DAC – action a corresponds to the claimed behavior, and the action/behavior is provided to the environment execution as an input to acquire the resulting state and reward.),
the reinforcement learning device comprises:
inputs the state acquired by the simulation to the agent model and acquires a measure (Par. [0081], Algorithm 2 Online DAC, “s<-env.observe() a~ π(s; φ)” – the observed state s from the environment is input to the policy model π; Claim 4, “an output of the policy model, π(s), for a given observation (s) of an environment state, are parameters of probability distributions over a domain of an action space” – policy model corresponds to the agent model, and the output π(s) corresponds to the measure);
calculates the behavior based on the measure defined in advance (Par. [0081], Algorithm 2 Online DAC, “a ~ π(s; φ) s’, r <- env.execute(a)”; Par. [0082], “selects and executes an action using the actor; observes the resulting state and reward”; Par. [0083], “an actor may have been previously trained using another learning paradigm (e.g., learning from demonstration), or have been hard coded by some means” – action a corresponds to the behavior, The action a is defined before executing env.execute(a), and the action a is then used as the input to the environment execution to acquire the resulting state s’ and reward r.);
updates the agent model according to the setting of the agent model based on the state, the reward, the flag, and the behavior (Par. [0065], Algorithm 1, “DAC_offline_step(data, θ, φ, θ′, φ′, k, c) … sample minibatch (s, a, r, s′) from data … OPTIMIZER_UPDATE…”; Par. [0082], “observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step.” – Macglashan updates the action-value model and policy model using transition data. State s/s’ corresponds to the claimed state, action a corresponds to the claimed behavior, reward r corresponds to the claimed reward, and θ, φ, θ′, φ′, k, and c correspond to settings of the agent model/update process. Macglashan teaches repeated environment execution through a loop, and it would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether the transition is part of a continuing simulation or whether the simulation execution has ended.),
the calculation of the behavior, and the update of the agent model are repeated until a predetermined condition according to the flag and the setting is satisfied (Par. [0082], “It then repeats a series of interactions with the environment in which it observes the current state; selects and executes an action using the actor; observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step”; Par. [0065], Algorithm 1 Offline DAC, “for c times do” – Macglashan repeats executing an action using the actor, which corresponds to repeating the calculation of the behavior, and repeats running the offline algorithm step, which corresponds to repeating the update of the agent model. “for c times” teaches a predetermined update condition because the update step is repeated a set number of times according to the parameter c. As discussed above, it would have been obvious to use a flag indicating whether simulation execution has ended to determine whether the repeated online interaction should continue or terminate.).
Macglashan does not explicitly teach a search amount estimation unit for estimating the search amount;
calculates the behavior based on a search amount defined in advance;
updates the search amount based on a prediction reward obtained for the reward and the search amount in a previous trial, and
repeating update of the search amount until a predetermined condition is satisfied.
However, Khamassi teaches a search amount estimation unit for estimating the search amount (Page 1, Abstract, “We apply a meta-learning algorithm based on the comparison between variations of short-term and long-term reward running averages to simultaneously tune and the width of the Gaussian distribution from which continuous action parameters are drawn.” – the meta-learning algorithm is interpreted as the estimation unit as the instant Specification describes the CPU acting as the search amount estimation unit; Page 2, Introduction, “At each timestep, we use the difference between the two averages to simultaneously tune the inverse temperature βt used for selecting between discrete actions αj, and the width ot of the Gaussian distribution from which each continuous action parameter thetaia is sampled around its current value.” – Khamassi’s ot, i.e., the width of the Gaussian distribution used to sample continuous action parameters, corresponds to the claimed search amount because it controls the range/spread of candidate continuous action parameters explored during behavior selection.);
calculates the behavior based on a search amount defined in advance (Page 3, Active exploration algorithm, “continuous … parameters with which action αj is executed at timestep t are selected from a Gaussian exploration function centered on the current values … in state st … where the width ot of the Gaussian is a meta-parameter which will be tuned through meta-learning; Page 4, Algorithm 1, “initialize … β0 and o0” - ot corresponds to the claimed search amount because ot controls the width of the Gaussian exploration used to select continuous action parameters and o0 corresponds to a search amount defined in advance because it is initialized before the repeated learning steps.);
updates the search amount based on a prediction reward obtained for the reward and the search amount in a previous trial (Page 3, Active exploration algorithm, “We compute short- and long-term reward running averages: where t1 and t2 are two time constants. We then update βt and ot with: …” – Gaussian width ot corresponds to the claimed search amount because ot controls the spread/range of continuous action parameters sampled during exploration. The reward-running averages correspond to the claimed prediction-reward information because they are reward-derived values used to update ot.), and
repeating update of the search amount until a predetermined condition is satisfied (Page 1, Introduction, “pre-adjusting parameters such as the exploration rate based on the prior determination of the total number of episodes in the experiment.“; Page 4, Algorithm 1, “for t= 0, 1, 2, … do … if meta-learning then update reward running averages … update βt and ot end if” – ot is repeatedly updated in the reinforcement-learning loop until the meta-learning condition is satisfied, where ot corresponds to the search amount. The meta-learning condition corresponds to a setting for performing the search-amount update. The search-amount update is also repeated during the learning loop until the predetermined experiment length/number of episodes is reached.).
Macglashan and Khamassi are analogous art because they are from the same field of endeavor and contain functional similarities. They both relate to reinforcement learning systems for selecting actions in continuous action spaces and improving learning through actor-critic/exploration techniques.
Therefore, at the time of effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the above reinforcement learning device, as taught by Macglashan, and incorporate a meta-learning update of the Gaussian width, which corresponds to the claimed search amount for continuous action exploration, as taught by Khamassi.
One of ordinary skill in the art would have been motivated to improve the exploration/exploitation tradeoff, thereby improving the reinforcement learning model’s ability to select effective continuous action parameters and adapt during learning, as suggested by Khamassi (Page 2, Introduction).
Regarding claim 2, the combination of Macglashan and Khamassi teaches all the limitations of the base claims as outlined above.
Khamassi further teaches wherein calculates the prediction reward based on a parameter of a learning rate of the prediction reward determined in the setting and the reward (Page 3, Active exploration algorithm, “We compute short- and long-term reward running averages: Δr̄(t) = (r(t) − r̄(t))/t1 and Δr̄̄(t) = (r̄(t) − r̄̄(t))/t2 where t1 and t2 are two time constants” – The short-term and long-term reward running averages correspond to the prediction reward because they are reward-derived values representing prior reward performance. Khamassi calculates the reward running averages based on the reward r(t). t1 and t2 correspond to parameters of a learning rate of the prediction reward determined in the setting because t1 and t2 are predetermined time constants, and the update-rate coefficients 1/t1 and 1/t2 control how quickly the reward running averages are updated based on the received reward. The rate at which the reward running averages are updated corresponds to the learning rate), and updates a search amount based on the calculated prediction reward and a parameter for estimating the search amount in the setting (Page 3, Active exploration algorithm, “We then update βt and ot with: … where μ is a learning rate” – ot corresponds to the claimed search amount because ot is the width of the Gaussian distribution used to sample continuous action parameters. The reward-running-average difference r̄(t) − r̄̄(t) corresponds to the calculated prediction reward because it is calculated from the reward running averages. μ corresponds to the parameter for estimating the search amount in the setting because μ is a defined learning-rate parameter used in the meta-learning equation ot = G(μ(r̄(t) − r̄̄(t))) to calculate/update ot when the meta-learning setting is enabled.).
Regarding claim 3, the combination of Macglashan and Khamassi teaches all the limitations of the base claims as outlined above.
Macglashan further teaches wherein the behavior determined is probabilistically determined according to a probability density function using a random variable and an average and a variance of a normal distribution represented by the measure (Par. [0040], “information about agent’s action(s) may include, without limitation, a probability distribution over agent’s action(s) and/or outgoing information meant to influence the agent’s ultimate choice of action”; Par. [0048], “It also requires the actor to define a parametric stochastic policy. That means the output of the actor, π(s), for a given observation(s) of the environment state, are the parameters of probability distributions over the domain of the action space.… for continuous n-dimensional action spaces, the output parameters are often the mean and covariance of a multivariate Gaussian distribution over the action space”; Par. [0079], “Assuming the probability distribution of the actor may be sampled by re-parameterizing the actor into a deterministic function f of the state and some externally sampled noise (E)”; Par. [0080], “Many parametric continuous distributions allow for the actor to be re-parameterized to use externally sampled noise, including the common Gaussian distribution” – action corresponds to the claimed behavior. Macglashan teaches that action information includes a probability distribution over actions, and further teaches using a Gaussian distribution for continuous action spaces. Gaussian distribution corresponds to a normal distribution. Thus, the behavior/action is probabilistically determined according to a probability distribution. Externally sampled noise (E) corresponds to the claimed random variable. Actor output π(s) provides the parameters of the probability distribution, including mean and covariance. For a one-dimensional action, the covariance is the variance. For a multi-dimensional action, the covariance provides the variance information for each action dimension.).
Regarding claim 5, Macglashan teaches a reinforcement learning method for performing reinforcement learning for a continuous behavior space (Par. [0043], “the DAC algorithm can optimize reinforcement learning problems with discrete and continuous action spaces” – continuous action space is interpreted as continuous behavior space),
causing a computer to execute processing of (Par. [0025], “The example embodiments described herein can be implemented in an operating environment comprising computer-executable instructions (e.g., software) installed on a computer, in hardware, or in a combination of software and hardware.”):
wherein
predetermined settings for simulation and an agent model are stored (Par. [0009], “method of training a policy model and an action-value model of an agent,”; Par. [0082], “The online algorithm takes the same hyperparameters as the offline algorithm, except instead of a dataset, it receives a reference to an environment with which it can interact.”; Par. [0010], “γ is a discount factor in a domain [0, 1) that defines how valued future rewards are to more immediate rewards”; Par. [0068], “t is a "temperature" hyperparameter that defines how greedy the target distribution is towards the highest scoring Q-value”; Par. [0011], “non-transitory computer-readable storage medium with an executable program stored thereon” – γ and t correspond to predetermined settings because they are defined parameters used by the reinforcement-learning algorithm. Stored executable programs on a storage medium teaches that the reinforcement-learning program/settings are stored),
in the simulation based on the setting in the reinforcement learning, a state in a next trial (Par. [0081], Algorithm 2 Online DAC, “s’, r <- env.execute(a)”; Par. [0082], “it observes the current state; selects and executes an action using the actor; observes the resulting state and reward” - s’ corresponds to the next state after executing action a. Because Macglashan repeats the environment interaction loop, the resulting state s’ corresponds to the state in the next trial.), a reward according to the state (Par. [0082], “observes the resulting state and reward” – r corresponds to the reward associated with the resulting state), and a flag indicating whether simulation execution has ended (Par. [0081], Algorithm 2 Online DAC, “loop … end loop” – Macglashan teaches repeated environment execution through the environment loop that determines whether the simulation should continue or end. It would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether simulation execution has ended.) are acquired with a behavior defined in advance as an input (Par. [0081], Algorithm 2 Online DAC – action a corresponds to the claimed behavior, and the action/behavior is provided to the environment execution as an input to acquire the resulting state and reward.),
the state acquired by the simulation is input to the agent model and a measure is acquired (Par. [0081], Algorithm 2 Online DAC, “s<-env.observe() a~ π(s; φ)” – the observed state s from the environment is input to the policy model π; Claim 4, “an output of the policy model, π(s), for a given observation (s) of an environment state, are parameters of probability distributions over a domain of an action space” – policy model corresponds to the agent model, and the output π(s) corresponds to the measure),
the behavior is calculated based on the measure defined in advance (Par. [0081], Algorithm 2 Online DAC, “a ~ π(s; φ) s’, r <- env.execute(a)”; Par. [0082], “selects and executes an action using the actor; observes the resulting state and reward”; Par. [0083], “an actor may have been previously trained using another learning paradigm (e.g., learning from demonstration), or have been hard coded by some means” – action a corresponds to the behavior, The action a is defined before executing env.execute(a), and the action a is then used as the input to the environment execution to acquire the resulting state s’ and reward r.),
the agent model is further updated according to the setting of the agent model based on the state, the reward, the flag, and the behavior (Par. [0065], Algorithm 1, “DAC_offline_step(data, θ, φ, θ′, φ′, k, c) … sample minibatch (s, a, r, s′) from data … OPTIMIZER_UPDATE…”; Par. [0082], “observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step.” – Macglashan updates the action-value model and policy model using transition data. State s/s’ corresponds to the claimed state, action a corresponds to the claimed behavior, reward r corresponds to the claimed reward, and θ, φ, θ′, φ′, k, and c correspond to settings of the agent model/update process. Macglashan teaches repeated environment execution through a loop, and it would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether the transition is part of a continuing simulation or whether the simulation execution has ended.),
the calculation of the behavior and the update of the agent model are repeated until a predetermined condition according to the flag and the setting is satisfied (Par. [0082], “It then repeats a series of interactions with the environment in which it observes the current state; selects and executes an action using the actor; observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step”; Par. [0065], Algorithm 1 Offline DAC, “for c times do” – Macglashan repeats executing an action using the actor, which corresponds to repeating the calculation of the behavior, and repeats running the offline algorithm step, which corresponds to repeating the update of the agent model. “for c times” teaches a predetermined update condition because the update step is repeated a set number of times according to the parameter c. As discussed above, it would have been obvious to use a flag indicating whether simulation execution has ended to determine whether the repeated online interaction should continue or terminate.).
Macglashan does not explicitly teach calculating the behavior based on a search amount defined in advance;
the search amount is updated based on a prediction reward obtained for the reward and the search amount in a previous trial, and
repeating the update of the search amount until a predetermined condition is satisfied.
However, Khamassi teaches calculating the behavior based on a search amount defined in advance (Page 3, Active exploration algorithm, “continuous … parameters with which action αj is executed at timestep t are selected from a Gaussian exploration function centered on the current values … in state st … where the width ot of the Gaussian is a meta-parameter which will be tuned through meta-learning; Page 4, Algorithm 1, “initialize … β0 and o0” - ot corresponds to the claimed search amount because ot controls the width of the Gaussian exploration used to select continuous action parameters and o0 corresponds to a search amount defined in advance because it is initialized before the repeated learning steps.);
the search amount is updated based on a prediction reward obtained for the reward and the search amount in a previous trial (Page 3, Active exploration algorithm, “We compute short- and long-term reward running averages: where t1 and t2 are two time constants. We then update βt and ot with: …” – Gaussian width ot corresponds to the claimed search amount because ot controls the spread/range of continuous action parameters sampled during exploration. The reward-running averages correspond to the claimed prediction-reward information because they are reward-derived values used to update ot.), and
repeating the update of the search amount until a predetermined condition is satisfied (Page 1, Introduction, “pre-adjusting parameters such as the exploration rate based on the prior determination of the total number of episodes in the experiment.“; Page 4, Algorithm 1, “for t= 0, 1, 2, … do … if meta-learning then update reward running averages … update βt and ot end if” – ot is repeatedly updated in the reinforcement-learning loop until the meta-learning condition is satisfied, where ot corresponds to the search amount. The meta-learning condition corresponds to a setting for performing the search-amount update. The search-amount update is also repeated during the learning loop until the predetermined experiment length/number of episodes is reached.).
Macglashan and Khamassi are analogous art because they are from the same field of endeavor and contain functional similarities. They both relate to reinforcement learning systems for selecting actions in continuous action spaces and improving learning through actor-critic/exploration techniques.
Therefore, at the time of effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the above reinforcement learning device, as taught by Macglashan, and incorporate a meta-learning update of the Gaussian width, which corresponds to the claimed search amount for continuous action exploration, as taught by Khamassi.
One of ordinary skill in the art would have been motivated to improve the exploration/exploitation tradeoff, thereby improving the reinforcement learning model’s ability to select effective continuous action parameters and adapt during learning, as suggested by Khamassi (Page 2, Introduction).
Regarding claim 6, Macglashan teaches a non-transitory computer readable medium storing a program executable by a computer to perform a process for reinforcement learning program processing for performing reinforcement learning for a continuous behavior space, causes a computer to execute processing of (Par. [0011], “non-transitory computer-readable storage medium with an executable program stored thereon, wherein the program instructs one or more processors to perform the following steps”; Par. [0006], “DAC provides support for discrete-action problems and continuous-action problems”):
wherein
predetermined settings for simulation and an agent model are stored (Par. [0009], “method of training a policy model and an action-value model of an agent,”; Par. [0082], “The online algorithm takes the same hyperparameters as the offline algorithm, except instead of a dataset, it receives a reference to an environment with which it can interact.”; Par. [0010], “γ is a discount factor in a domain [0, 1) that defines how valued future rewards are to more immediate rewards”; Par. [0068], “t is a "temperature" hyperparameter that defines how greedy the target distribution is towards the highest scoring Q-value”; Par. [0011], “non-transitory computer-readable storage medium with an executable program stored thereon” – γ and t correspond to predetermined settings because they are defined parameters used by the reinforcement-learning algorithm. Stored executable programs on a storage medium teaches that the reinforcement-learning program/settings are stored),
in the simulation based on the setting in the reinforcement learning, a state in a next trial (Par. [0081], Algorithm 2 Online DAC, “s’, r <- env.execute(a)”; Par. [0082], “it observes the current state; selects and executes an action using the actor; observes the resulting state and reward” - s’ corresponds to the next state after executing action a. Because Macglashan repeats the environment interaction loop, the resulting state s’ corresponds to the state in the next trial.), a reward according to the state (Par. [0082], “observes the resulting state and reward” – r corresponds to the reward associated with the resulting state), and a flag indicating whether simulation execution has ended (Par. [0081], Algorithm 2 Online DAC, “loop … end loop” – Macglashan teaches repeated environment execution through the environment loop that determines whether the simulation should continue or end. It would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether simulation execution has ended.) are acquired with a behavior defined in advance as an input (Par. [0081], Algorithm 2 Online DAC – action a corresponds to the claimed behavior, and the action/behavior is provided to the environment execution as an input to acquire the resulting state and reward.),
the state acquired by the simulation is input to the agent model and a measure is acquired (Par. [0081], Algorithm 2 Online DAC, “s<-env.observe() a~ π(s; φ)” – the observed state s from the environment is input to the policy model π; Claim 4, “an output of the policy model, π(s), for a given observation (s) of an environment state, are parameters of probability distributions over a domain of an action space” – policy model corresponds to the agent model, and the output π(s) corresponds to the measure),
the behavior is calculated based on the measure and a search amount defined in advance (Par. [0081], Algorithm 2 Online DAC, “a ~ π(s; φ)” – action a corresponds to the behavior, and policy output π(s; φ) corresponds to the claimed measure used to calculate the behavior.),
the agent model is further updated according to the setting of the agent model based on the state, the reward, the flag, and the behavior (Par. [0065], Algorithm 1, “DAC_offline_step(data, θ, φ, θ′, φ′, k, c) … sample minibatch (s, a, r, s′) from data … OPTIMIZER_UPDATE…”; Par. [0082], “observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step.” – Macglashan updates the action-value model and policy model using transition data. State s/s’ corresponds to the claimed state, action a corresponds to the claimed behavior, reward r corresponds to the claimed reward, and θ, φ, θ′, φ′, k, and c correspond to settings of the agent model/update process. Macglashan teaches repeated environment execution through a loop, and it would have been obvious to a person of ordinary skill in the art to implement the loop continuation/termination information as a flag indicating whether the transition is part of a continuing simulation or whether the simulation execution has ended.),
the calculation of the behavior and the update of the agent model are repeated until a predetermined condition according to the flag and the setting is satisfied (Par. [0082], “It then repeats a series of interactions with the environment in which it observes the current state; selects and executes an action using the actor; observes the resulting state and reward; adds the transition to its dataset; and then runs the offline algorithm step”; Par. [0065], Algorithm 1 Offline DAC, “for c times do” – Macglashan repeats executing an action using the actor, which corresponds to repeating the calculation of the behavior, and repeats running the offline algorithm step, which corresponds to repeating the update of the agent model. “for c times” teaches a predetermined update condition because the update step is repeated a set number of times according to the parameter c. As discussed above, it would have been obvious to use a flag indicating whether simulation execution has ended to determine whether the repeated online interaction should continue or terminate.).
Macglashan does not explicitly teach calculating the behavior based on a search amount defined in advance;
the search amount is updated based on a prediction reward obtained for the reward and the search amount in a previous trial, and
repeating the update of the search amount until a predetermined condition is satisfied.
However, Khamassi teaches calculating the behavior based on a search amount defined in advance (Page 3, Active exploration algorithm, “continuous … parameters with which action αj is executed at timestep t are selected from a Gaussian exploration function centered on the current values … in state st … where the width ot of the Gaussian is a meta-parameter which will be tuned through meta-learning; Page 4, Algorithm 1, “initialize … β0 and o0” - ot corresponds to the claimed search amount because ot controls the width of the Gaussian exploration used to select continuous action parameters and o0 corresponds to a search amount defined in advance because it is initialized before the repeated learning steps.);
the search amount is updated based on a prediction reward obtained for the reward and the search amount in a previous trial (Page 3, Active exploration algorithm, “We compute short- and long-term reward running averages: where t1 and t2 are two time constants. We then update βt and ot with: …” – Gaussian width ot corresponds to the claimed search amount because ot controls the spread/range of continuous action parameters sampled during exploration. The reward-running averages correspond to the claimed prediction-reward information because they are reward-derived values used to update ot.), and
repeating the update of the search amount until a predetermined condition is satisfied (Page 1, Introduction, “pre-adjusting parameters such as the exploration rate based on the prior determination of the total number of episodes in the experiment.“; Page 4, Algorithm 1, “for t= 0, 1, 2, … do … if meta-learning then update reward running averages … update βt and ot end if” – ot is repeatedly updated in the reinforcement-learning loop until the meta-learning condition is satisfied, where ot corresponds to the search amount. The meta-learning condition corresponds to a setting for performing the search-amount update. The search-amount update is also repeated during the learning loop until the predetermined experiment length/number of episodes is reached.).
Macglashan and Khamassi are analogous art because they are from the same field of endeavor and contain functional similarities. They both relate to reinforcement learning systems for selecting actions in continuous action spaces and improving learning through actor-critic/exploration techniques.
Therefore, at the time of effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the above reinforcement learning device, as taught by Macglashan, and incorporate a meta-learning update of the Gaussian width, which corresponds to the claimed search amount for continuous action exploration, as taught by Khamassi.
One of ordinary skill in the art would have been motivated to improve the exploration/exploitation tradeoff, thereby improving the reinforcement learning model’s ability to select effective continuous action parameters and adapt during learning, as suggested by Khamassi (Page 2, Introduction).
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Macglashan USPGPUB 2020/0302323 A1 (hereinafter Macglashan) in view of Khamassi et al. (Active exploration in parameterized reinforcement learning, 2016) (hereinafter Khamassi), and further in view of Lee et al. US 2021/0191342 A1 (hereinafter Lee).
Regarding claim 4, the combination of Macglashan and Khamassi teaches all the limitations of the base claims as outlined above.
Macglashan and Khamassi do not explicitly teach wherein the behavior is an air conditioning control method.
However, Lee teaches wherein the behavior is an air conditioning control method (Par. [0060], “methods and systems for training a reinforcement learning (RL) model for control of an HVAC system are disclosed. A RL model can be trained using simulated and real experience data to determine a control action for the HVAC system based on the current state of the system.”; Par. [0180], “Controller 610 can generate control signals based on the output of the RL model 606 and send to the HVAC system 612 to alter an operational state”; Par. [0181], “the received instruction may comprise a new setpoint for the system or control instructions for a subcomponent of HVAC system 612”).
Macglashan, Khamassi, and Lee are analogous art because they are from the same field of endeavor and contain functional similarities. They all relate to reinforcement learning systems that select actions based on state information.
Therefore, at the time of effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the above reinforcement learning device, as taught by Macglashan and Khamassi, and incorporate applying the learned behavior selection to HVAC control, as taught by Lee.
One of ordinary skill in the art would have been motivated to “reduce the energy consumption and cost of the building system”, as suggested by Lee (Par. [0002]).
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Gu et al. [US 11,288,568 B2] teaches methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for computing Q values for actions to be performed by an agent interacting with an environment from a continuous action space of actions.
Kimura et al. [US 2019/0272465 A1] teaches a computer-implemented method, computer program product, and system are provided for estimating a reward in reinforcement learning.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PETER XU whose telephone number is (571)272-0792. The examiner can normally be reached Monday-Friday 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mohammad Ali can be reached at (571) 272-4105. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PETER XU/ Examiner, Art Unit 2119
/ZIAUL KARIM/ Primary Examiner, Art Unit 2119
Prosecution Insights