DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2024/08/09. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–11 and 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (an abstract idea) without reciting significantly more.
Regarding independent claims 1 and 20
Step 1 -- whether the claim falls within any statutory category. See MPEP 2106.03.
Independent claim 1 is drawn to a computer-implemented method for training a neural network and therefore falls within the process category. Independent claim 20 is drawn to a system comprising one or more computers and one or more storage devices storing instructions and therefore falls within the machine category. Accordingly, claims 1 and 20 each fall within at least one of the four statutory categories of invention.
Step 2A Prong One -- whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, the claim is directed to a method for training a neural network by repeatedly updating values of model parameters. The following limitations recite an abstract idea:
"sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter"; and
"updating current values of the model parameters to reduce the discrepancy, for at least one observation from the database, between (i) a reward value for the observation and the corresponding action calculated by the neural network based on the current values of the model parameters, and (ii) a target reward value based on a modified reward value for the corresponding successive action calculated by the neural network based on the sampled values of the model parameters."
These limitations recite mathematical concepts under MPEP § 2106.04(a)(2), subsection I, because they describe mathematical relationships and mathematical calculations. Drawing a sampled value for each parameter from a parameterized distribution defined by a provisional value and a knowledge parameter is a mathematical operation; computing a reward value and a target reward value by the neural network from the parameter values is a mathematical calculation; and updating the parameter values to reduce a discrepancy between the two values is the mathematical operation of minimizing a loss by iterative adjustment of parameters (i.e., gradient descent). As recognized in the 2024 Guidance Update on Patent Subject Matter Eligibility (AI Example 47), the iterative adjustment of network parameter values to minimize a loss function constitutes mathematical calculations. The recitation that these operations are performed by a neural network or a computer does not remove the limitations from the mathematical-concepts grouping.
Accordingly, independent claim 1 recites a judicial exception, namely an abstract idea in the form of mathematical concepts. The analysis proceeds to Step 2A Prong Two.
Independent claim 20 is a system claim reciting the same training operations, sampling each model parameter from the recited distribution and updating the parameter values to reduce the recited discrepancy, and therefore recites the same mathematical-concepts abstract idea as claim 1, for the same reasons. See MPEP § 2106.04(a)(2), subsection I.
Step 2A Prong Two -- whether the claim as a whole integrates the recited judicial exception into a practical application, or whether the claim is "directed to" the judicial exception. See MPEP 2106.04(d).
Regarding independent claim 1, the additional elements beyond the abstract idea are "a neural network … based on a plurality of model parameters," "a database of observations … actions … successive observation … and … rewards," "an agent control system for controlling an agent" (recited in the preamble as the field of use), and the recitation that the method is "computer-implemented."
These additional elements do not integrate the recited judicial exception into a practical application. The recited neural network and computer are claimed at a high level of generality and serve merely as tools to perform the abstract sampling and parameter-updating calculations; this amounts to no more than mere instructions to apply the judicial exception using a generic computer component. See MPEP § 2106.05(f). The recited database, and the receipt and storage of observations, actions, successive observations, and rewards therein, constitute data gathering and storage that are incidental to the abstract idea and amount to insignificant extra-solution activity. See MPEP § 2106.05(g). The preamble recitation of "an agent control system for controlling an agent" states an intended field of use and does not impose a meaningful limit, because the body of claim 1 recites only the training calculations and does not require applying the trained neural network to control any agent. See MPEP § 2106.05(h).
The claim does not recite a particular improvement to the functioning of a computer or to any other technology or technical field. See MPEP §§ 2106.04(d)(1) and 2106.05(a). To the extent the asserted advance is an improvement in the sample efficiency of the training procedure, that advance is realized by the recited mathematical algorithm itself, the per-parameter sampling and the discrepancy-reduction computation, and an improvement to the abstract idea (the algorithm) is not an improvement to a computer or other technology and does not integrate the judicial exception into a practical application. See MPEP § 2106.04(d)(1) and the 2024 Guidance Update (an improved mathematical technique for training a neural network, without more, is an improvement to the abstract idea rather than to technology). The claim also does not recite a particular machine integral to the claim (MPEP § 2106.05(b)), a transformation of a particular article (MPEP § 2106.05(c)), or any other meaningful limitation beyond generally applying the abstract idea (MPEP § 2106.05(e)).
Accordingly, the additional elements, individually and in combination, do not integrate the recited judicial exception into a practical application. Independent claim 1 is directed to the abstract idea under Step 2A.
Regarding independent claim 20, the additional elements are "one or more computers" and "one or more storage devices storing instructions" that, when executed, cause the operations to be performed, together with the same neural network and database recited functionally. These elements are generic computer components recited at a high level of generality and merely provide the tools that perform and store the abstract training calculations; they amount to no more than instructions to apply the exception on a generic computer (MPEP § 2106.05(f)) and generally link the abstract idea to a computer environment (MPEP § 2106.05(h)). Like claim 1, claim 20 recites no application of the trained neural network to control an agent and no improvement to the functioning of a computer or other technology. See MPEP §§ 2106.04(d)(1) and 2106.05(a). Independent claim 20 is therefore directed to the abstract idea under Step 2A.
Step 2B -- whether the claim amounts to significantly more than the judicial exception. See MPEP § 2106.05.
Regarding independent claim 1, the additional elements, the generic neural network, the computer implementation, and the database receiving and storing the observations, actions, successive observations, and rewards, do not amount to significantly more than the judicial exception. Implementing the abstract calculations on a generic neural network and computer is well-understood, routine, and conventional activity and amounts to mere instructions to apply the exception (MPEP §§ 2106.05(d), 2106.05(f)), and the receipt and storage of data in a database is well-understood, routine, and conventional data-gathering and data-storage activity (MPEP §§ 2106.05(d), 2106.05(g)). Considered individually and in combination, these elements provide no inventive concept.
Regarding independent claim 20, the one or more computers and one or more storage devices are generic computer components performing generic functions and storing instructions, which is well-understood, routine, and conventional (MPEP §§ 2106.05(d), 2106.05(f)). Considered individually and in combination, the additional elements of claim 20 provide no inventive concept.
Accordingly, claims 1 and 20 do not recite additional elements that amount to significantly more than the recited judicial exception and are ineligible under 35 U.S.C. 101.
Regarding dependent claims 2–11
Step 1. Dependent claims 2–11 depend from independent claim 1 and are drawn to method claims; each therefore falls within the process category. See MPEP 2106.03.
Step 2A Prong One. Claims 2–11 include the mathematical-concepts abstract idea of claim 1 and further narrow it with additional mathematical relationships and calculations: setting the provisional value equal to the current value and periodically resynchronizing it (claims 2–3); updating the knowledge parameter toward a gradient-based sensitivity value (claim 4); defining that sensitivity as an increasing function of the gradient magnitude and as proportional to the square of the gradient component (claims 5–6); including an additive term selected from a distribution in the target (claim 7); selecting the sampled value from a distribution centered on the provisional value with a standard deviation inversely related to the knowledge parameter (claim 8); and defining the target as a sum of the reward and a future reward term, including a maximum over actions, computed on the sampled parameters (claims 9–10). These limitations recite mathematical concepts under MPEP § 2106.04(a)(2), subsection I. Claim 11 further defines the neural network as a sequence of layers of neural units applying a non-linear function having a non-zero derivative; this limitation characterizes the generic computational tool that performs the abstract calculations and does not remove the claim from the abstract idea.
Step 2A Prong Two. The additional limitations of claims 2–11 do not integrate the judicial exception into a practical application. They merely refine the mathematical training algorithm or further specify the generic neural network used to perform it; none requires applying the trained neural network to control an agent, effects a particular transformation, or recites an improvement to the functioning of a computer or other technology. See MPEP §§ 2106.04(d)(1), 2106.05(a), 2106.05(b), 2106.05(c), 2106.05(e). As to claim 11, specifying a multi-layer architecture with a non-zero-derivative activation function amounts to using a generic neural network as a tool to perform the abstract idea and does not, by itself, improve the functioning of a computer. See MPEP § 2106.05(f) and the 2024 Guidance Update (reciting a neural network as a tool, without a specific technological application or improvement, does not integrate the exception).
Step 2B. For the reasons given for claim 1, the additional elements of claims 2–11, considered individually and in combination, amount to no more than generic computer implementation of the abstract idea and well-understood, routine, conventional activity, and provide no inventive concept. See MPEP §§ 2106.05(d), 2106.05(f).
Accordingly, dependent claims 2–11 are rejected under 35 U.S.C. 101.
Claims 12–19 -- eligible (integration into a practical application)
Claims 12–19 recite the same mathematical-concepts abstract idea identified above but are not rejected under 35 U.S.C. 101, because they integrate the judicial exception into a practical application under Step 2A Prong Two.
Claim 12 (and claim 13, which depends therefrom) recites "transmitting control data to the agent to cause the agent to perform the action." Independent claim 14 likewise recites "transmitting control data to the agent to cause the agent to perform the action," following selection of the action from a plurality of actions based on the reward values output by the trained neural network. Claims 15–17 further require that the environment is a real-world environment observed by sensors and that the agent is an electromechanical agent or real-world apparatus whose control parameters are varied by the actions (claims 15–16), or that the network trained in a simulated environment is deployed to control a real-world apparatus (claim 17). Claims 18–19 require that the agent is a user of a digital assistant that instructs the user to perform the actions and captures observations of the user to determine task success.
Unlike claims 1–11 and 20, which recite only the training calculations, claims 12–19 apply the result of the judicial exception by using the trained neural network to select and transmit control data that causes an agent to perform an action to accomplish a task in an environment. This affirmative control action is a meaningful limitation that applies the exception in a particular and concrete manner, and, in claims 15–17, applies it with a particular real-world apparatus. These additional elements integrate the recited judicial exception into a practical application under MPEP §§ 2106.04(d)(1) and 2106.05(a), and, as to claims 15–17, under MPEP §§ 2106.05(b)–(c). Because claims 12–19 are not directed to the judicial exception under Step 2A, no Step 2B analysis is necessary, and these claims are eligible under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1–4, 7, 9–10, 12–17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Hafner et al. (Hafner), US 2021/0201156 A1, "Sample-Efficient Reinforcement Learning," in view of Fortunato et al. (Fortunato), Non-Patent Literature, "Noisy Networks for Exploration," published on July 9, 2019, and replied upon at pages 1-5, and further in view of Azizzadenesheli et al. (Azizzadenesheli), Non-Patent Literature, "Efficient Exploration through Bayesian Deep Q-Networks," published on January 24, 2019, and relied upon at pages 1-4.
Regarding independent Claim 1, Hafner teaches a computer-implemented method for training a neural network of an agent control system, including:
a computer-implemented method for training a neural network of an agent control system for controlling an agent to perform a task in an environment, the neural network being configured to receive an input characterizing the state of the environment and to generate an output, the control system being configured to select an action for the agent to perform based on the output of the neural network (Hafner, Figure 1, paragraphs [0019] and [0033]: a reinforcement learning system (a neural network) that controls an agent interacting with an environment by processing data (input) characterizing the current state of the environment to select an action to be performed by the agent and generate an action selection output 122);
the method employing a database of observations of the state of the environment, corresponding actions performed by the agent when the environment was in that state, corresponding successive observation of corresponding successive states of the environment, and corresponding rewards associated with the actions and indicative of the contribution of the action to performing the task (Hafner, paragraphs [0025]-[0026], [0029]: the observations may include data from one or more sensors monitoring part of a plant or service facility such as current, voltage, power, temperature, and other sensors and/or electronic signals representing the functioning of electronic and/or mechanical items of equipment. and the agent may control actions in a real-world environment including items of equipment, for example, in a data center (a database); paragraphs [0042]-[0043]: during training, transitions generated as a result of the agent interacting with the environment, wherein each transition includes an initial observation, the action performed by the agent in response to the initial observation, a reward, and a next observation characterizing the next state of the environment, and wherein rewards measure progress of the agent in completing a task);
the neural network being based on a plurality of model parameters of the neural network and is configured, upon receiving an observation and an action, to generate a reward value (Hafner, ¶ [0006] and ¶¶ [0037]–[0038]: each Q network receives “(i) an input observation … and (ii) data identifying an action … to generate a Q value for the input observation-action pair,” which “is an estimate of a ‘return’”);
repeatedly updating current values of a plurality of the model parameters (Hafner, ¶[0041]: “[t]he training engine 116 is configured to train the policy neural network 110 by repeatedly updating the model parameters 118”); and
updating current values of the model parameters to reduce the discrepancy, for at least one observation from the database, between (i) a reward value for the observation and the corresponding action calculated by the neural network based on the current values of the model parameters, and (ii) a target reward value (Hafner, ¶ [0077], FIG. 3, step 312: training proceeds on “a square of the difference, between the Q value and the final target Q value”).
Hafner teaches generating its target with parameter sets other than the current ones — maintaining an ensemble of Q networks and contemplating that an ensemble "can instead be replaced with a single Bayesian neural network" that is "sampled multiple times" (Hafner, ¶ [0098]) — and thus teaches something related to sampling model parameters from a distribution. Hafner does not, however, teach "sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter."
In the same field of endeavor, Fortunato teaches sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter (Fortunato, p. 4, § 3: "θ = µ + Σ⊙ε, where ζ = (µ, Σ) is a set of vectors of learnable parameters"). Here each network parameter θ is drawn from a distribution centered on the provisional value µ, with the per-parameter spread set by the learnable quantity Σ , the recited respective knowledge parameter , and a fresh set of parameters is drawn for use during learning (Fortunato, p. 5, § 3.1: the agent "samples a new set of parameters after every step of optimisation"; p. 2: "the variance of the perturbation … learned using gradients from the reinforcement learning loss function").
Hafner and Fortunato are analogous art, both being from the same field of endeavor of exploration in deep reinforcement learning by introducing uncertainty into the parameters of a value/Q network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to realize the parameter sampling that Hafner already contemplates (its single sampled Bayesian network) using the per-parameter learnable distribution θ = µ + Σ⊙ε of Fortunato. The motivation to combine Hafner and Fortunato is as recited by Fortunato (Abstract, p. 1): per-weight learned perturbations "aid efficient exploration" while remaining "straightforward to implement and adding little computational overhead," predictably improving the sample-efficient training of Hafner.
The combination of Hafner and Fortunato does not expressly teach that the target reward value is based on a modified reward value for the corresponding successive action calculated by the neural network based on the sampled values of the model parameters.
In the same field of endeavor, Azizzadenesheli teaches this feature (Azizzadenesheli, p. 1, Abstract: a "Thompson sampling approach" that performs exploration "through Gaussian sampling"). As set out in Algorithm 1 (p. 4, § 3), the system first draws the sampled parameters , right col., line 7, "Draw w_a ∼ N(w_a^{target}, Cov_a)" , and then computes the value for the successive action with those sampled parameters , line 15, "â := arg max_{a'} w_{a'}^⊤ φ_θ(x_{τ+1})," and line 16, "y_τ ← r_τ + w_{â}^{target⊤} φ_{θtarget}(x_{τ+1})" , before reducing the discrepancy at line 17, "Update θ ← θ − α·∇θ(y_τ − w{a_τ}^⊤ φ_θ(x_τ))²." In other words, the modified reward value for the successive action that enters the target is, as claimed, calculated by the network on sampled parameter values drawn from the posterior (Azizzadenesheli, p. 4, col. 2: "A sample of Q(x,a) is w_a^⊤ φ_θ(x) where w_a is drawn from the posterior distribution").
Hafner, Fortunato, and Azizzadenesheli are analogous art, all being from the same field of endeavor of exploration in deep reinforcement learning via parameter-space uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to compute the bootstrapped target of Hafner for the successive action using the per-parameter sampled values of Fortunato, in the manner taught by Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is as recited by Azizzadenesheli (p. 3 and Abstract, p. 1): evaluating the target with sampled parameters uses both the estimated Q values "and its uncertainty estimates to carry out a more efficient exploration," whereby the agent reaches "higher rewards substantially faster."
Regarding Claim 2, Claims 2 depends from Claim 1. Every limitation carried forward from Claim 1 is rejected under the same rationale set forth for Claim 1 above. The following addresses only the limitations newly introduced in Claims 2:
in which the provisional value of each model parameter is the current value of the model parameter (Hafner, ¶¶ [0079]–[0080]).
In the base case taught by Hafner, the parameters from which the network output is generated are simply the present parameters of the Q network, the system being able to "use the current values when updating the policy neural network" (Hafner, ¶ [0080]). Because the provisional value from which the parameters are drawn is, in that case, the same as the value currently maintained for the parameter, Hafner teaches that the provisional value of each model parameter is the current value of the model parameter, as recited.
Regarding Claim 3, Claims 3 depends from Claim 1. Every limitation carried forward from Claim 1 is rejected under the same rationale set forth for Claim 1 above. The following addresses only the limitations newly introduced in Claims 3:
further comprising, following a plurality of said updates to the current values of the model parameters, a step of updating the provisional value of each model parameter to be the current value of the model parameter (Hafner, ¶¶ [0079]–[0080]).
Hafner teaches that the system "maintains current values and older values for the parameters" of the Q network and uses "the current values when updating the policy neural network and use[s] the older values" in generating the target, the older values being periodically set to the current values during training (Hafner, ¶¶ [0079]–[0080]). The older values are the provisional values from which the target is generated; periodically setting those older values equal to the current values, after a number of updates have been made, reads on the recited step.
Regarding Claim 4, Claim 4 depends from Claim 1 and is rejected for the reasons given for Claim 1, further in view of the following:
updating the respective knowledge parameter for each parameter to bring it closer to a corresponding sensitivity value indicative of the sensitivity to the corresponding model parameter of the discrepancy between the reward value and a value based on the target reward value (Fortunato, p. 2; p. 4, § 3; p. 5, Eqs. (12)).
Hafner teaches "the discrepancy between the reward value and a value based on the target reward value," reducing the squared difference between the Q value computed on the current parameter values and the target Q value during training (Hafner, ¶ [0077], FIG. 3, step 312), the recited "reward value" corresponding to Hafner's Q value computed on the current parameters and the recited "value based on the target reward value" corresponding to Hafner's target Q value. Hafner does not, however, teach "updating the respective knowledge parameter for each parameter to bring it closer to a corresponding sensitivity value indicative of the sensitivity to the corresponding model parameter of" that discrepancy, because Hafner maintains no respective per-parameter knowledge parameter to update.
In the same field of endeavor, Fortunato teaches "updating the respective knowledge parameter for each parameter to bring it closer to a corresponding sensitivity value indicative of the sensitivity to the corresponding model parameter of the discrepancy between the reward value and a value based on the target reward value." The per-parameter quantity Σ , the knowledge parameter mapped in Claim 1 , is trained jointly with the network weights by gradient descent on the reinforcement-learning loss: "θ = µ + Σ⊙ε, where ζ = (µ, Σ) is a set of vectors of learnable parameters" (Fortunato, p. 4, § 3), with "the variance of the perturbation … learned using gradients from the reinforcement learning loss function" (Fortunato, p. 2). The gradient taken with respect to that knowledge parameter, ∇L̄(ζ) = E[∇_{µ,Σ} L(µ + Σ⊙ε)] (Fortunato, p. 5, Eqs. (12)–(13)), is the sensitivity, to the corresponding model parameter, of the discrepancy taught by Hafner; moving each knowledge parameter Σ along that gradient brings it closer to the corresponding sensitivity value, as recited.
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning by introducing a learned, per-parameter measure of uncertainty into a value/Q neural network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to adjust the per-parameter knowledge parameter of the combination by gradient descent on the training loss , that is, toward the sensitivity of the discrepancy taught by Hafner , in the manner taught by Fortunato. The motivation to combine Hafner and Fortunato is as recited by Fortunato (Abstract, p. 1; p. 2): learning the per-parameter noise from "gradients from the reinforcement learning loss function" drives each parameter's exploration width to track its own sensitivity, which "aids efficient exploration" while "adding little computational overhead," predictably improving the sample-efficient training of Hafner. The further combination with Azizzadenesheli is retained for the reasons set forth for Claim 1.
Regarding Claim 7, Claim 7 depends from Claim 4. Every limitation carried forward from Claims 1 and 4 is rejected under the same rationale set forth for those claims above. The following addresses only the limitation newly introduced in Claim 7:
in which the value based on the target reward value further includes a term selected from a distribution (Azizzadenesheli, p. 4, § 3).
The combination of Hafner and Fortunato teaches the target reward value and the discrepancy reduced during training (Claim 1), but does not expressly teach that the value based on the target reward value further includes a term selected from a distribution.
In the same field of endeavor, Azizzadenesheli teaches that the target value used to train the network is formed with an additive stochastic term drawn from a distribution (Azizzadenesheli, p. 4, § 3: "the target value y ∼ w_a^⊤ φ_θ(x) + ε … where ε ∼ N(0, σ_ε²) is an i.i.d. Gaussian noise"). The "value based on the target reward value" of the claim corresponds to Azizzadenesheli's target value y, and the additive Gaussian term ε ∼ N(0, σ_ε²) included in that value reads on the recited term selected from a distribution.
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning by introducing parameter- and value-level uncertainty into a value/Q neural network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to include in the value based on the target reward value of the combination an additive term selected from a distribution, in the manner taught by Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is the same as set forth for Claim 1, and is as recited by Azizzadenesheli (Abstract, p. 1; p. 3): incorporating the posterior-distributed term in the target effects Thompson sampling that "uses both estimated Q function and its uncertainty estimates to carry out a more efficient exploration," whereby the agent predictably reaches "higher rewards substantially faster."
Regarding Claims 9, Claim 9 depends from Claim 1 and is rejected for the reasons given for Claim 1, further in view of the following:
in which the target reward value includes a term which is a sum of the reward associated with the action and a future reward term proportional to an output of the neural network for the sampled values of the model parameters and the subsequent observation (Hafner, ¶¶ [0096]–[0097], FIG. 5, step 506; Azizzadenesheli, p. 4, Algorithm 1, line 16).
Hafner teaches forming the target as the training reward summed with a discounted future term given by the network output on a sampled parameter set and the subsequent observation: the system generates "L Q values for the time step (step 504)" and "L × N candidate target Q values for the time step (step 506)," each candidate target being the training reward summed with discounted future Q-values computed by ensemble members across the trajectory (Hafner, ¶¶ [0096]–[0097], FIG. 5). Hafner computes the future term using an ensemble member rather than expressly using "the sampled values of the model parameters" recited in Claim 1; to that extent, Hafner does not teach the future reward term being "an output of the neural network for the sampled values of the model parameters."
In the same field of endeavor, Azizzadenesheli teaches forming this same sum, with the future reward term computed for the sampled values of the model parameters and the subsequent observation: "y_τ ← r_τ + w_{â}^{target⊤} φ_{θtarget}(x_{τ+1})" (Azizzadenesheli, p. 4, Algorithm 1, line 16), where r_τ is the reward and the future term w_{â}^{target⊤} φ_{θtarget}(x_{τ+1}) is the network output for the sampled weights w and the subsequent observation x_{τ+1}, the weights being drawn from the posterior (Azizzadenesheli, p. 4, col. 2: "A sample of Q(x,a) is w_a^⊤ φ_θ(x) where w_a is drawn from the posterior distribution").
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning by computing bootstrapped value targets on parameters carrying uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to form the target reward value of the combination as the sum of the reward and a future reward term given by the network output for the sampled parameter values of Fortunato and the subsequent observation, in the manner taught by Hafner and Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is the same as set forth for Claim 1, and is as recited by Azizzadenesheli (Abstract, p. 1; p. 3): bootstrapping the target from a future term computed on the sampled parameters effects Thompson-sampling exploration that "uses both estimated Q function and its uncertainty estimates to carry out a more efficient exploration," whereby the agent predictably reaches "higher rewards substantially faster."
Regarding Claim 10, Claim 10 depends from Claim 9 and is rejected for the reasons given for Claims 1 and 9, further in view of the following:
in which the future reward term is proportional to the maximum, over a set of possible actions, of the respective reward value output by the neural network for the sampled values of the model parameters, upon receiving the subsequent observation and the possible action (Azizzadenesheli, p. 4, Algorithm 1, line 15).
Hafner teaches selecting the action having the highest Q value when generating its target (Hafner, ¶ [0088]) and so teaches a maximum-valued future term generally, but the combination of Hafner and Fortunato does not expressly teach that the future reward term is "the maximum, over a set of possible actions, of the respective reward value output by the neural network for the sampled values of the model parameters, upon receiving the subsequent observation and the possible action."
In the same field of endeavor, Azizzadenesheli teaches this feature. Having drawn the sampled parameter values, Azizzadenesheli forms the future term by maximizing the network output over the set of possible actions, evaluated on those sampled values and the subsequent observation: "â := arg max_{a'} w_{a'}^⊤ φ_θ(x_{τ+1})" (Azizzadenesheli, p. 4, Algorithm 1, line 15). The resulting w_{â}^⊤ φ is the maximum, over the possible actions a′, of the respective reward value output by the network for the sampled weights w upon receiving the subsequent observation x_{τ+1} and the possible action, reading on the recited future reward term.
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning by computing bootstrapped value targets on parameters carrying uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to form the future reward term of the combination as the maximum, over the possible actions, of the network output evaluated on the sampled parameter values and the subsequent observation, in the manner taught by Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is the same as set forth for Claims 1 and 9, and is as recited by Azizzadenesheli (Abstract, p. 1; p. 3): bootstrapping the target from the maximum-action value computed on the sampled parameters effects Thompson-sampling exploration that "uses both estimated Q function and its uncertainty estimates to carry out a more efficient exploration," whereby the agent predictably reaches "higher rewards substantially faster."
Regarding Claims 12, Claim 12 depends from Claim 1 and is rejected for the reasons given for Claim 1, further in view of the following:
further comprising supplementing the database by, in each of a plurality of iterations: receiving a current observation of the environment, selecting a corresponding action for the agent to perform using the neural network, transmitting control data to the agent to cause the agent to perform the action, receiving a corresponding subsequent observation of the environment and a corresponding reward, and adding the current observation, the corresponding action, the corresponding subsequent observation and the corresponding reward to the database (Hafner, ¶¶ [0027], [0042], [0048], FIG. 1).
Hafner teaches "supplementing the database by, in each of a plurality of iterations" the recited acting steps: transitions generated as the agent interacts with the environment are "stored in a transition buffer 114," each transition comprising "an initial observation, the action performed by the agent …, a reward, and a [next] observation" (Hafner, ¶ [0042]). Hafner teaches receiving the current observation and selecting the action using the policy/Q neural network (action selection output 122; Hafner, ¶¶ [0033]–[0034]), "transmitting control data to the agent to cause the agent to perform the action" as action output that "directly define[s] the action … by defining the values of torques" applied to the agent (Hafner, ¶¶ [0027], [0048]), receiving the subsequent observation and reward, and adding all four to the transition buffer (Hafner, ¶ [0042]).
Regarding Claim 13, Claim 13 depends from Claim 12 and is rejected for the reasons given for Claims 1 and 12, further in view of the following:
further comprising obtaining a second sampled value for each of the model parameters from a distribution based on the provisional value of the model parameter and the respective knowledge parameter, and, in each of said plurality of iterations, said selecting the action for the agent to perform comprises: for each of a plurality of actions, obtaining a corresponding second modified reward value calculated by the neural network based on the current observation, the second sampled values of the model parameters and the corresponding one of the plurality of actions; and selecting the action for the agent to perform from the plurality of actions based on the corresponding second modified reward values (Fortunato, p. 5, § 3.1; Azizzadenesheli, p. 4, § 3 and Algorithm 1, lines 9 and 15).
Hafner teaches selecting the action for the agent to perform using the neural network during the acting loop (as set forth for Claim 12), but does not teach "obtaining a second sampled value for each of the model parameters from a distribution based on the provisional value of the model parameter and the respective knowledge parameter," because Hafner maintains no respective per-parameter knowledge parameter and does not draw a per-parameter sample for action selection.
In the same field of endeavor, Fortunato teaches "obtaining a second sampled value for each of the model parameters from a distribution based on the provisional value of the model parameter and the respective knowledge parameter," drawing a fresh sample of the parameters from the distribution centered on the provisional value µ with the per-parameter knowledge parameter Σ (θ = µ + Σ⊙ε) for use in acting, the parameters being "re-sampled before every action" (Fortunato, p. 5, § 3.1).
The combination of Hafner and Fortunato does not teach that "said selecting the action for the agent to perform comprises: for each of a plurality of actions, obtaining a corresponding second modified reward value calculated by the neural network based on the current observation, the second sampled values of the model parameters and the corresponding one of the plurality of actions; and selecting the action for the agent to perform from the plurality of actions based on the corresponding second modified reward values." In the same field of endeavor, Azizzadenesheli teaches this feature: having drawn the sampled parameters, the agent computes, for each candidate action, the network output on those sampled values and selects the action maximizing it , "we draw a new w_a … and follow the resulting policy, i.e., a_TS := max_a w_a^⊤ φ_θ(x)" (Azizzadenesheli, p. 4, § 3), as implemented at Algorithm 1, lines 9 and 15 ("Execute a_t = arg max_{a'} w_{a'}^⊤ φ_θ(x_t)"). The output w_{a'}^⊤ φ_θ(x) for each action is the recited second modified reward value calculated on the second sampled values of the model parameters, the current observation, and the corresponding action, and the arg-max selection is the recited selecting of the action based on the corresponding second modified reward values.
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning in which an agent selects its actions under parameters carrying uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to select the actions of Hafner's acting loop using a second, freshly sampled set of the per-parameter-distributed parameters of Fortunato and the sampled-parameter greedy selection of Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is the same as set forth for Claims 1 and 12: per Fortunato (p. 5, § 3.1; Abstract), re-sampling the parameters before each action yields a consistent, state-dependent exploratory policy that "aids efficient exploration," and per Azizzadenesheli (Abstract, p. 1; p. 3), selecting the action under the freshly sampled parameters effects Thompson sampling that carries out "more efficient exploration," whereby the agent predictably reaches "higher rewards substantially faster."
Regarding independent Claim 14, Hafner teaches a computer-implemented method of controlling an agent to perform a task in an environment, including:
a computer-implemented method of controlling an agent to perform a task in an environment, the method comprising repeatedly: receiving an observation of a state of the environment (Hafner, ¶¶ [0030]–[0033], FIG. 1: the system 100 "controls an agent 102 interacting with an environment 104 by selecting actions 106," receiving the observation 120 characterizing the current state);
for each of a plurality of actions, obtaining a corresponding reward value calculated based on the observation by a neural network (Hafner, ¶¶ [0036]–[0037]: the action selection output 122 may include a respective Q-value for each action that can be performed by the agent);
selecting an action for the agent to perform from the plurality of actions based on the corresponding reward values (Hafner, ¶ [0088]: "the system can select the action that has the highest Q value");
transmitting control data to the agent to cause the agent to perform the action (Hafner, ¶¶ [0027], [0048]: the action output "directly define[s] the action … by defining the values of torques that should be applied to the joints of a robotic agent");
wherein the neural network has a plurality of model parameters and is configured, upon receiving an observation and an action, to generate a reward value (Hafner, ¶ [0006] and ¶¶ [0037]–[0038]: each Q network generates "a Q value for the input observation-action pair," which "is an estimate of a 'return'");
wherein the neural network has been trained using a training process employing a database of observations of the state of the environment, corresponding actions performed by the agent when the environment was in that state, corresponding successive observation of corresponding successive states of the environment, and corresponding rewards associated with the actions and indicative of the contribution of the action to performing the task (Hafner, ¶ [0042]: each transition "includes an initial observation, the action performed …, a reward, and a [next] observation");
the training process including repeatedly updating current values of a plurality of the model parameters (Hafner, ¶ [0041]: the training engine 116 trains "by repeatedly updating the model parameters 118"); and
updating current values of the model parameters to reduce the discrepancy, for at least one observation from the database, between (i) a reward value for the observation and the corresponding action calculated by the neural network based on the current values of the model parameters, and (ii) a target reward value (Hafner, ¶ [0077], FIG. 3, step 312: "a square of the difference, between the Q value and the final target Q value").
The recited control limitations — receiving an observation, obtaining a per-action reward value via the neural network, selecting the highest-valued action, and transmitting it as control data — correspond, respectively, to Hafner's receipt of the observation, the per-action Q values of output 122, the selection of the highest-Q action, and the transmission of that action (e.g., joint torques) to the agent. The recited training limitations are identical to the corresponding limitations of Claim 1 and are rejected based on the same rationale applied to Claim 1.
Hafner teaches generating its target with parameter sets other than the current ones , maintaining an ensemble of Q networks and "older values" of the parameters, and even contemplating that an ensemble "can instead be replaced with a single Bayesian neural network" that is "sampled multiple times" (Hafner, ¶ [0098]), and thus teaches something related to sampling model parameters from a distribution. Hafner does not, however, teach "sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter."
In the same field of endeavor, Fortunato teaches "sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter" (Fortunato, p. 4, § 3: "θ = µ + Σ⊙ε, where ζ = (µ, Σ) is a set of vectors of learnable parameters"), each parameter θ being drawn from a distribution centered on the provisional value µ with the per-parameter spread set by the learnable knowledge parameter Σ (Fortunato, p. 5, § 3.1: the agent "samples a new set of parameters after every step of optimisation").
Hafner and Fortunato are analogous to the claimed invention as both are from the same field of endeavor of exploration in deep reinforcement learning by introducing uncertainty into the parameters of a value/Q neural network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to realize the parameter sampling that Hafner already contemplates using the per-parameter learnable distribution θ = µ + Σ⊙ε of Fortunato. The motivation to combine Hafner and Fortunato is as recited by Fortunato (Abstract, p. 1): per-weight learned perturbations "aid efficient exploration" while remaining "straightforward to implement and adding little computational overhead," predictably improving the sample-efficient training of Hafner.
The combination of Hafner and Fortunato, however, does not teach "a target reward value based on a modified reward value for the corresponding successive action calculated by the neural network based on the sampled values of the model parameters."
In the same field of endeavor, Azizzadenesheli teaches this feature (Azizzadenesheli, p. 1, Abstract: a "Thompson sampling approach" performing exploration "through Gaussian sampling"). As set out in Algorithm 1 (p. 4, § 3), the system draws the sampled parameters , line 7, "Draw w_a ∼ N(w_a^{target}, Cov_a)" , and computes the value for the successive action with those sampled parameters , line 15, "â := arg max_{a'} w_{a'}^⊤ φ_θ(x_{τ+1})," and line 16, "y_τ ← r_τ + w_{â}^{target⊤} φ_{θtarget}(x_{τ+1})" , before reducing the discrepancy at line 17. The modified reward value for the successive action that enters the target is thus calculated by the network on sampled parameter values drawn from the posterior (Azizzadenesheli, p. 4, col. 2: "A sample of Q(x,a) is w_a^⊤ φ_θ(x) where w_a is drawn from the posterior distribution").
Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning via parameter-space uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to compute the bootstrapped target of Hafner for the successive action using the per-parameter sampled values of Fortunato, in the manner taught by Azizzadenesheli. The motivation to combine Hafner, Fortunato, and Azizzadenesheli is as recited by Azizzadenesheli (p. 3 and Abstract, p. 1): evaluating the target with sampled parameters uses both the estimated Q values "and its uncertainty estimates to carry out a more efficient exploration," whereby the agent reaches "higher rewards substantially faster."
Regarding Claims 15, claim 15 depends from Claim 14 and is rejected for the reasons given for Claim 14, further in view of the following:
in which the environment is a real-world environment, the observations are collected by sensors arranged to observe the real-world environment, and the agent is an electromechanical agent arranged to move in the environment (Hafner, ¶¶ [0023]–[0026]).
Hafner teaches that "the environment is a real-world environment," that "the observations are collected by sensors arranged to observe the real-world environment," and that "the agent is an electromechanical agent arranged to move in the environment." Specifically, Hafner teaches a "robot or other mechanical agent" and an "autonomous or semi-autonomous land[,] air[,] or sea vehicle," with observations that include sensor data such as "joint position, joint velocity," "image or video data," and other sensor signals (Hafner, ¶¶ [0023]–[0026]). The recited real-world environment, sensors, and electromechanical agent correspond, respectively, to Hafner's real-world environment, its image/position/velocity sensors collecting the observations, and its robot or vehicle moving through the environment.
Regarding Claim 16, claim 16 depends from Claim 14 and is rejected for the reasons given for Claim 14, further in view of the following:
in which the agent is a real-world apparatus, the agent control system being a component of the apparatus and the actions varying control parameters of the apparatus (Hafner, ¶¶ [0027]–[0029]).
Hafner teaches that "the agent is a real-world apparatus," that "the agent control system [is] a component of the apparatus," and that "the actions vary[] control parameters of the apparatus." Specifically, the agent is a real-world robot or vehicle; the actions "may be control inputs to control the robot, e.g., torques for the joints of the robot," or, for a vehicle, "actions to control navigation, e.g., steering, and movement, e.g., braking and/or acceleration of the vehicle" (Hafner, ¶¶ [0027]–[0029]); and the policy neural network generating those control inputs serves as the control component. The recited real-world apparatus, control-system component, and control-parameter-varying actions correspond, respectively, to Hafner's robot/vehicle, its policy neural network, and its torque/steering/braking control inputs.
Regarding Claim 17, Claim 17 depends on Claim 14. Every limitation carried forward from Claim 14 is rejected under the same rationale set forth for Claim 14 above. The following addresses only the limitation newly introduced in Claim 17:
in which the neural network was trained using a simulated agent in a simulated environment, and said control data is transmitted to an agent, which is a real-world apparatus in a real-world environment (Hafner, ¶¶ [0023]–[0026], [0105]–[0106]).
Hafner teaches both legs of this limitation. As to training with a simulated agent in a simulated environment, Hafner teaches that "[t]he simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation," with the agent a simulated vehicle (Hafner, ¶¶ [0105]–[0106]). As to transmitting the resulting control data to a real-world apparatus, Hafner teaches controlling a real-world robot or vehicle by transmitting actions such as torques and navigation controls (Hafner, ¶¶ [0023]–[0027]). It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to train the network of the combination in Hafner's disclosed simulated environment and then transmit the resulting control data to Hafner's disclosed real-world apparatus. The motivation is Hafner's stated advantage (¶ [0005]) of minimizing the number of actual real-world samples required for training, because "collecting actual samples from the environment adds wear to the agent, increases the chance of mechanical failure of the agent, and is very time-intensive" — training in simulation before real-world deployment predictably realizing that benefit.
Regarding independent Claim 20, Claim 20 is an independent system claim reciting the same training operations as Claim 1, and is rejected for the reasons given for Claim 1, further in view of the following:
a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the recited training operations (Hafner, ¶ [0030] and Hafner claim 14).
Hafner teaches "a system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform" the recited operations: the reinforcement learning system 100 is "an example of a system implemented as computer programs on one or more computers in one or more locations" (Hafner, ¶ [0030]), and Hafner is itself claimed as such a system of one or more computers and storage devices storing instructions. The training operations recited in Claim 20 are identical to the corresponding limitations of Claim 1 and are rejected based on the same rationale applied to Claim 1.
Accordingly, and as established for Claim 1, Hafner does not teach "sampling a sampled value for each of the model parameters from a model parameter distribution based on a provisional value for each model parameter and a respective knowledge parameter for the model parameter."
In the same field of endeavor, Fortunato teaches this limitation, drawing each parameter from a distribution centered on the provisional value µ with the per-parameter knowledge parameter Σ ("θ = µ + Σ⊙ε, where ζ = (µ, Σ) is a set of vectors of learnable parameters"; Fortunato, p. 4, § 3; p. 5, § 3.1). Hafner and Fortunato are analogous to the claimed invention as both are from the same field of endeavor of exploration in deep reinforcement learning by introducing uncertainty into the parameters of a value/Q neural network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to realize the parameter sampling that Hafner contemplates using the per-parameter learnable distribution of Fortunato. The motivation to combine Hafner and Fortunato is as recited by Fortunato (Abstract, p. 1): per-weight learned perturbations "aid efficient exploration" while "adding little computational overhead," predictably improving the sample-efficient training of Hafner.
The combination of Hafner and Fortunato, however, does not teach "a target reward value based on a modified reward value for the corresponding successive action calculated by the neural network based on the sampled values of the model parameters."
In the same field of endeavor, Azizzadenesheli teaches this limitation, drawing the sampled parameters ("Draw w_a ∼ N(w_a^{target}, Cov_a)") and computing the target for the successive action on those sampled parameters ("y_τ ← r_τ + w_{â}^{target⊤} φ_{θtarget}(x_{τ+1})") (Azizzadenesheli, p. 4, Algorithm 1, lines 7 and 16), the weights being "drawn from the posterior distribution" (Azizzadenesheli, p. 4, col. 2). Hafner, Fortunato, and Azizzadenesheli are analogous to the claimed invention as all are from the same field of endeavor of exploration in deep reinforcement learning via parameter-space uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to compute the bootstrapped target of Hafner for the successive action using the per-parameter sampled values of Fortunato, in the manner taught by Azizzadenesheli.
The motivation to combine Hafner, Fortunato, and Azizzadenesheli is as recited by Azizzadenesheli (p. 3 and Abstract, p. 1): evaluating the target with sampled parameters uses both the estimated Q values "and its uncertainty estimates to carry out a more efficient exploration," whereby the agent reaches "higher rewards substantially faster."
Claims 5, 6, and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Hafner, in view of Fortunato, and Azizzadenesheli, and further in view of Zhang et al. (Zhang), Non-Patent Literature, "Noisy Natural Gradient as Variational Inference.", published on February 26, 2018 and cited in the IDS filed on 08/09/2024, and relied upon at pages 2-5.
Regarding Claims 5, claim 5 depends from Claim 4 and is rejected for the reasons given for Claims 1 and 4, further in view of the following:
in which the sensitivity value is an increasing function of the magnitude of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters (Zhang, p. 2; p. 5).
The combination of Hafner, Fortunato, and Azizzadenesheli teaches updating the knowledge parameter toward a sensitivity value given by the gradient of the discrepancy with respect to each parameter (Claim 4, via Fortunato), but does not teach that "the sensitivity value is an increasing function of the magnitude of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters."
In the same field of endeavor, Zhang teaches "the sensitivity value being an increasing function of the magnitude of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters." Zhang sets the per-parameter quantity governing each weight's posterior to the Fisher information, which is the (co)variance of the log-likelihood gradient (Zhang, p. 2: "F = Cov_{x∼p_D, y∼p(y|x,w)}[∇_w log p(y|x,w)]"), and estimates it per parameter as a running average of the squared gradient (Zhang, p. 5: "f ← β₂·f + (1 − β₂)·(∇_w log p(y|x,w))²"). The negative log-likelihood being the discrepancy reduced during training, its gradient with respect to each parameter is the recited gradient component, and the Fisher quantity necessarily increases as the magnitude of that gradient component increases, reading on the recited increasing function.
Hafner, Fortunato, Azizzadenesheli, and Zhang are analogous to the claimed invention as all are from the same field of endeavor of training neural networks with a learned, per-parameter measure of parameter uncertainty derived from the training gradients. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to set the gradient-based sensitivity value of the combination as the Fisher-information quantity of Zhang, which increases with the magnitude of the gradient component. The motivation to combine Hafner, Fortunato, Azizzadenesheli, and Zhang is as recited by Zhang (p. 2–3): the Fisher information is the curvature-based, steepest-descent metric that yields a "better-calibrated" posterior over the parameters, so that each parameter's uncertainty reflects how strongly the loss responds to that parameter, predictably improving the calibration of the knowledge-parameter update of the combination and thereby its exploration efficiency.
Regarding Claim 6, Claim 6 depends from Claim 5 and is rejected for the reasons given for Claims 1, 4, and 5, further in view of the following:
in which the sensitivity value is proportional to the square of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters (Zhang, p. 5).
The combination of Hafner, Fortunato, and Azizzadenesheli teaches updating the knowledge parameter toward a gradient-based sensitivity value (Claim 4), but does not teach that "the sensitivity value is proportional to the square of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters."
In the same field of endeavor, Zhang teaches "the sensitivity value being proportional to the square of the corresponding component of the gradient of the discrepancy with respect to the values of the parameters." In its Noisy-Adam procedure, Zhang maintains the per-parameter Fisher estimate as a running average of the square of the gradient: "f ← β₂·f + (1 − β₂)·(∇_w log p(y|x,w))²" (Zhang, p. 5). The gradient ∇_w log p(y|x,w) of the discrepancy (the negative log-likelihood) with respect to each parameter is the recited gradient component, and the per-parameter sensitivity value f is proportional to the square of that component, reading on the recited limitation.
Hafner, Fortunato, Azizzadenesheli, and Zhang are analogous to the claimed invention as all are from the same field of endeavor of training neural networks with a learned, per-parameter measure of parameter uncertainty derived from the training gradients. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to set the gradient-based sensitivity value of the combination as the squared-gradient (Fisher) quantity of Zhang. The motivation to combine Hafner, Fortunato, Azizzadenesheli, and Zhang is as recited by Zhang (p. 2–3): the squared-gradient Fisher quantity is the curvature-based, steepest-descent metric that yields a "better-calibrated" posterior over the parameters, so that each parameter's uncertainty reflects how strongly the loss responds to that parameter, predictably improving the calibration of the knowledge-parameter update of the combination and thereby its exploration efficiency.
Regarding Claim 8, claim 8 depends from Claim 1 and is rejected for the reasons given for Claim 1, further in view of the following:
in which the sampled value for each model parameter is selected from a distribution centered on the respective provisional value of the model parameters and having a standard deviation which is inversely related to the respective knowledge parameter (Fortunato, p. 4, § 3; Zhang, p. 3 and p. 5).
Hafner teaches sampling model parameters from a distribution generally , maintaining an ensemble of Q networks and contemplating that the ensemble "can instead be replaced with a single Bayesian neural network" that is "sampled multiple times" , but does not teach that "the sampled value for each model parameter is selected from a distribution centered on the respective provisional value of the model parameters."
In the same field of endeavor, Fortunato teaches that "the sampled value for each model parameter is selected from a distribution centered on the respective provisional value of the model parameters," drawing each parameter as θ = µ + Σ⊙ε, a distribution centered on the provisional value µ with a per-parameter spread Σ (Fortunato, p. 4, § 3).
Hafner and Fortunato are analogous to the claimed invention as both are from the same field of endeavor of exploration in deep reinforcement learning by introducing uncertainty into the parameters of a value/Q neural network. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to draw each parameter of Hafner from a distribution centered on its provisional value, in the manner taught by Fortunato. The motivation to combine Hafner and Fortunato is as recited by Fortunato (Abstract, p. 1): centering the learned per-weight perturbations on the provisional parameter values "aids efficient exploration" while "adding little computational overhead," predictably improving the sample-efficient training of Hafner.
The combination of Hafner and Fortunato, however, does not teach the distribution "having a standard deviation which is inversely related to the respective knowledge parameter."
In the same field of endeavor, Zhang teaches a distribution "having a standard deviation which is inversely related to the respective knowledge parameter." Zhang sets the posterior precision of each weight to the per-parameter Fisher information , the recited knowledge parameter , as "Λ = F̄ + η⁻¹ I" (Zhang, p. 3), draws the weights from a Gaussian whose covariance is the inverse of that precision, "Wₗ ∼ MN(Mₗ, …)" governed by Λ⁻¹ (Zhang, p. 5), and estimates the per-parameter Fisher as a running average of the squared gradient, "f ← β₂·f + (1 − β₂)·(∇_w log p(y|x,w))²" (Zhang, p. 5). Because the sampling standard deviation is proportional to 1/√Λ ∝ 1/√F, a larger knowledge parameter (precision) yields a smaller standard deviation , the recited inverse relation.
Hafner, Fortunato, Azizzadenesheli, and Zhang are analogous to the claimed invention as all are from the same field of endeavor of training neural networks with a learned, per-parameter measure of parameter uncertainty. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to set the standard deviation of the per-parameter sampling distribution of the combination inversely to the Fisher-information knowledge parameter of Zhang. The motivation to combine Hafner, Fortunato, Azizzadenesheli, and Zhang is as recited by Zhang (p. 2–3): tying the sampling width inversely to the Fisher precision concentrates exploratory perturbation on parameters that are poorly determined (low precision, large standard deviation) while holding well-determined parameters (high precision, small standard deviation) stable, yielding a better-calibrated posterior over the parameters and predictably improving the efficiency of exploration in the combination.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Hafner, in view of Fortunato, and Azizzadenesheli, and further in view of Maas et al. (Maas), Non-Patent Literature, "Rectifier Nonlinearities Improve Neural Network Acoustic Models.", published June 2013.
Claim 11 depends on Claim 1. Every limitation carried forward from Claim 1 is rejected under the same rationale set forth for Claim 1 above. The following addresses only the limitation newly introduced in Claim 11:
in which the neural network is composed of a sequence of layers, each layer being composed of a plurality of neural units, and outputs of the neural units in each layer of the sequence, except the last layer of the sequence, being inputs to the neural units of the successive layer of the sequence, the output of each neural unit being a non-linear function of an argument based on corresponding ones of the parameters and the inputs to the neural unit, the derivative of the functions with respect to the argument being non-zero (Hafner, ¶¶ [0004], [0045]–[0046]; Maas, p. 2).
Hafner teaches that the neural network is composed of a sequence of layers, each composed of a plurality of neural units, with the output of each layer serving as input to the next: the network employs "one or more layers of nonlinear units" (Hafner, ¶ [0004]) arranged as a sequence of convolutional, recurrent (LSTM), and/or fully-connected layers (Hafner, ¶¶ [0045]–[0046]). Hafner teaches that the units are nonlinear but does not expressly teach that "the derivative of the functions with respect to the argument [is] non-zero."
In the same field of endeavor, Maas teaches a leaky rectified linear unit: "[t]he leaky rectified linear function (LReL) has a non-zero gradient over its entire domain, unlike the standard ReL function" (Maas, p. 2). The leaky rectifier's non-zero gradient over its entire domain reads on the recited non-linear function whose derivative with respect to the argument is non-zero.
Hafner and Maas are analogous to the claimed invention as all are from the same field of endeavor of constructing and training multi-layer neural networks. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to employ the leaky-rectifier non-linearity of Maas as the non-linear function of the neural units of Hafner's layers. The motivation to combine Hafner and Maas is as recited by Maas: providing a non-zero slope (and hence a non-zero derivative) over the negative region of the argument keeps the unit active and maintains gradient flow through it during backpropagation, avoiding the inactive or "dead" units that occur with a zero-derivative rectifier, which predictably improves the gradient-based training of Hafner's network.
Claims 18 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Hafner, in view of Fortunato, and Azizzadenesheli, and further in view of Leroyer et al. (Leroyer), US 2019/0362506 A1.
Regarding Claim 18, Claim 18 depends from Claim 14 and is rejected for the reasons given for Claims 1 and 14, further in view of the following:
wherein the agent comprises a user of a digital assistant, the method comprising: obtaining information defining the task from the digital assistant; and using the digital assistant to instruct the user to perform the actions (Leroyer, ¶¶ [0003]–[0004], [0037]).
The combination of Hafner, Fortunato, and Azizzadenesheli teaches the trained control method of Claim 14 but does not teach "wherein the agent comprises a user of a digital assistant, … obtaining information defining the task from the digital assistant; and using the digital assistant to instruct the user to perform the actions."
In the same field of endeavor, Leroyer teaches this limitation. Leroyer discloses a digital assistant application that defines and stores a "list of predefined movements (e.g., push-ups, sit-ups, etc.)" (Leroyer, ¶ [0037]), thereby obtaining from the digital assistant the information that defines the task, and that guides the user through the movement, having "been developed to guide individuals through the exercise" and providing feedback "in a form of a visual feedback via a display device" and audio feedback (Leroyer, ¶¶ [0003]–[0004]), thereby using the digital assistant to instruct the user to perform the actions. The recited user of a digital assistant corresponds to Leroyer's exercising subject; the recited information defining the task corresponds to the predefined movement selected for the subject; and the recited instructing of the user corresponds to Leroyer's visual and audio guidance.
Hafner, Fortunato, Azizzadenesheli, and Leroyer are analogous to the claimed invention as all are directed to selecting and guiding the actions of an agent to perform a task in an environment based on captured observations of the agent's state. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to apply the reinforcement-learning-trained action-selection of the combination of Hafner, Fortunato, and Azizzadenesheli to the digital assistant of Leroyer, in which the agent is a user whom the assistant instructs. The motivation to combine Hafner, Fortunato, Azizzadenesheli, and Leroyer is that Leroyer already selects which guidance and instructions to present to the user; using the combination's reinforcement-learning-trained policy to select those instructions predictably improves the adaptiveness and personalization of the guidance presented to the user so as to maximize the user's successful completion of the task.
Regarding Claim 19, Claim 19 depends from Claim 18 and is rejected for the reasons given for Claims 1, 14, and 18, further in view of the following:
further comprising capturing, using the digital assistant, visual or audio observations of the user performing the actions; and determining whether the user has successfully achieved the task (Leroyer, ¶¶ [0003], [0030]–[0031], [0037], [0042]).
The combination of Hafner, Fortunato, and Azizzadenesheli does not teach "capturing, using the digital assistant, visual or audio observations of the user performing the actions; and determining whether the user has successfully achieved the task."
In the same field of endeavor, Leroyer teaches this limitation. Leroyer captures visual observations of the user performing the actions, the system being "developed to guide individuals through the exercise … using video capability of a smartphone," whereby "[t]he captured video (or still images) may be processed" (Leroyer, ¶ [0003]), via a client device "equipped with a built-in camera 204" and an "image capture and movement analysis component 210" that performs "real-time movement analysis" (Leroyer, ¶¶ [0030]–[0031]). Leroyer further determines whether the user has successfully achieved the task by grading and scoring the user's movement, the key positions of the movement being "analyzed and scored" and a "grade of a movement" and "cumulative repetitions" being generated (Leroyer, ¶¶ [0037], [0042]).
Hafner, Fortunato, Azizzadenesheli, and Leroyer are analogous to the claimed invention as all are directed to selecting and guiding the actions of an agent to perform a task in an environment based on captured observations of the agent's state. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to capture observations of the user and determine the user's task success in the manner taught by Leroyer within the digital-assistant method of the combination. The motivation to combine Hafner, Fortunato, Azizzadenesheli, and Leroyer is that Leroyer's capture of images of the user and generation of a grade provide the observations and the task-success determination needed to drive the reinforcement-learning-trained policy of the combination, predictably enabling the digital assistant to evaluate the user's performance and adapt its instructions so as to maximize the user's successful completion of the task.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights