Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/08/2024, 01/03/2025, 06/11/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 7-20 are rejected under 35 U.S.C. 103 as being unpatentable over Reed et al. (US 2020/0104680 A1) in view of Pathak et al. (“Self-Supervised Exploration via Disagreement”).
Regarding claim 1, Reed explicitly discloses:
sampling a latent from a set of possible latents; (Reed, ¶[0006]: “For each of one or more given observations in the sequence of observations, the method includes generating, from the latent representation of the given observation and the latent representations of one or more observations that precede the given observation in the sequence of observations, a context latent representation of the given observation which jointly summarizes the latent representation of the given observation and the latent representations of the one or more observations that precede the given observation in the sequence of observations.”)
selecting actions to be performed by an agent to interact with an environment over a sequence of time steps using an action selection neural network that is conditioned on the sampled latent; (Reed, ¶[0010]: “the encoder neural network is configured for use by a reinforcement learning system to generate latent representations of input observations of an environment which are used to select actions to be performed by the reinforcement learning agent to interact with the environment.”, ¶[0070]: “FIG. 2 illustrates an example data flow 200 for using an action selection neural network 202 to select actions 204 to be performed by an agent 206 interacting with an environment 208 at each of multiple time steps to accomplish a task.”)
determining a respective reward received for each time step in the sequence of time steps, comprising, for each of one or more time steps: (Reed, ¶[0090]: “The training system 300 may determine the reward received by the agent for performing an action at a time step based ( at least in part) on an "imitation score" for the subsequent observation that results from the agent performing the action”)
providing an observation representing a state of the environment at the time step to each discriminator model in an ensemble of multiple discriminator models, (Reed, FIG. 4:
PNG
media_image1.png
894
598
media_image1.png
Greyscale
¶[0027]: “The latent representation of the expert observation is processed using the discriminator neural network to generate a given imitation score.”, ¶[0031]: “In some implementations, the method further includes processing the latent representation of the observation using a second discriminator neural network to generate a second imitation score”)
wherein each discriminator model processes the observation to generate a respective prediction output that predicts which latent from the set of possible latents the action selection neural network was conditioned on to cause the environment to enter the state characterized by the observation; and (Reed, ¶[0105]: “The discriminator network training system 400 is configured to train the discriminator neural network 310 to process an observation characterizing a state of the environment to generate an imitation score characterizing a likelihood that the observation is included in an expert trajectory 402.”)
training the action selection neural network based on the rewards using a reinforcement learning technique. (Reed, ¶[0123]: “The system updates the current values of the action selection network parameters based on the reward using a reinforcement learning training technique (610).”)
Reed fails to disclose:
determining the reward for the time step based at least in part on a measure of disagreement between the prediction outputs generated by the ensemble of multiple discriminator models; and
However, Pathak explicitly discloses:
determining the reward for the time step based at least in part on a measure of disagreement between the prediction outputs generated by the ensemble of multiple discriminator models; and (Pathak, Pg. 3, Col. 2, ¶1]: “Concretely, the intrinsic reward
r
t
i
is defined as the variance across the output of different models in the ensemble:
PNG
media_image2.png
44
411
media_image2.png
Greyscale
… Given the agent’s rollout sequence and the intrinsic reward
r
t
i
at each timestep t, the policy is trained to maximize the
sum of expected reward, i.e.,
PNG
media_image3.png
26
217
media_image3.png
Greyscale
discounted by a factor. Note that the agent is selfsupervised and does not need any extrinsic reward to explore. The agent policy and the forward model ensemble are jointly trained in an online manner on the data collected by the agent during exploration.”, Pg. 2, Fig. 1: “At time step t, the agent in the state xt interacts with the environment by taking action at sampled from the current policy π and ends up in the state xt+1. The ensemble of forward models {f1, f2,…fn} takes this current state xt and the executed action at as input to predict the next state estimates
PNG
media_image4.png
26
160
media_image4.png
Greyscale
. The variance over the ensemble of network output is used as intrinsic reward
r
t
i
to train the policy π. In practice, we encode the state x into an embedding space Ф(x) for all the prediction purposes.”)
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Reed and Pathak. Reed teaches using discriminator neural network system to generate an action selection model in imitation reinforcement learning framework. Pathak teaches training an ensemble of dynamics models and incentivizing the agent to explore such that the disagreement of those ensembles is maximized. One of ordinary skill would have motivation to combine Reed and Pathak to improve exploration in reinforcement learning by avoiding inefficient reward/ gradient estimators and avoiding unreliable uncertainty measures like classifier confidence.
Regarding claim 2, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein for each of the one or more time steps, the prediction output generated by each discriminator model for the time step comprises a respective score distribution over the set of possible latents that defines a respective score for each latent in the set of possible latents. (Reed, ¶[0024]: “The latent representation of the observation is processed using a discriminator neural network to generate an imitation score, where the imitation score characterizes a likelihood that the observation is included in an expert trajectory. An expert trajectory is a sequence of observations characterizing respective states of the environment while a given agent interacts with the environment by performing a sequence of actions that accomplish a particular task.”, ¶[0031]: “the method further includes processing the latent representation of the observation using a second discriminator neural network to generate a second imitation score, where the second imitation score characterizes a likelihood that the observation is: (i) included in an expert trajectory, and (ii) within a threshold number of observations of a last observation of the expert trajectory.”)
Regarding claim 7, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
determining a respective accuracy of the prediction output generated by each discriminator model for the time step; and (Reed, ¶[0024]: “The latent representation of the observation is processed using a discriminator neural network to generate an imitation score, where the imitation score characterizes a likelihood that the observation is included in an expert trajectory.”)
determining the reward for the time step based at least in part on the accuracies of the prediction outputs generated by the discriminator models for the time step. (Reed, ¶[0024]: “The latent representation of the observation is processed using a discriminator neural network to generate an imitation score, where the imitation score characterizes a likelihood that the observation is included in an expert trajectory. An expert trajectory is a sequence of observations characterizing respective states of the environment while a given agent interacts with the environment by performing a sequence of actions that accomplish a particular task. A reward is determined from the imitation score”, ¶[0026]: “determining the reward from the task reward in addition to the imitation score includes determining the reward as a weighted linear combination of the task reward and the imitation score.”)
Regarding claim 8, the combination of Reed and Pathak discloses all the limitations of claim 7 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein determining the reward for the time step based at least in part on the accuracies of the prediction outputs generated by the discriminator models for the time step comprises: (Reed, ¶[0024]: “The latent representation of the observation is processed using a discriminator neural network to generate an imitation score, where the imitation score characterizes a likelihood that the observation is included in an expert trajectory. An expert trajectory is a sequence of observations characterizing respective states of the environment while a given agent interacts with the environment by performing a sequence of actions that accomplish a particular task. A reward is determined from the imitation score”
determining the reward based at least in part on an average of the accuracies of the prediction outputs generated by the discriminator models for the time step. (Reed, ¶[0024]: “The latent representation of the observation is processed using a discriminator neural network to generate an imitation score, where the imitation score characterizes a likelihood that the observation is included in an expert trajectory. An expert trajectory is a sequence of observations characterizing respective states of the environment while a given agent interacts with the environment by performing a sequence of actions that accomplish a particular task. A reward is determined from the imitation score”, ¶[0026]: “determining the reward from the task reward in addition to the imitation score includes determining the reward as a weighted linear combination of the task reward and the imitation score.”)
Regarding claim 9, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein each discriminator model has a respective set of discriminator model parameters, and (Reed, ¶[0111]: “The discriminator neural network may be trained in parallel with the action selection neural network. That is, the discriminator neural network may be trained using observations resulting from actions selected in accordance with the latest values of the action selection network parameters, and the action selection network may be trained using imitation scores generated in accordance with the latest values of the discriminator network parameters”)
wherein respective values of the respective set of discriminator model parameters are different for each discriminator model. (Reed, ¶[0027]: “The latent representation of the expert observation is processed using the discriminator neural network to generate a given imitation score.”, ¶[0028]: “The current values of the discriminator neural network parameters are adjusted based on the loss.”, ¶[0031]: “the method further includes processing the latent representation of the observation using a second discriminator neural network to generate a second imitation score, where the second imitation score characterizes a likelihood that the observation”)
Regarding claim 10, the combination of Reed and Pathak discloses all the limitations of claim 9 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein each discriminator model is trained on independently sampled batches of training examples from a replay memory. (Reed, ¶[0066]: “At each training iteration, the training engine 118 may train the encoder neural network 102 on a respective "batch" (set) of sequences of observations sampled from the training data 104.”, ¶[0094]: “the training system 300 trains the action selection network 202 on "experience tuples" stored in a replay buffer 304 that characterize previous interactions of the agent with the environment… In some cases, the replay buffer 304 may store experience tuples characterizing interactions of multiple agents with respective environments, e.g., interactions of multiple agents with different instantiations of a simulated environment.”)
Regarding claim 11, the combination of Reed and Pathak discloses all the limitations of claim 10 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein each training example in the replay memory comprises:
(i) a training observation characterizing a state of the environment during a previous interaction of the agent with the environment, and (Reed, ¶0094]: “In particular, the training system 300 trains the action selection network 202 on "experience tuples" stored in a replay buffer 304 that characterize previous interactions of the agent with the environment. Each experience tuple may specify: (i) an observation characterizing the state of the environment at a time step,”)
(ii) a target output that defines a latent that the action selection neural network was conditioned on to cause the environment to enter the state characterized by the observation. (Reed, ¶[0094]: “In particular, the training system 300 trains the action selection network 202 on "experience tuples" stored in a replay buffer 304 that characterize previous interactions of the agent with the environment. Each experience tuple may specify:.. (iv) a "task reward" that may indicate, e.g., whether the agent has accomplished the task, or the progress of the agent towards accomplishing a task.”)
Regarding claim 12, the combination of Reed and Pathak discloses all the limitations of claim 9 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein before being trained, the respective values of the set of discriminator model parameters of each discriminator model are initialized to be different than the set of discriminator model parameters of each other discriminator model. (Reed, ¶[0095]: “At each of multiple training iterations, the training system 300 selects a "batch" (set) of experience tuples 306 from the replay buffer 304, and generates a respective latent representation 308 of the subsequent observation specified by each experience tuple 306”, ¶[0100]: “The training system 300 determines a respective reward 314 corresponding to each experience tuple 306 based on the imitation score 312 for the subsequent observation specified by the experience tuple 306. For example, for an experience tuple (s, a, rtask, s’) where s is the initial observation, a is the action performed in response to the observation s, rtask is the task reward, and s' is the subsequent observation resulting from performing the action a, the training system 300 may determine the reward r corresponding to the experience tuple as:”)
Regarding claim 13, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
for each time step in the sequence of time steps: determining a reward for the time step based at least in part on a task reward that measures a progress of the agent in accomplishing a task in the environment. (Reed, ¶[0094]: “"task reward" that may indicate, e.g., whether the agent has accomplished the task, or the progress of the agent towards accomplishing a task. The task reward may be sparse (i.e., rarely non-zero), e.g., the agent may receive a non-zero task reward only at the time step that the task is accomplished.”)
Regarding claim 14, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein the set of possible latents comprises only finitely many possible latents. (Reed, ¶[0045]: “Generating the imitation scores based on the latent representations of observations results in the action selection network being trained to imitate the expert demonstrations in "latent space" (i.e., the space of possible latent representations)”, ¶[0058]: “The latent representation of an observation can be represented as an ordered collection of numerical values, e.g., a vector or matrix of numerical values”)
Regarding claim 15, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein selecting actions to be performed by the agent to interact with the environment over the sequence of time steps using the action selection neural network that is conditioned on the sampled latent comprises, for each time step: (Reed, ¶[0010]: “the encoder neural network is configured for use by a reinforcement learning system to generate latent representations of input observations of an environment which are used to select actions to be performed by the reinforcement learning agent to interact with the environment.”, ¶[0070]: “FIG. 2 illustrates an example data flow 200 for using an action selection neural network 202 to select actions 204 to be performed by an agent 206 interacting with an environment 208 at each of multiple time steps to accomplish a task.”)
processing an observation characterizing a state of the environment at the time step and the sampled latent using the action selection neural network to generate an action selection output; and (Reed, ¶[0070]: “At each time step, the action selection network 202 processes an observation 210 characterizing the current state of the environment 208 in accordance with the current values of its model parameters 214 to generate action scores 212 that are used to select an action 204 to be performed by the agent 206 in response to the observation. At each time step, the state of the environment 208 at the time step (as characterized by the observation 210) depends on the state of the environment 208 at the previous time step and the action 204 performed by the agent 206 at the previous time step.”)
selecting the action to be performed at the time step based on the action selection output. (Reed, ¶[0087]: “The action scores 212 generated by the action selection network 202 at each time step may include a respective numerical value for each action in a set of possible actions that can be performed by the agent at the time step. The action scores 212 can be used in any of a variety of ways to determine the action 204 to be performed by the agent 206 at a time step… the action with the highest action score 212 may selected as the action to be performed at the time step.””)
Regarding claim 16, the combination of Reed and Pathak discloses all the limitations of claim 15 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein for each time step, the action selection output comprises a respective score for each action in a set of possible actions. (Reed, ¶[0087]: “The action scores 212 generated by the action selection network 202 at each time step may include a respective numerical value for each action in a set of possible actions that can be performed by the agent at the time step. The action scores 212 can be used in any of a variety of ways to determine the action 204 to be performed by the agent 206 at a time step.”)
Regarding claim 17, the combination of Reed and Pathak discloses all the limitations of claim 16 (as shown in the rejection above).
Reed in view of Pathak further discloses:
wherein for each time step, selecting the action to be performed at the time step based on the action selection output comprises: selecting the action having the highest score according to the action selection output. (Reed, ¶[0087]: “the action with the highest action score 212 may selected as the action to be performed at the time step.”)
Regarding claim 18, the combination of Reed and Pathak discloses all the limitations of claim 1 (as shown in the rejection above).
Reed in view of Pathak further discloses:
further comprising, after training the action selection neural network, using the action selection neural network to select actions to be performed by a real-world agent interacting with a real-world environment. (Reed, ¶[0084]: “Training an agent in a simulated environment may enable the agent to learn from large amounts of simulated training data while avoiding risks associated with training the agent in a real world environment, e.g., damage to the agent due to performing poorly chosen actions. An agent trained in a simulated environment may thereafter be deployed in a real-world environment.”, ¶[0089]: “To enable the action selection network 202 to select actions that cause the agent to accomplish a task, the training system 300 trains the action selection network 202 to select actions that maximize a cumulative measure of rewards received by the agent using reinforcement learning technique”)
Independent claims 19 and 20 are rejected under the same rationale with claim 1 because they are analogous claims.
Claim(s) 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over Reed et al. (US 2020/0104680 A1) in view of Pathak et al. (“Self-Supervised Exploration via Disagreement”) and further in view of Lakshminarayanan et al. (“Simple and Scalable Predictive Uncertainty
Estimation using Deep Ensembles”).
Regarding claim 3, the combination of Reed and Pathak discloses all the limitations of claim 2 (as shown in the rejections above).
Reed in view of Pathak further discloses:
further comprising, for each of the one or more time steps, determining the measure of disagreement between the prediction outputs generated by the ensemble of multiple discriminator models, comprising: (Pathak, Pg. 3, Col. 2, ¶1]: “Concretely, the intrinsic reward
r
t
i
is defined as the variance across the output of different models in the ensemble:
PNG
media_image2.png
44
411
media_image2.png
Greyscale
… Given the agent’s rollout sequence and the intrinsic reward
r
t
i
at each timestep t, the policy is trained to maximize the
sum of expected reward, i.e.,
PNG
media_image3.png
26
217
media_image3.png
Greyscale
discounted by a factor. Note that the agent is selfsupervised and does not need any extrinsic reward to explore. The agent policy and the forward model ensemble are jointly trained in an online manner on the data collected by the agent during exploration.”, Pg. 2, Fig. 1: “At time step t, the agent in the state xt interacts with the environment by taking action at sampled from the current policy π and ends up in the state xt+1. The ensemble of forward models {f1, f2,…fn} takes this current state xt and the executed action at as input to predict the next state estimates
PNG
media_image4.png
26
160
media_image4.png
Greyscale
. The variance over the ensemble of network output is used as intrinsic reward
r
t
i
to train the policy π. In practice, we encode the state x into an embedding space Ф(x) for all the prediction purposes.”)
Reed in view of Pathak fails to disclose:
determining a combined score distribution over the set of possible latents for the time step by combining the respective score distribution generated by each discriminator model for the time step; and
determining the measure of disagreement based on: (i) the combined score distribution for the time step, and (ii) the respective score distribution generated by each discriminator model for the time step.
However, Lakshminarayanan explicitly discloses:
determining a combined score distribution over the set of possible latents for the time step by combining the respective score distribution generated by each discriminator model for the time step; and (Lakshminarayanan, Pg. 2, ¶[2]: “…dropout may also be interpreted as ensemble model combination [54] where the predictions are averaged over an ensemble of NNs (with parameter sharing).”, Pg. 5, ¶[1]: “We treat the ensemble as a uniformly-weighted mixture model and combine the predictions as
PNG
media_image5.png
27
294
media_image5.png
Greyscale
For classification, this corresponds to averaging the predicted probabilities. For regression, the prediction is a mixture of Gaussian distributions. For ease of computing quantiles and predictive probabilities, we further approximate the ensemble prediction as a Gaussian whose mean and variance are respectively the mean and variance of the mixture. The mean and variance of a mixture
PNG
media_image6.png
24
237
media_image6.png
Greyscale
are given by
PNG
media_image7.png
30
217
media_image7.png
Greyscale
and
PNG
media_image8.png
30
388
media_image8.png
Greyscale
respectively”) [Examiner’s note: Lakshminarayanan teaches each NN in the ensemble outputs its own predictive distribution, which is interpreted as a respective score distribution, then Lakshminarayanan states that the ensemble combines the predictions as p(y|x). For classification, the reference says this is done by averaging the predicted probabilities, thus this teaches determining a combined score distribution by combining the individual score or probabilities distributions generated by each model.]
determining the measure of disagreement based on: (i) the combined score distribution for the time step, and (ii) the respective score distribution generated by each discriminator model for the time step. (Lakshminarayanan, Pg. 7, ¶[4]: “Another advantage of using an ensemble is that it enables us to easily identify training examples where the individual networks disagree or agree the most. This disagreement provides another useful qualitative way to evaluate predictive uncertainty.”, Pg. 7, Footnote 9: “… we define disagreement as
PNG
media_image9.png
26
241
media_image9.png
Greyscale
where KL denotes the Kullback-Leibler divergence and
PNG
media_image10.png
22
224
media_image10.png
Greyscale
is the prediction of the ensemble”) [Examiner’s note: This section teaches measuring disagreement because it defines disagreement as the sum of KL divergences between each individual model’s predictive distribution
PNG
media_image11.png
23
64
media_image11.png
Greyscale
and the ensemble prediction
PNG
media_image12.png
21
63
media_image12.png
Greyscale
. The ensemble prediction is expressly defined as the average of the individual predictive distributions. Therefore, the disagreement is measured based on both the combined score distribution and the respective score distribution generated by each model]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Reed, Pathak and Lakshminarayanan. Reed teaches using discriminator neural network system to generate an action selection model in imitation reinforcement learning framework. Pathak teaches training an ensemble of dynamics models and incentivizing the agent to explore such that the disagreement of those ensembles is maximized. Lakshminarayanan teaches using multiple neural networks, where each network outputs its own predictive probability distribution, and then the system combine/ averages those distributions to form an ensemble predictive distribution. One of ordinary skill would have motivation to combine Reed, Pathak and Lakshminarayanan to improve uncertainty estimation and reliability of the discriminator’s prediction output by using an ensemble of models and combining their predictive distributions.
Regarding claim 4, the combination of Reed and Pathak discloses all the limitations of claim 2 (as shown in the rejections above).
Reed in view of Pathak fails to disclose:
wherein determining the combined score distribution over the set of possible latents for the time step comprises averaging the score distributions generated by the discriminator models for the time step
However, Lakshminarayanan explicitly discloses:
wherein determining the combined score distribution over the set of possible latents for the time step comprises averaging the score distributions generated by the discriminator models for the time step. (Lakshminarayanan, Pg. 2, ¶[2]: “…dropout may also be interpreted as ensemble model combination [54] where the predictions are averaged over an ensemble of NNs (with parameter sharing).”, Pg. 5, ¶[1]: “We treat the ensemble as a uniformly-weighted mixture model and combine the predictions as
PNG
media_image5.png
27
294
media_image5.png
Greyscale
For classification, this corresponds to averaging the predicted probabilities. For regression, the prediction is a mixture of Gaussian distributions. For ease of computing quantiles and predictive probabilities, we further approximate the ensemble prediction as a Gaussian whose mean and variance are respectively the mean and variance of the mixture. The mean and variance of a mixture
PNG
media_image6.png
24
237
media_image6.png
Greyscale
are given by
PNG
media_image7.png
30
217
media_image7.png
Greyscale
and
PNG
media_image8.png
30
388
media_image8.png
Greyscale
respectively”)
Claim(s) 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Reed et al. (US 2020/0104680 A1) in view of Pathak et al. (“Self-Supervised Exploration via Disagreement”) and further in view of Lakshminarayanan et al. (“Simple and Scalable Predictive Uncertainty
Estimation using Deep Ensembles”) and Houlsby et al. (“Bayesian Active Learning for Classification and Preference Learning”).
Regarding claim 5, the combination of Reed, Pathak and Lakshminarayanan discloses all the limitations of claim 3 (as shown in the rejections above).
Reed in view of Pathak and Lakshminarayanan further discloses:
wherein determining the measure of disagreement based on: (i) the combined score distribution for the time step, and (ii) the respective score distribution generated by each discriminator model for the time step, comprises: (Lakshminarayanan, Pg. 7, ¶[4]: “Another advantage of using an ensemble is that it enables us to easily identify training examples where the individual networks disagree or agree the most. This disagreement provides another useful qualitative way to evaluate predictive uncertainty.”, Pg. 7, Footnote 9: “… we define disagreement as
PNG
media_image9.png
26
241
media_image9.png
Greyscale
where KL denotes the Kullback-Leibler divergence and
PNG
media_image10.png
22
224
media_image10.png
Greyscale
is the prediction of the ensemble”) [Examiner’s note: This section teaches measuring disagreement because it defines disagreement as the sum of KL divergences between each individual model’s predictive distribution
PNG
media_image11.png
23
64
media_image11.png
Greyscale
and the ensemble prediction
PNG
media_image12.png
21
63
media_image12.png
Greyscale
. The ensemble prediction is expressly defined as the average of the individual predictive distributions. Therefore, the disagreement is measured based on both the combined score distribution and the respective score distribution generated by each model]
determining the measure of disagreement based on the respective entropies of the combined score distribution for the time step and the respective score distribution generated by each discriminator model for the time step. (Lakshminarayanan, Pg. 7, ¶[4]: “Another advantage of using an ensemble is that it enables us to easily identify training examples where the individual networks disagree or agree the most. This disagreement provides another useful qualitative way to evaluate predictive uncertainty.”, Pg. 7, Footnote 9: “… we define disagreement as
PNG
media_image9.png
26
241
media_image9.png
Greyscale
where KL denotes the Kullback-Leibler divergence and
PNG
media_image10.png
22
224
media_image10.png
Greyscale
is the prediction of the ensemble”) [Examiner’s note: This section teaches measuring disagreement because it defines disagreement as the sum of KL divergences between each individual model’s predictive distribution
PNG
media_image11.png
23
64
media_image11.png
Greyscale
and the ensemble prediction
PNG
media_image12.png
21
63
media_image12.png
Greyscale
. The ensemble prediction is expressly defined as the average of the individual predictive distributions. Therefore, the disagreement is measured based on both the combined score distribution and the respective score distribution generated by each model]
Reed in view of Pathak and Lakshminarayanan fails to disclose:
determining an entropy of the combined score distribution for the time step;
determining a respective entropy of the respective score distribution generated by each
discriminator model for the time step; and
However, Houlsby explicitly discloses:
determining an entropy of the combined score distribution for the time step;
(Houlsby, Pg. 3, ¶[3-4]: “Using this insight it is simple to show that the objective can be rearranged to compute entropies in y space:
PNG
media_image13.png
77
463
media_image13.png
Greyscale
… Eqn.(2) also provides us with an interesting intuition about the objective; we seek the x for which the model is marginally most uncertain about y (high H[y|x,D]), but for which individual settings of the parameters are confident
PNG
media_image14.png
34
299
media_image14.png
Greyscale
. This can be interpreted as seeking the x for which the parameters under the posterior disagree about the outcome the most, so we refer to this objective as Bayesian Active Learning by Disagreement (BALD).”) [Examiner’s note: “combined score distribution entropy” is being interpreted as H[y|x, D] because this is the entropy of the marginal/ overall predictive distribution.]
determining a respective entropy of the respective score distribution generated by each
discriminator model for the time step; and (Houlsby, Pg. 3, ¶[3-4]: “Using this insight it is simple to show that the objective can be rearranged to compute entropies in y space:
PNG
media_image13.png
77
463
media_image13.png
Greyscale
… Eqn.(2) also provides us with an interesting intuition about the objective; we seek the x for which the model is marginally most uncertain about y (high H[y|x,D]), but for which individual settings of the parameters are confident
PNG
media_image14.png
34
299
media_image14.png
Greyscale
. This can be interpreted as seeking the x for which the parameters under the posterior disagree about the outcome the most, so we refer to this objective as Bayesian Active Learning by Disagreement (BALD).”) [Examiner’s note: “respective score distribution entropy” is being interpreted as H[y | x, ϴ] inside
PNG
media_image15.png
53
234
media_image15.png
Greyscale
because this is the entropy of the predictive distribution for each sampled model.]
It would have obvious to one of ordinary skill in the art before the effective filing date of the present application to combine Reed, Pathak, Lakshminarayanan and Houlsby. Reed teaches using discriminator neural network system to generate an action selection model in imitation reinforcement learning framework. Pathak teaches training an ensemble of dynamics models and incentivizing the agent to explore such that the disagreement of those ensembles is maximized. Lakshminarayanan teaches using multiple neural networks, where each network outputs its own predictive probability distribution, and then the system combine/ averages those distributions to form an ensemble predictive distribution. Houlsby teaches an information-theoretic active learning approach that measures uncertainty or disagreement using predictive entropy and KL-divergence-type quantities. One of ordinary skill would have motivation to combine Reed, Pathak, Lakshminarayanan and Houlsby to add a known way to quantify model disagreement using predictive entropy, specifically by comparing the entropy of the combined predictive distribution with the average entropy of the individual model predictions.
Regarding claim 6, the combination of Reed, Pathak and Lakshminarayanan discloses all the limitations of claim 5 (as shown in the rejections above).
Reed in view of Pathak and Lakshminarayanan further discloses:
wherein determining the measure of disagreement based on the respective entropies of the combined score distribution for the time step and the respective score distribution generated by each discriminator model for the time step comprises: (Lakshminarayanan, Pg. 7, ¶[4]: “Another advantage of using an ensemble is that it enables us to easily identify training examples where the individual networks disagree or agree the most. This disagreement provides another useful qualitative way to evaluate predictive uncertainty.”, Pg. 7, Footnote 9: “… we define disagreement as
PNG
media_image9.png
26
241
media_image9.png
Greyscale
where KL denotes the Kullback-Leibler divergence and
PNG
media_image10.png
22
224
media_image10.png
Greyscale
is the prediction of the ensemble”) [Examiner’s note: This section teaches measuring disagreement because it defines disagreement as the sum of KL divergences between each individual model’s predictive distribution
PNG
media_image11.png
23
64
media_image11.png
Greyscale
and the ensemble prediction
PNG
media_image12.png
21
63
media_image12.png
Greyscale
. The ensemble prediction is expressly defined as the average of the individual predictive distributions. Therefore, the disagreement is measured based on both the combined score distribution and the respective score distribution generated by each model]
Reed in view of Pathak and Lakshminarayanan fails to disclose:
determining the measure of disagreement as a difference between: (i) the entropy of the combined score distribution for the time step, and (ii) an average of the entropies of the score distributions generated by each discriminator model for the time step.
However, Houlsby explicitly teaches:
determining the measure of disagreement as a difference between: (i) the entropy of the combined score distribution for the time step, and (ii) an average of the entropies of the score distributions generated by each discriminator model for the time step. (Houlsby, Pg. 3, ¶[3-4]: “Using this insight it is simple to show that the objective can be rearranged to compute entropies in y space:
PNG
media_image13.png
77
463
media_image13.png
Greyscale
… Eqn.(2) also provides us with an interesting intuition about the objective; we seek the x for which the model is marginally most uncertain about y (high H[y|x,D]), but for which individual settings of the parameters are confident
PNG
media_image14.png
34
299
media_image14.png
Greyscale
. This can be interpreted as seeking the x for which the parameters under the posterior disagree about the outcome the most, so we refer to this objective as Bayesian Active Learning by Disagreement (BALD).”) [Examiner’s note: “respective score distribution entropy” is being interpreted as H[y | x, ϴ] inside
PNG
media_image15.png
53
234
media_image15.png
Greyscale
because this is the entropy of the predictive distribution for each sampled model, “combined score distribution entropy” is being interpreted as H[y|x, D] because this is the entropy of the marginal/ overall predictive distribution, “measure of disagreement as a difference” is being interpreted as
PNG
media_image16.png
46
367
media_image16.png
Greyscale
]
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY TRAN whose telephone number is (571)270-0693. The examiner can normally be reached Monday - Friday 7:30 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY TRAN/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights