DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 8-9, 17, 20-22, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rohit Prakash Prabhavalkar et al. (hereinafter Prabhavalkar) (US 20200043483 A1, 2020-02-06) in view of Emilio Parisotto et al. (hereinafter Parisotto) (“STABILIZING TRANSFORMERS FOR REINFORCEMENT LEARNING”, 2019-10-13).
NOTE: As stated in applicant spec, “Each sub network can be implemented as a group of one or more neural network layers in the action selection neural network 120.”
NOTE: In Prabhavalkar, input and output time steps may be the same; “[0068] The input time steps u and the output time steps i can be different, for example, so that there are multiple input time steps for each output time step, but this is not required.”
Regarding claim 1, Prabhavalkar teaches;
A system for controlling an agent interacting with an environment to perform a task, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement:
([0080] Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate)
NOTE: Teaches one or more computers (computer program products are implemented using at least one computer) and one or more storage devices storing instructions that (computer program encoded on a computer readable medium), when executed by the one or more computers (execution by a processing apparatus), cause the one or more computers to implement the invention (embodiments of the invention can be implemented as one or more computer program products).
an action selection neural network configured to generate an action selection output that is used to select an action to be performed by the agent,
([Abstract] the speech recognition model including an encoder, an attention module, and a decoder. The encoder and decoder each include one or more recurrent neural network layers.)
NOTE: The speech recognition model is a neural network.
([0075] In response to generating the transcription 420 using the speech recognition model 402, the electronic device 410 can perform any of various actions.)
NOTE: Teaches an action selection neural network (the speech recognition model) configured to generate an action selection output (the generated transcription) that is used to select an action to be performed by the agent (the generated transcript [action selection output] is used to select an action to be performed by the electronic device [agent]).
the action selection neural network comprising: an encoder sub network ([Abstract] the speech recognition model including an encoder, … encoder … include one or more recurrent neural network layers) configured, at each of a plurality of time steps, to receive an encoder sub network input comprising a current observation characterizing a current state of the environment and to generate an encoded representation of the current observation;
([0067] During the speech recognition process, the extracted feature values are provided as inputs to the encoder 404 of the speech recognition model 402. The encoder 404 generates an encoded feature representation as an output... This process of generating an encoded feature representation, h.sup.enc, can be done for each of multiple input frames, representing different input time steps. These time steps are denoted with subscript u below. For example, the inputs can be provided over a set of input steps {0, 1, 2, 3, . . . u}, each corresponding to an input feature vector {v.sub.0, v.sub.1, v.sub.2, v.sub.3, . . . v.sub.u}, which results in a corresponding set of encoding outputs {h.sub.0, h.sub.1, h.sub.2, h.sub.3, . . . h.sub.u}. [0064] A feature extraction module 416 processes the audio data 414 to extract (e.g., generate) a set of feature values that are indicative of acoustic characteristics of the utterance.)
NOTE: Teaches an encoder sub network (the encoder, which includes multiple NN layers in the speech recognition model, and is considered a subnetwork of the model) configured, at each of a plurality of time steps, to receive an encoder sub network input (for each input time step, the extracted feature values are provided as input to the encoder) comprising a current observation characterizing a current state of the environment (each input time step corresponds to a current observation, where the extracted feature values characterize the acoustic state of the audio environment at that time step) and to generate an encoded representation of the current observation (the encoder generates encoded representations of the current observation at each time step, h.sub.0, h.sub.1, etc.).
an attention sub network ([Abstract] model including an … attention module [0066] the attention module 406, … may be implemented using one or more neural network layers) configured, at each of the plurality of time steps, to receive an attention sub network input comprising the encoded representation of the current observation and to generate an attention sub network output at least in part by applying an attention mechanism using the one or more multi-head attention layers over the encoded representation of the current observation and encoded representations of one or more previous observations characterizing one or more previous states of the environment;
([0068] The output of the encoder 404 is processed using the attention module 406 to generate one or more context vectors c.sub.i for each output time step i... For each context output vector c.sub.i, the attention module 406 can compute attention based on the encodings for one or more input steps u, e.g., the encoding for the current input step as well as encodings for previous input steps. For example, the attention module 406 can generate an attention context output c.sub.i over the set of all the encoder outputs of the utterance, e.g., the entire set {h.sub.0, h.sub.1, h.sub.2, h.sub.3, h.sub.u}. The attention context vector c.sub.i can be a vector representing a weighted summary of the current and previous encodings for frames (e.g., portions) of the utterance being recognized. [0078] the attention mechanism can have multiple heads)
NOTE: Teaches an attention sub network (the attention module) configured, at each of the plurality of time steps, to receive an attention sub network input comprising the encoded representation of the current observation (input to the attention module includes the encoded current observation at each time step) and to generate an attention sub network output (generates attention context output c.sub.i) at least in part by applying an attention mechanism using the one or more multi-head attention layers (the attention module, implemented using one or more layers, uses a multi-head attention mechanism) over the encoded representation of the current observation and encoded representations of one or more previous observations (computes attention context output over all encoder outputs, including current and previous encoder outputs) characterizing one or more previous states of the environment (as previously taught).
a recurrent sub network ([Abstract] model including an … decoder … include one or more recurrent neural network layers) configured, at each of the plurality of time steps, to receive a recurrent sub network input
([0070] For example, the decoder 408 can process inputs including context vector c.sub.i as well as output y.sub.i-1 to generate decoder output used to create output y.sub.i. The output distribution y.sub.i is a function of the decoder state s.sub.i and context vector c.sub.i. The decoder state s.sub.i is a function of the previous decoder state, s.sub.i-1, the previously emitted character, as indicated by the previous output distribution y.sub.i-1, and the previous context vector c.sub.i-1.)
NOTE: Teaches a recurrent sub network (the decoder) configured, at each of the plurality of time steps, to receive a recurrent sub network input derived from the attention sub network output (the outputs of the attention module [context vectors] at each time step are used as input to the decoder) to update a current hidden state of the recurrent sub network that corresponds to the time step (decoder state s.sub.i-1 is updated to s.sub.i using corresponding context vector c.sub.i-1) and to generate a recurrent sub network output (the decoder state and context vector are used to generate decoder output).
and an action selection sub network configured, at each of the plurality of time steps, to receive an action selection sub network input comprising the recurrent sub network output and to generate the action selection output that is used to select the action to be performed by the agent in response to the current observation.
([0038] decoder network 130 which models an output distribution over the next target conditioned on the sequence of previous predictions: P(y.sub.u|y*.sub.u-1; y*.sub.u-2 . . . ; y*.sub.0; x)... 1 the output of the decoder network 130 can be processed by a softmax layer 140 to obtain the probability distribution. [0073] The probability distribution can then be used to select and assign scores to candidate orthgraphic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process for determining the transcription. [0075] In response to generating the transcription 420 using the speech recognition model 402, the electronic device 410 can perform any of various actions.)
NOTE: Teaches an action selection sub network (the softmax layer, which is at least one neural network layer, which can be considered a subnetwork of the model) configured, at each of the plurality of time steps, to receive an action selection sub network input comprising the recurrent sub network output (the output of the decoder for each time step is processed by / provided as input to the softmax layer) and to generate the action selection output (the softmax layer generates the probability distribution, which determines the transcription, which can be considered the action selection output) that is used to select the action to be performed by the agent in response to the current observation (the resulting transcription is used to select an action to be performed by the electronic device).
Prabhavalkar fails to teach but Parisotto teaches;
an attention … network comprising one or more multi-head attention layers and a position-wise multi-layer perceptron ([pg. 3] Transformer variants, showing just a single layer block (there are L layers total) … with multi-head attention and position-wise MLP), generate an attention … network output at least in part by applying an attention mechanism using the one or more multi-head attention layers ([pg. 2] The Multi-Head Attention (MHA) submodule computes in parallel H soft attention operations for every time step.)
a gating layer configured ([pg. 4] Y(l) = g(l)MHA(E(l−1), ReLU(Ȳ(l))) … where g is a gating layer function), at each of the plurality of time steps, to receive i) the encoded representation of the current observation ([pg. 2] input to the transformer block is an embedding from the previous layer E(l−1) ∈ RT×D, where T is the number of time steps … E(0) is an … embedding of the per-timestep observations in an RL environment) and ii) the attention sub network output ([pg. 2] Ȳ(l) = MultiHeadAttention(E(l−1))), and to generate a([pg. 4] Y(l) = g(l)MHA(E(l−1), ReLU(Ȳ(l))) … where g is a gating layer function),
OBVIOUSNESS TO COMBINE PARISOTTO:
Prabhavalkar teaches an attention subnetwork that receives encoded representations of current observations as input to generate attention subnetwork outputs, and using the attention subnetwork outputs as input to a recurrent subnetwork. Parisotto teaches an attention based neural network architecture having one or more multi-head attention layers and a position-wise multi-layer perceptron, generating output at least in part by applying the one or more multi-head attention layers, and a gating layer configured, at each of the plurality of time steps, to receive the input representation and the output generated by the multi-head attention operation to generate a gated representation. Additionally, Parisotto states that their proposed transformer model architecture improves stability and learning speed over previous attention-based models (transformers, which are built around attention mechanisms), while still achieving state-of-the-art performance; “[Abstract] We propose architectural modifications that substantially improve the stability and learning speed of the original Transformer and XLvariant. The proposed architecture, the Gated Transformer-XL (GTrXL), … achieves state-of-the-art results on the multi-task DMLab-30 benchmark suite, exceeding the performance of an external memory architecture.” Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify the attention subnetwork of Prabhavalkar to incorporate the multi-head attention, position-wise multi-layer perceptron, and to further apply the gating mechanism taught by Parisotto, while retaining the recurrent subnetwork of Prabhavalkar, predictably improving stability and learning speed while achieving improved performance. In the resulting combination, Parisotto’s gating mechanism would receive the encoded representation supplied to the attention processing and the output generated using the multi-head attention mechanism, and the resulting gated representation would serve as the recurrent subnetwork input provided to the recurrent subnetwork of Prabhavalkar.
Regarding claim 2, Prabhavalkar teaches;
wherein the encoded representation of the current observation comprises an input vector having a respective input value at each of a plurality of input positions in an input order.
([0067] This process of generating an encoded feature representation, h.sup.enc, can be done for each of multiple input frames, representing different input time steps. These time steps are denoted with subscript u below. For example, the inputs can be provided over a set of input steps {0, 1, 2, 3, . . . u}, each corresponding to an input feature vector {v.sub.0, v.sub.1, v.sub.2, v.sub.3, . . . v.sub.u}, which results in a corresponding set of encoding outputs {h.sub.0, h.sub.1, h.sub.2, h.sub.3, . . . h.sub.u}.)
NOTE: Teaches the encoded representation of the current observation (encoded feature representation) comprises an input vector having a respective input value at each of a plurality of input positions in an input order. ({v.sub.0, v.sub.1, v.sub.2, v.sub.3, . . . v.sub.u}).
Regarding claim 3, Prabhavalkar fails to explicitly teach but Parisotto teaches;
wherein the attention
([pg. 3] Figure 1: Transformer variants, showing just a single layer block (there are L layers total). Left: Canonical Transformer(-XL) block with multi-head attention and position-wise MLP submodules and the standard layer normalization (Ba et al., 2016) placement with respect to the residual connection (He et al., 2016a).)
NOTE: Teaches an attention-based network (transformer) comprising a plurality of attention layers (L layers total, where each layer block has multi-head attention).
each configured to receive an attention layer input for each of a plurality of layer input positions
([pg. 2] The input to the transformer block is an embedding from the previous layer E(l−1) ∈ RT×D, where T is the number of time steps, D is the hidden dimension, and l ∈ [0, L] is the layer index with L being the total number of layers.)
NOTE: Teaches each layer configured to receive an attention layer input (each layer block receives an embedding from the previous layer as input) for each of a plurality of layer input positions (time steps).
and, for each particular layer input position in a layer input order: apply the attention mechanism over the attention layer inputs at the layer input positions using one or more queries derived from the attention layer input at the particular layer input position to generate a respective attention layer output for the particular layer input position.
[pg. 18]
PNG
media_image1.png
155
831
media_image1.png
Greyscale
NOTE: Teaches for each particular layer input position in a layer input order (for each input position / time step t, in the order of the temporal sequence of observations): apply the attention mechanism over the attention layer inputs at the layer input positions using one or more queries derived from the attention layer input at the particular layer input position (calculating query Q^t, derived from input position t, to compute the soft attention) to generate a respective attention layer output for the particular layer input position (producing an output tensor for every time step).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 4, Prabhavalkar fails to teach but Parisotto teaches;
wherein the attention mechanism is a masked attention mechanism.
[pg. 19]
PNG
media_image2.png
369
913
media_image2.png
Greyscale
NOTE: Teaches masked softmax being used in the multihead attention.
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 5, Prabhavalkar teaches;
wherein the recurrent sub network comprises one or more long short-term memory (LSTM) layers.
([0015] In some implementations, the encoder and decoder each include a plurality of unidirectional LSTM layers.)
NOTE: Teaches the recurrent sub network (the decoder) comprises one or more long short-term memory (LSTM) layers.
Regarding claim 8, Prahavalkar fails to teach but Parisotto teaches;
wherein applying the gating mechanism to i) the encoded representation of the current observation and ii) the attention sub network output comprises: applying a gated recurrent unit (GRU) on i) the encoded representation of the current observation and ii) the attention sub network output.
([pg. 4] Gated-Recurrent-Unit-type gating: The Gated Recurrent Unit (GRU) … We adapt its powerful gating mechanism as an untied activation function in depth)
NOTE: Parisotto teaches applying the gating mechanism to i) the encoded representation of the current observation and ii) the attention sub network output (as taught in claim 1 above) by where the gating mechanism is a gated recurrent unit (GRU).
OBVIOUSNESS: Using the same reasoning from claim 1.
Regarding claim 9, Prabhavalkar teaches;
wherein at each of the plurality of time steps, the attention sub network input comprises the encoded representation of the current observation and the encoded representations of the one or more previous observations characterizing the one or more previous states of the environment.
([0068] The output of the encoder 404 is processed using the attention module 406 to generate one or more context vectors c.sub.i for each output time step i... For each context output vector c.sub.i, the attention module 406 can compute attention based on the encodings for one or more input steps u, e.g., the encoding for the current input step as well as encodings for previous input steps. For example, the attention module 406 can generate an attention context output c.sub.i over the set of all the encoder outputs of the utterance, e.g., the entire set {h.sub.0, h.sub.1, h.sub.2, h.sub.3, h.sub.u}. The attention context vector c.sub.i can be a vector representing a weighted summary of the current and previous encodings for frames (e.g., portions) of the utterance being recognized.)
NOTE: Teaches at each of the plurality of time steps, the attention sub network input comprises the encoded representation of the current observation and the encoded representations of the one or more previous observations (the attention module input comprises current and previous encoded observations, each observation represented by a time step) characterizing the one or more previous states of the environment (the encoded observations represent states of the environment, as previously taught).
Regarding claim 17,
Claim 17 is substantially similar to claim 1, except for one limitation, which is taught by Prabhavalkar;
selecting an action to be performed by the agent based on the action selection output; and transmitting to the agent control data instructing the agent to perform the selected action.
([0075] In response to generating the transcription 420 using the speech recognition model 402, the electronic device 410 can perform any of various actions. For example, the electronic device 410 can analyze the transcription 420 to detect a hotword (e.g., wake word) or command in the utterance received from user 412. In some implementations, the electronic device 410 determines whether one or more predetermined commands are present in the transcription 420, and when the command is identified the electronic device performs an action corresponding to the identified command. For example, the system 400 can identify and execute a particular command (e.g., activate a virtual assistant, play a song, set a timer, add an item to a list, and so on), change an operating mode of the electronic device 410, send the transcription 420 as a request or query to a server, provide search results generated using the transcription 420 as a query, display the transcription 420 of the utterance, or enter the transcription 420 into a text area of a user interface (e.g., during a dictation mode).)
NOTE: Teaches selecting an action to be performed by the agent based on the action selection output (the transcription is used to select an action to be performed by the electronic device) and transmitting to the agent control data instructing the agent to perform the selected action (command data is transmitted to the device to perform a selected action).
The remaining limitations are taught using the same reasoning provided in claim 1.
Regarding claim 20,
Claim 20 is substantially similar to claim 2 and is rejected using the same reasoning.
Regarding claim 21,
Claim 21 is substantially similar to claim 4 and is rejected using the same reasoning.
Regarding claim 22, Prabhavalkar teaches;
Claim 22 is substantially similar to claim 5 and is rejected using the same reasoning.
Regarding claim 24,
Claim 24 is substantially similar to claim 9 and is rejected using the same reasoning.
Claim(s) 12, 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Jinhua Zhu et al. (hereinafter Zhu) (“MASKED CONTRASTIVE REPRESENTATION LEARNING FOR REINFORCEMENT LEARNING”, 2020-10-15), in view of Alexei Baevski et al. (hereinafter Baevski) (“wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” 22 Oct 2020), further in view of Ankesh Anand et al. (hereinafter Anand) (“Unsupervised State Representation Learning in Atari,” 5 Nov 2020).
Regarding claim 12, Zhu teaches;
A method of training an action selection neural network ([Abstract] The CNN encoder and Transformer are jointly trained) that is configured to generate an action selection output that is used to select an action to be performed by an agent interacting with an environment to perform a task ([Abstract] During inference, the CNN encoder and the policy network are used to take actions) the method comprising:
obtaining, from a replay buffer, a current observation ([pg. 4] We sample T consecutive observations from the replay buffer by chronological order. Denote the sampled observations as S = (s1,s2,··· ,sT)) that stores experience data generated as a result of the agent interacting with the environment ([pg. 3] take action at, get reward rt and get a new observation st+1); processing, using an encoder sub network of the action selection neural network, the current observation to generate an encoded representation of the current observation ([pg. 3] the CNN encoder … maps s ∈ S into a d-dimension representation … the encoded representation … fθ(st)); generating a masked input vector by: randomly selecting one or more input positions within a data structure (Note: input positions are selected to be masked randomly at a given probability: [pg. 4] sampled observations as S = (s1, s2,··· , sT). To leverage the masked training technique, define M = (M1, M2,··· , MT), where for each i ∈ [M], w.p. m, Mi = 1; w.p. 1−m, Mi =0. m ∈ [0,1] is a hyperparameter. If Mi = 1, ... si is modified as follows: w.p. 80%, the si is replaced with a zero vector; w.p. 10%, it is replaced by another sj randomly sampled from B; w.p. 10%, it remains unchanged. The modified observation is denoted as si. Let S= (s̄1, s̄2,··· , s̄T));
processing, using at least an attention sub network of the action selection neural network (ϕ consists of L identical blocks. Each block is stacked by two layers, including a self-attention layer attn(···)) the masked input vector ([pg. 4] leverage the masked training technique … The modified observation is denoted as s̄ i. Let S̄ = (s̄1, s̄2,··· , s̄T) … we get H0 = (h1,h2,··· ,hT) where hi = fθ(s̄i), s̄i ∈ S̄) to generate a prediction of the respective input value at each of the one or more randomly selected input positions ([pg. 4] ϕ is used to reconstruct the masked input by leveraging the global input … Let hli denote the output of i-th position of the l-th block, and Hl = (hl1, hl2, ···, hlT))
evaluating a contrastive learning objective function ([pg. 4] contrastive loss) that measures, for each of the one or more randomly selected input positions: a first difference between ([pg. 4] for any masked position, the reconstructed hLi (i.e., qi) should be similar to its original feature ki ) i) the prediction of the respective input value at the input position ([pg. 4] the reconstructed hLi (i.e., qi)) and ii) the respective input value in the encoded representation ([pg. 4] The key set K = (k1, k2,··· ,KT) … is encoded … ki = fθk (si), si ∈ S),
[pg. 4]
PNG
media_image3.png
23
708
media_image3.png
Greyscale
and a second difference between ([pg. 4] for any masked position, the reconstructed hLi (i.e., qi) should be … dissimilar to the others [kj]) i) the prediction of the respective input value at the input position ([pg. 4] the reconstructed hLi (i.e., qi)) …
[pg. 4]
PNG
media_image3.png
23
708
media_image3.png
Greyscale
and training the action selection neural network, based on a computed gradient of the contrastive learning objective function and a reinforcement learning loss ([pg. 4] The contrastive loss ... Lct [pg. 5] the RL training objective function … Lrl … The overall objective function of our approach is minθ,ω,ϕ Lall = Lrl + λLct,)
[pg. 14]
PNG
media_image4.png
598
913
media_image4.png
Greyscale
Zhu fails to teach but Baevski teaches;
generating a masked input vector by: randomly selecting one or more input positions ([pg. 3] mask the latent speech representations output by the encoder, we randomly sample … certain proportion p of all time steps) within a data structure representing the encoded representation of the current observation ([pg. 2] convolutional feature encoder f : X → Z which takes as input raw audio X and outputs latent speech representations z1, ..., zT for T time-steps), and applying a mask to a respective input value at each of the one or more randomly selected input positions ([pg. 3] then mask the subsequent M consecutive time steps from every sampled index)
OBVIOUSNESS TO COMBINE BAEVSKI:
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to modify Zhu such that the current observation is first processed by the encoder to generate and encoded representation of the current observation, and one or more input positions within the encoded representation are thereafter randomly selected and masked, as taught by Baevski, before the masked encoded representation is processed by the attention network. Zhu and Baevski both employ an encoder, masking, an attention/transformer network, and a contrastive learning objective for learning representations from input observations. Baevski further teaches generating, from an input, a position-indexed latent representation and randomly masking selected positions within that latent representation before contextual processing. On page 8, Baevski explains that continuous latent representations retain more information to “enable better context representations”. Accordingly, one of ordinary skill in the art would have been motivated to apply Baevski’s latent-position masking technique to Zhu so that Zhu’s attention network processes a masked input vector formed by masking selected input positions within the encoded representation of the current observation, thereby permitting contextual prediction from information retained in the encoded latent representation.
Zhu and Baevski fail to teach but Anand teaches;
difference between ([pg. 3] The local-local objective in equation 3 maximizes the mutual information between the local feature at time t with the corresponding local feature at time t + 1) i) ([pg. 4] φm,n(xt)) and ii) a respective input value at a corresponding input position ([pg. 4] (m,n) spatial location … φm,n(xt+1)) in an encoded representation ([pg. 4] φ is the representation encoder … fm,n(xt,xt+1) = φm,n(xt)TWlφm,n(xt+1)) of an augmented current observation (Note: Anand’s φm,n(xt+1) constitutes an augmented current observation in light of the applicant’s specification, which states; “the augmented current observation can be a future observation that characterizes a future state of the environment that is after the current state”)
OBVIOUSNESS TO COMBINE ANAND:
It would have been obvious to one of ordinary skill in the art, before the effective filing date, to further modify the combined system of Zhu and Baevski such that the prediction associated with a masked input position of an encoded current observation is compared with the respective encoded value at a corresponding position in an encoded representation of a subsequent observation, as taught by Anand. Zhu already seeks to exploit correlations among consecutive observations for improved representation learning, and Baevski teaches performing masked contrastive prediction at positions within an encoded latent representation. Anand similarly addresses representation learning from temporally related observations and teaches an objective that maximizes mutual information between a local encoded feature of a current observation and the corresponding local encoded feature of a subsequent observation. Additionally, Anand states that “[pg. 8] contrasting spatiotemporally (and not just temporally) is important across the board for capturing all categories of latent factors.” Accordingly, a skilled artisan would have been motivated to apply Anand’s teaching of corresponding position comparison across temporally related encoded observations to the position indexed latent representations of Zhu as modified by Baevski, thereby preserving positional correspondence when performing the contrastive comparison and capturing information shared across successive observations.
Regarding claim 14, Zhu and Baevski fail to explicitly teach but Anand teaches;
wherein the augmented current observation comprises a future observation characterizing a future state of the environment that is after the current state ([pg. 3] The local-local objective in equation 3 maximizes the mutual information between the local feature at time t with the corresponding local feature at time t + 1)
OBVIOUSNESS:
Using the same reasoning from claim 12
Regarding claim 15, Zhu teaches;
wherein the augmented [input position] ([pg. 1-2] CURL aims to learn the encoder in a self-supervised manner, where an input x is augmented (e.g., random crop)) or color space- transformed current observation.
Regarding claim 16, Zhu teaches;
processing the current observation ([pg. 3] consecutive observations as the input) using the action selection neural network having a plurality of action selection network parameters ([pg. 3] CNN encoder parameterized by θ … policy network πω parameterized by ω) to generate the action selection output ([pg. 3] to get the action … at); determining, based on the action selection output, the reinforcement learning loss (at is used in the RL loss, see below);
[pg. 4]
PNG
media_image5.png
140
1049
media_image5.png
Greyscale
and determining, based on the reinforcement learning loss, an update to current values of the action selection network parameters (see below).
[pg. 5]
PNG
media_image6.png
135
1005
media_image6.png
Greyscale
[pg. 14]
PNG
media_image7.png
413
1194
media_image7.png
Greyscale
Claim(s) 6, 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Prabhavalkar (US 20200043483 A1, 2020-02-06) as applied to claims 1-2, 5, 9, 17, 20, 22, 24 above, and further in view of Volodymyr Mnih et al. (hereinafter Mnih) (“Playing Atari with Deep Reinforcement Learning”, 2013-12-19).
Regarding claim 6, Prahavalkar fails to teach but Mnih teaches;
wherein the action selection output comprises a Q value for each of a possible set of actions that is an estimate of a return that would be received if the agent performed the action in response to the current observation.
[pg. 2]
PNG
media_image8.png
208
1046
media_image8.png
Greyscale
NOTE: Teaches wherein the action selection output comprises a Q value for each of a possible set of actions that is an estimate of a return that would be received if the agent performed the action in response to the current observation. The optimal action value function Q* produces the maximum expected return based on the action taken in response to the sequence.
OBVIOUSNESS TO COMBINE MNIH WITH PRAHAVALKAR:
Mnih is analgous art to the present disclosure as it pertains to an action selection network. Specifically, Mnih pertains to a deep learning model trained with a variant of Q-learning, to learn control policies from input. Mnih also states that a deep learning model trained using the disclosed variant of Q-learning has better action selection performance than all previous approaches and a human expert in some cases; “[Abstract] We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.” Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the variant of Q-learning disclosed by Mnih to train the model taught by Prahavalkar to improve action selection performance.
Regarding claim 23, Prahavalkar fails to teach but Mnih teaches;
Claim 23 is substantially similar to claim 6 and is rejected using the same reasoning.
Response to Arguments
Applicant’s arguments, filed 08/13/2026, with respect to 35 U.S.C. § 101 have been fully considered and are persuasive. The 35 U.S.C. § 101 abstract idea rejections of the claims have been withdrawn.
Applicant's arguments filed 08/13/2026, with respect to 35 U.S.C. § 102 and 103 have been fully considered but they are not persuasive. Starting on page 5, the applicant states that, “With respect to the rejection of claim 7, the Office Action acknowledges that Prabhavalkar fails to teach a gating layer and instead relies upon Parisotto as allegedly disclosing them. However, Parisotto states that "GTrXL offers an easy-to-train, simple-to-implement but substantially more expressive architectural alternative to the standard multi-layer LSTM ubiquitously used for RL agents in partially observable environments" (Parisotto, Abstract), and "the GTrXL architecture can function as a drop-in replacement to the LSTM networks ubiquitously used in RL" Parisotto, page 2). In other words, rather than disclosing an action selection neural network that includes both a gating layer and a recurrent sub network, and disclosing how to generate an input for the recurrent sub network using the gating layer, Parisotto discloses how to replace a recurrent neural network (e.g., LSTM network) using an alternative neural network that includes a gating layer. A prior art reference must be considered in its entirety, as a whole, including portions that would lead away from the claimed invention. See MPEP § 2141.02(VI); see also W.L. Gore & Associates, Inc. v. Garlock, Inc., 721 F.2d 1540 (Fed. Cir. 1983). A conclusion of non-obviousness is proper "when the prior art teaches away" from a claimed combination. KSR Int'l Co. v. Teleflex Inc., 127 S. Ct. 1727, 1740 (2007). Because Parisotto explicitly teaches "a drop-in replacement to the LSTM networks," the Office Action fails to explain how the proposed combination of Prabhavalkar and Parisotto would render "a gating layer configured, at each of the plurality of time steps, to receive i) the encoded representation of the current observation and ii) the attention sub network output, and to generate a recurrent sub network input by applying a gating mechanism to the encoded representation of the current observation and the attention sub network output," as recited by amended claim 1, as obvious.” Examiner respectfully disagrees. As reflected by the present office action, Prabhavalkar is relied upon for teaching an encoder subnetwork providing encoded representations of a current observation to an attention subnetwork used to generate an input for a recurrent subnetwork, while Parisotto is used to teach an attention based neural network architecture comprising a gating layer configured, at each of the plurality of time steps, to receive a representation of a current input and an attention sub network output, to generate a gated representation. In the proposed combination, the attention based neural network architecture of Parisotto would be used to modify Prabhavalkar’s attention subnetwork processing, and the resulting gated representation would be used for the recurrent subnetwork input. Parisotto’s statement that GTrXL “can function as a drop-in replacement” for LSTM networks describes one possible use of the GTrXL architecture; it does not criticize, discredit, or otherwise discourage employing Parisotto’s gated attention teaching in an architecture that retains a downstream recurrent subnetwork. Moreover, the proposed combination does not replace Prabhavalkar’s recurrent subnetwork with GTrXL. Rather, Parisotto’s teachings are applied to modify the attention processing of Prabhavalkar, while retaining Prabhavalkar’s recurrent subnetwork. Accordingly, Applicant’s teaching away argument is not persuasive. This understanding is also consistent with the Applicant’s specification, which itself identifies Parisotto when describing examples of configurations of the attention layers and other components of the attention subnetwork; “Examples of the configurations of the attention layers the specifics of the other components of attention sub network are described in more detail in … Parisotto, et al., Stabilizing transformers for reinforcement learning, arXiv:1910.06764, the entire contents of which are hereby incorporated by reference herein in their entirety.”
The applicant further states; “Moreover, claim 1 as amended requires a gating layer to apply "a gating mechanism to ... the attention sub network output," where the "attention sub network compris[es] one or more multi-head attention layers and a position-wise multi-layer perceptron." Because the cited portion of Parisotto only teaches a gating layer function interposed between a multi-head attention block and a multi-layer perception of a "GTrXL layer block" (see equations (5)-(8) of Parisotto), Applicant respectfully submits that the Office Action has not shown that Parisotto teaches what happens outside of the attention sub network, e.g., what happens to the output of the attention sub network. Indeed, the cited portion of Parisotto only teaches how to generate the output of the GTrXL layer block by applying a gating layer function within the GTrXL layer block. The Office Action does not assert that Zhu, Mnih, or Schwarzer discloses these features. Accordingly, Applicant respectfully submits that claim 1 and its dependent claims are in condition for allowance. Independent claim 17 and its dependent claims are allowable for corresponding reasons.” Examiner respectfully disagrees. Amended claim 1 requires an attention sub network comprising one or more multi-head attention layers and a position-wise multi-layer perceptron, and that the attention sub network output is generated “at least in part by applying an attention mechanism using the one or more multi-head attention layers.” Parisotto teaches an output generated by the multi-head attention operation, Ȳ(l), and further teaches applying a gating mechanism to that output and the input representation E(l−1). Applicant’s argument that Parisotto does not disclose what occurs “outside of the attention sub network” also does not address the rejection as a whole. The rejection does not rely on Parisotto alone for the entire claimed architecture. Prabhavalkar teaches an encoder subnetwork providing an encoded representation to an attention subnetwork, and further teaches providing the resulting attention representation as input to a recurrent subnetwork, while Parisotto is relied upon to modify the attention subnetwork. In the proposed combination, the gated representation resulting from Parisotto’s gating mechanism is provided as the recurrent subnetwork input taught by Prabhavalkar. Accordingly, the combined teachings of Prabhavalkar and Parisotto teach the recited arrangement and claim 1 and its dependent claims are not in condition for allowance under 35 USC 103. Independent claim 17 and its dependent claims are not allowable for corresponding reasons.
The applicant further states; “that the cited portion of Zhu does not disclose or suggest this combination of features of amended claim 12. With respect to the rejection of claim 13, the Office Action cites a portion of Zhu that teaches "We sample T consecutive observations from the replay buffer by chronological order. Denote the sampled observations as S = (s1, s2, - - - , sT )... si is modified as follows: w.p. 80%, the si is replaced with a zero vector; w.p. 10%, it is replaced by another sj randomly sampled from B; w.p. 10%, it remains unchanged," at 3.2 ALGORITHM. While the cited portion of Zhu discloses replacing a sampled observation si with a zero vector, it does not disclose or suggest what to do with individual "input positions within a data structure representing the encoded representation of the current observation." In particular, the cited portion of Zhu does not disclose or suggest "randomly selecting one or more input positions within a data structure representing the encoded representation of the current observation." Nor does it disclose or suggest "applying a mask to a respective input value at each of the one or more randomly selected input positions." Nor does it disclose or suggest generating "a prediction of the respective input value at each of the one or more randomly selected input positions." The Office Action does not assert that Schwarzer discloses these features. Accordingly, Applicant respectfully submits that claim 12 and its dependent claims are in condition for allowance.” However, as necessitated by the amendments, the present rejection does not rely on Zhu alone for these limitations. Rather, Baevski is relied upon to teach first encoding an input observation into a position indexed latent representation and then randomly selecting and masking positions within that encoded representation. Thus, Baevski, rather than Zhu, supplies the claimed selection and masking of positions within a data structure representing the encoded representation of a current observation. Zhu is relied upon for the remaining mask prediction framework, including processing masked inputs using the attention/transformer network and generating reconstructed outputs corresponding to masked positions. As modified by Baevski, Zhu’s masked-prediction technique is applied to positions selected within the encoded latent representation rather than to entire observations selected before encoding. Accordingly, the combination of Zhu and Baevski teaches randomly selecting positions within an encoded representation of the current observation, masking the respective encoded values, and generating predictions corresponding to the selected masked positions. Accordingly, claim 12 and its dependent claims are not in condition for allowance under 35 USC 103.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CESAR B PAULA/ Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights