DETAILED ACTION
This Office Action is in response to communications filed on March 3rd, 2026 for Application No. 18/076,978, in which claims 1-20 are presented for examination. The amendments filed March 3rd, 2026 have been entered, where claims 1-3 and 12-20 are amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: “214” (Para. [0098]). Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
The drawings are also objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: “116” in Fig. 1. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1-20 are objected to because of the following informalities:
“immmediate” (Claim 1, ln. 30; Claim 19, ln. 34; Claim 20, Ln. 36) should be “immediate” (objection applies equally to dependent claims 2-18).
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6-7, 11-15, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer et al. (hereinafter Shazeer) (“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer”) in view of Bengio et al. (hereinafter Bengio) (“Conditional computation in neural networks for faster models”) and Liu et al. (hereinafter Liu) (“Generative Question Refinement with Deep Reinforcement Learning in Retrieval-based QA System”).
Regarding Claim 1, Shazeer teaches a method performed by one or more data processing apparatus for training a task neural network having one or more conditional computational layers, wherein each conditional computation layer comprises (i) a gating sub-layer comprising a plurality of gating parameters and (ii) an expert sub-layer comprising a plurality of expert neural networks, to perform a machine learning task, the method comprising (Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the “approach to conditional computation . . . [using a] general purpose neural network” that is “trainable” is the method for training a neural network, and where the “Gated Mixture-of-Experts Layer” is a “conditional computation” layer, which comprises a gating sub-layer, “The MoE consists of . . . a trainable gating network”, and an expert sub-layer with a plurality of expert neural networks, “The MoE consists of a number of experts, each a simple feed-forward neural network”; see also Pg. 2, Fig. 1, where both the “Gating network” and “Expert[s]” can reasonably be described as sub-layers; Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where the method is used to perform machine learning “tasks of language modeling and machine translation”, and, as a result, the “neural network” with the “general purpose neural network component” is a task neural network; Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”, where the “gating function” has a “trainable weight matrix” of a plurality of gating parameters; and Pg. 7, Para. 1-2, “table 1 compares the results of these models to the best previously-published result on this dataset . . . Computational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs”, where a “dataset” processing apparatus, “on clusters containing 16-32 Tesla K40 GPUs”, executed the method):
sampling a batch of training examples, wherein each training example includes a network input and a respective target output sequence (Pg. 15, Para. 2, “Training: The models were trained on . . . [batches of training data.] Each batch consisted of a set of sentences totaling roughly 300,000 words”, where, at least but not necessarily exclusively in regard to the “machine translation” model, “each batch” of training examples, includes a network input, “source”, and a respective target output sequence, “sentence[s]” of “target vocabulary of 32K wordpieces”, see Pg. 17, Para. 2-6, “MACHINE TRANSLATION - EXPERIMENTAL DETAILS . . . We use a shared source and target vocabulary of 32K wordpieces . . . Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU”)
that comprises a respective ground truth output token at each of a plurality of output positions, wherein each ground truth output token is selected from a vocabulary of output tokens (Pg. 17, Para. 2, “Similar to GNMT, to effectively deal with rare words, we used subword units (also known as “wordpieces") (Schuster & Nakajima, 2012) for inputs and outputs in our system”, where “wordpieces” are “output” tokens from “tokenized” “target” “sentences” of the “target vocabulary of 32K wordpieces”, see Pg. 17, Para. 8, “We evaluated our models using the perplexity and the standard BLEU score metric. We reported tokenized BLEU score as computed by the multi-bleu.pl script” and Pg. 17, Para. 2-6, “MACHINE TRANSLATION - EXPERIMENTAL DETAILS . . . We use a shared source and target vocabulary of 32K wordpieces . . . Each training batch consisted of a set of sentence pairs containing roughly 16000 words per GPU”, which in regard to the “target . . . workpieces” for “training” includes data that is within the broadest reasonable interpretation of the ground truth, the value “pair[ed]” with the “input”, for each of a plurality of output positions, see Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”);
for each training example, processing the network input using the task neural network to generate a network output that includes, for each of the plurality of output positions in the target output sequence, a respective score distribution over the vocabulary of output tokens (Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”; see also Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a model output with a plurality of output positions; Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”, which as discussed above, is the task neural network; Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text”), comprising:
for each of the one or more conditional computation layers: receiving a layer input sequence for the conditional computation layer that is generated from at least the network input and that comprises a respective layer input for each of the plurality of output positions (Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the conditional computational layer, “the MoE” layer, receives a layer input sequence “a given input x” as layer input for “each position in the text” , which is the network input, and generates an output, “output y”, for each of these “position[s] in the text”, see Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost”);
processing each layer input of the layer input sequence using the gating sub-layer and in accordance with current values of the gating parameters to generate a respective set of gating scores for each layer input (Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the “gating network” sub layer generates a set of gating scores, which are used in order to determine “G(x)i” for “i” “expert[s]”, for each layer input, “a given input x”, where the “gating function” processes inputs in accordance with a “trainable weight matrix” current values of the gating parameters, see Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”, and where each layer input of the layer input sequence, “The MoE is called once for each position in the text”, is processed using the gating sub-layer, see Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”);
for each layer input: selecting an expert neural network from the plurality of expert neural networks in the expert sub-layer based at least in part on the respective set of gating scores for the layer input (Pg. 2, Fig. 1, “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations”, where the arrows entering the “MoE layer[s]” represent the layer input and “select[ing] two experts” requires selecting an expert from the plurality of experts, “Expert 1” to “Expert n”, in the expert sub-layer, “MoE layer”; see also Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example”, where the “experts are themselves neural networks” and the respective gating scores for the layer input determines “G(x)”, which determines which experts are selected “We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”); and
processing the layer input using the respective selected expert neural network to generate a respective expert output for the layer input (Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the layer input, the “given input x”, is processed to generate the respective expert output for the layer input, “The output y of the MoE module”, and where “Wherever G(x)i = 0, we need not compute Ei(x)”); and
generating a layer output sequence for the conditional computation layer from the expert outputs for the layer inputs (Pg. 2-3, Para. 4-1, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE) . . . The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position . . . On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost”, where the layer output sequence for the “conditional computation” layer is the output if the “MoE” for “each position in the text”, which “select[s] a potentially different combination of experts at each position” to process the layer inputs);
for each gating sub-layer and for each output position in each of the target output sequences, generating . . . [a gradient] for the gating sub-layer for the output position from at least a respective score assigned to the ground truth output token at the output position by the score distribution generated by the task neural network for the output position (Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where, as discussed above, a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a score distribution for the respective outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which, in combination with a required respective score assigned to the ground truth output tokens at each output token, determine the “Gradients” that are “input” into the gating sub-layer during “back-propagation”, see Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model . . . Gradients also backpropagate through the gating network to its inputs”; see also Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text” to generate the corresponding output for each position of the target output sequence; for more information see Pg. 15, Para. 2, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016)”); and
training each of the gating sub-layers using . . . [gradients] for the gating sub-layer for the output positions through . . . [back-propagation] to optimize a . . . [gradient] received by the gating sub-layer (Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network . . . Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network”),
wherein . . . for each output position in each of the target output sequence, . . . [a gradient is generated] for the output position that depends on the respective score assigned to the ground truth output token at the output position by the score distribution generated by the task neural network for the output position (Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where, as discussed above, a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a score distribution for the respective outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which, in combination with a required respective score assigned to the ground truth output tokens at each output token, determine the “Gradients” that are “input” into the gating sub-layer during “back-propagation”, see Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model . . . Gradients also backpropagate through the gating network to its inputs”; see also Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text” to generate the corresponding output for each position of the target output sequence; for more information see Pg. 15, Para. 2, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016)”)
and . . . for each future output position that follows the output position in the target output sequence that depend on a respective score assigned to the ground truth output token at the future output position by the score distribution generated by the task neural network for the future output position (Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where, as discussed above, a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a score distribution for the respective outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which, in combination with a required respective score assigned to the ground truth output tokens at each output token, where any output sequence corresponding to more than a single token, such as an output associated with “batch sizes [that] are approximately 2.5 million words”, see Pg. 16. Para. 4, “For all models, training batch sizes are approximately 2.5 million words. Models are trained once-through over about 100 billion words”, will include an output position and future output positions, all of which are used to determine the “Gradients” that are “input” into the gating sub-layer during “back-propagation”, see Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model . . . Gradients also backpropagate through the gating network to its inputs”; see also Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text” to generate the corresponding output for each position of the target output sequence; for more information see Pg. 15, Para. 2, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016)”; see also Pg. 5, Para. 2, “we can apply the MoE to all the time steps together as one big batch. Doing so increases the size of the input batch to the MoE layer by a factor of the number of unrolled time steps”, where application of the “MoE” occurs “together as one big batch”).
Shazeer does not explicitly disclose . . . an immmediate reward . . . the respective immediate rewards . . . reinforcement learning . . . reinforcement learning objective function that includes one or more terms that measure expected rewards . . . the expected rewards comprise . . . a expected reward generated from (i) the immediate reward . . . (ii) respective future immediate rewards . . . (where reward-based reinforcement learning is not specifically described as part of the training process, and, as a result, expected rewards comprising an immediate reward for an output position and future immediate rewards for future output positions are also not specifically described).
However, Bengio teaches . . . [generating] an immmediate reward [for gating functionality based on a respective score assigned to the ground truth output and the score distribution generated by the neural network for the output] . . . [training the gating functionality using] the immediate respective rewards [for the output through] . . . reinforcement learning . . . (Pg. 11-12, Para. 6-6, “In reinforcement learning, a sequence of state-action-reward tuples is described as a trajectory . . . the reward of the trajectory is the neural network cost C(x)”, where the “reward” is used train the “neural network” through “reinforcement learning”; Pg. 2, Para. 2-6, “We propose to learn input-dependent activation probabilities for every node (or blocks of nodes), while trying to jointly minimize the prediction errors at the output and the number of participating nodes at every layer, thus reducing the computational load . . . Our model consists in a typical fully-connected neural network model, joined with stochastic per layer policies that activate or deactivate nodes of the neural network in an input-dependent manner, both at train and test time . . .We train a different policy for each layer l”, where the “train[ing]” includes “learning” “per layer policies that activate or deactivate nodes”, which are used to implement gating functionality; see also Pg. 12, Para. 6, “the reward of the trajectory is the neural network cost C(x)”, where the “reward” is the current “neural network cost”, which is the “loss” of the model’s output, calculated on “per layer” basis and based on the score distribution generated by the neural network for the output compared to the ground truth value assigned to the ground truth output , and, as a result, the each reward in the respective rewards are within the broadest reasonable interpretation of immediate, see Pg. 2, Para. 5, “The cost C is the loss of the neural network architecture (in our case the negative log-likelihood)”)
[a] reinforcement learning objective function that includes one or more terms that measure an expected reward [received by the gating functionality] . . . (Pg. 11, Para. 6, “In reinforcement learning, a sequence of state-action-reward tuples is described as a trajectory t. The objective function of a parameterized policy πθ for the cumulative return of a trajectory t is described as:
PNG
media_image1.png
98
326
media_image1.png
Greyscale
”, where “J(θ)” is the “reinforcement learning” “objective function” and “r(St, At|S0=s0)” are the one or more terms that measure the expected reward “
E
t
π
θ
”, which, as discussed above, is received by the gating functionality, see Pg. 2, Para. 2-6, “We propose to learn input-dependent activation probabilities for every node (or blocks of nodes), while trying to jointly minimize the prediction errors at the output and the number of participating nodes at every layer, thus reducing the computational load . . . Our model consists in a typical fully-connected neural network model, joined with stochastic per layer policies that activate or deactivate nodes of the neural network in an input-dependent manner, both at train and test time . . .We train a different policy for each layer l”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the generating of a gradient for the gating sub-layer based on the score distributions of the output positions generated by the neural network and respective ground truth scores for the output tokens, where the gating sub-layer is trained using the gradients of the output positions through back-propagation to optimize the gradient received by the gating sub-layer of Shazeer with the generating of an immediate reward for training a gating functionality through reinforcement learning, where the immediate reward is based on a ground truth score and the score output by a neural network and the reinforcement learning includes an objective function where one or more terms measure are an expected reward received by the gating functionality during training of Bengio in order to intelligently optimize the conditional computation functionality of the neural network (Bengio, Pg. 1, Para. 3, “Conditional computation refers to activating only some of the units in a network . . . one needs to be able to decide in an intelligent fashion”; Bengio, Pg. 1, Abstract, “In this paper, we use reinforcement learning as a tool to optimize conditional computation policies”) in a manner that optimizes speed while maintaining prediction accuracy (Bengio, Pg. 1, Abstract, “More specifically, we cast the problem of learning activation-dependent policies for dropping out blocks of units as a reinforcement learning problem. We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy”), which solves a prevalent problem in supervised learning (Bengio, Pg. 1-2, Para. 2-3, “Large-scale neural networks, and in particular deep learning architectures, have seen a surge in popularity in recent years, due to their impressive empirical performance in complex supervised learning tasks . . . Yet the task of training such networks remains a challenging optimization problem . . . our solution to the proposed optimization problem”), and was discussed as a viable alternative approach in Shazeer (Shazeer, Pg. 4, Para. 3, “Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network”).
Additionally, Liu teaches . . . [a reinforcement learning objective function that includes one or more terms that measure expected rewards, wherein the] expected rewards comprise[,] (Pg. 1643, Col. 1, Abstract, “we train the model with deep reinforcement learning techniques that consider an appropriate wording of the generation as an immediate reward and the correlation between generated question and answer as time-delayed long-term rewards” and Pg. 1646-1647, Col. 2-1, Para. 5-4, “The question y ~ πθ (·|x) is generated according to πθ where θ is the policy’s parameter and the goal is to maximize the expected reward of the reformulated question under the policy, Ey~πθ (·|x) [R(y)] . . . The final objective is: Ey~πθ (·|x) [R(y) – b(x)] + λH[πθ (y|x)]”, where a “reinforcement learning” “objective” function includes one or more terms, “Ey~πθ (·|x) [R(y)]”, that measure “expected rewards, see Pg. 1647, Col. 2, Para. 1, “the expected rewards”)
[for each output position in each of the target output sequence] a expected reward generated from (i) the immediate reward [for the output position and] (ii) respective future immediate rewards [for each future output position that follows the output position in the target output sequence] (Pg. 1646, Col. 1, Para. 2, “Accumulated Reward . . . The Qrefine reward r of each word is the combination of the wording reward and the answer correlation reward,
PNG
media_image2.png
76
402
media_image2.png
Greyscale
. . . between wording reward rlm and answer correlation reward rac; Since we want Qrefine module to have the ability to infer reward even if not reaching the end of the generation process and the future reward will influence the current reward, we adopt the accumulated Qrefine reward R with the discounted factor ỿ and the accumulated Qrefine discounted reward of t-th word is represented as,
PNG
media_image3.png
46
544
media_image3.png
Greyscale
By using the accumulated reward, we are able to infer the answer correlation reward even if we are not reaching the end of the generation process”, where for each output position, “of each word”, in each of the target output sequences, see Pg. 1644, Col. 2, Para. 3, “Given an ill-formed question consists of x = [x1,x2, …, xn] of an arbitrary-length N, the well-formed question y = [y1,y2, …, yn] of a variable-length M. The aim of question refinement is to refine x to y”, the expected reward is generated from “rw” for the output position, “r(yt)”, which is an immediate reward, see Pg. 1645, Col. 2, Para. 4, “wording reward rw aims to give an immediate reward to each of words”, which is “influence[d]” by “future reward[s]” for each future output position that follows the output position, “t+1” to “M-t” in the target output sequence, “M”, which are within the broadest reasonable interpretation of future immediate rewards, such that the “expected reward” is generated from the immediate reward and respective immediate future rewards, Pg. 1646, Col. 2-, Para. 5, Equation 9; see generally Pg. 1646, Col. 1, Para. 1, “We train the model by maximizing answer correlation reward rac using ground-truth” and Pg. 1645, Col. 1, Para. 1, “the Seq2Seq model is formed as . . . softmax”)
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the training of gating sub-layers in a task neural network, wherein a reinforcement learning objective function that includes one or more terms that measure an expected reward is used to generate immediate rewards such that the rewards for an output position and each subsequent future output position that follows the output position in a target sequence depend on a respective score assigned to a ground truth output token at the respective position by a score distribution generated by the task neural network of Shazeer in view of Bengio with the reinforcement learning objective function that includes one or more terms that measure expected rewards, wherein the expected rewards comprise, for each output position in each of a target output sequence, an immediate reward for the output position and respective future immediate rewards for each future output position that follows the output position in the target output sequence of Liu in order to infer the answer correlations between words in a sequence during the generation process (Liu, Pg. 1646, Col. 1, Para. 2, “Since we want Qrefine module to have the ability to infer reward even if not reaching the end of the generation process and the future reward will influence the current reward, we adopt the accumulated Qrefine reward R with the discounted factor ỿ and the accumulated Qrefine discounted reward of t-th word is represented as,
PNG
media_image3.png
46
544
media_image3.png
Greyscale
By using the accumulated reward, we are able to infer the answer correlation reward even if we are not reaching the end of the generation process”), which will contribute to a task neural network with increased accuracy (see Liu, Pg. 1643, Col. 1, Abstract, “Experimental results on real-world datasets show that the proposed Qrefine can generate refined questions with high readability but fewer mistakes than original questions provided by users. Moreover, the refined questions also significantly improve the accuracy of answer retrieval”), by improving outputs in a controllable manner (see Liu, Pg. 1652, Col. 1, Para. 1, “We improve the question representation by incorporating character embedding and contextual word embedding such as BERT. To make the refinement process more controllable, we combine Seq2Seq model with deep reinforcement learning. We define a sequence generator by optimizing for a combination of imposed reward functions”).
Regarding Claim 2, Shazeer in view of Bengio and Liu teach the method of claim 1, further comprising training the selected experts on a supervised learning objective (Shazeer, Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model”, where the “train[ing]” is on a supervised learning objective, see Shazeer, Pg. 8, Para. 4-5, “Machine Translation . . . Datasets: We benchmarked our method on the WMT’14 En->Fr and En->De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively”, where a person of ordinary skill in the art would understand “training” a model for a “Machine Translation” task using “sentence pairs” to involve comparing the model output to its “paired” label value, which is within the broadest reasonable interpretation of training on a supervised learning objective; see generally Bengio Pg. 1, Para. 2, “Large-scale neural networks, and in particular deep learning architectures, have seen a surge in popularity in recent years, due to their impressive empirical performance in complex supervised learning tasks”)
that measures for each output position in each training network output an error between (i) the score distribution generated by the task neural network for the output position and (ii) a respective ground truth score distribution based on the ground truth output token at the output position (Shazeer, Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where, as discussed above, a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing (i) a score distribution for the respective outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which, in combination with a required (ii) respective score assigned to the ground truth output tokens at each output token, determine the “Gradients” that are “input” into the gating sub-layer during “back-propagation”, see Shazeer, Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model . . . Gradients also backpropagate through the gating network to its inputs”, which, in view of Bengio, includes a measure “network cost”, which is the “error”, between the (i) “prediction . . . at the output” and the (ii) required ground truth, as represented by the “loss”, see Bengio, Pg. 11-12, Para. 6-6, “In reinforcement learning, a sequence of state-action-reward tuples is described as a trajectory . . . the reward of the trajectory is the neural network cost C(x)”; Bengio, Pg. 2, Para. 2-6, “We propose to learn input-dependent activation probabilities for every node (or blocks of nodes), while trying to jointly minimize the prediction errors at the output”; and Bengio, Pg. 2, Para. 5, “The cost C is the loss of the neural network architecture (in our case the negative log-likelihood)”; see also Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Shazeer, Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text” to generate the corresponding output for each position of the target output sequence; for more information see Shazeer, Pg. 15, Para. 2, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016)”)
by backpropagating gradients of the supervised learning objective through the task neural network (Shazeer, Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model . . . Gradients also backpropagate through the gating network to its inputs”, where, as discussed above, the training is supervised, see Shazeer, Pg. 8, Para. 4-5, “Machine Translation . . . Datasets: We benchmarked our method on the WMT’14 En->Fr and En->De corpora, whose training sets have 36M sentence pairs and 5M sentence pairs, respectively”; see generally Bengio Pg. 1, Para. 2, “Large-scale neural networks, and in particular deep learning architectures, have seen a surge in popularity in recent years, due to their impressive empirical performance in complex supervised learning tasks”).
The reasons for obviousness were discussed in regard to the rejection of Claim 1 above and remain applicable here.
Regarding Claim 6, Shazeer in view of Bengio and Liu teach the method of claim 1, wherein selecting an expert neural network from the expert sub- layer based at least in part on the respective set of gating scores for each layer input comprises (Shazeer, Pg. 2, Fig. 1, “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations”, where the arrows entering the “MoE layer[s]” represent the layer input and “select[ing] two experts” requires selecting an expert from the plurality of experts, “Expert 1” to “Expert n”, in the expert sub-layer, “MoE layer”; see also Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example”, where the “experts are themselves neural networks” and the respective gating scores for the layer input determines “G(x)”, which determines which experts are selected “We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”):
processing the respective sets of gating scores for the layer inputs using a gating function to generate a respective set of assignation scores for each layer input (Shazeer, Pg. 4, Para. 2, “Noisy Top-K Gating: . . .
PNG
media_image4.png
202
704
media_image4.png
Greyscale
”, where, in the context of “Noisy Top-K Gating”, the respective sets of gating scores for the layer inputs, “H(x)i”, are processed by the “gating function” to generate the respective sets of assignation scores for each layer input, “G(x)i”, see Shazeer, Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the “gating network” sub layer generates a set of gating scores, “H(x)”, in order to determine the set of assignation scores, “G(x)i” for “i” “expert[s]”, for each layer input, “a given input x”, where the “gating function” processes inputs in accordance with a “trainable weight matrix” current values of the gating parameters, and Shazeer, Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”); and
selecting an expert neural network for each layer input based at least in part on the respective assignation scores for the layer input (Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where, as discussed above, the assignation scores, “G(x)i”, determined by the gating scores, “H(x)i”, are used to select an expert for each layer, “Wherever G(x)i = 0, we need not compute Ei(x)”).
Regarding Claim 7, Shazeer in view of Bengio and Liu teach the method of claim 6, wherein selecting an expert neural network for each layer input based at least in part on the respective assignation scores for the layer input comprises (Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where, as discussed above, the assignation scores, “G(x)i”, determined by the gating scores, “H(x)i”, are used to select an expert for each layer, “Wherever G(x)i = 0, we need not compute Ei(x)”):
selecting the expert neural network for the layer input corresponding to the largest assignation score for the layer input (Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”; where the largest assignation score will be associated with the “top k values” and thus will be selected because “G(x)i” will not be “equal to 0” for at least the largest two assignation values, “If we choose k > 1”, see Shazeer, Pg. 4, Para. 2-3, “We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to –[infinity] (which causes the corresponding gate values to equal 0) . . . If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network”).
Regarding Claim 11, Shazeer in view of Bengio and Liu teach the method of claim 1, wherein for each of the one or more expert sub-layers, the plurality of expert neural networks in the expert sub-layer are distributed across a plurality of respective computational devices (Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the “Gated Mixture-of-Experts Layer” is a “conditional computation” layer, which comprises an expert sub-layer with a plurality of expert neural networks, “The MoE consists of a number of experts, each a simple feed-forward neural network”; Shazeer, Pg. 4, Para. 5, “In a conventional distributed training setting, multiple copies of the model on different devices asynchronously process distinct batches of data, and parameters are synchronized through a set of parameter servers. In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert . . . The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts)”, where the plurality of “expert” neural networks in the expert sub layer are “shared” across a plurality of computational “devices”, which respectively host “a subset of the experts” that is “distributed” across the network; see also Shazeer, Pg. 7, Para. 2, “Computational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs”).
Regarding Claim 12, Shazeer in view of Bengio and Liu teach the method of claim 1, further comprising using the task neural network to perform the machine learning task after the task neural network has been trained to perform the machine learning task (Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires using the model trained on a machine learning task to perform a machine learning task to evaluate the its performance, which as discussed above is the task neural network, see also Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the trained model is a “neural network”).
Regarding Claim 13, Shazeer in view of Bengio and Liu teach the method of claim 12, wherein after the task neural network has been trained to perform the machine learning task, performing the machine learning task comprises (Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires using the model train on a machine learning task to perform a machine learning task to evaluate the its performance, see also Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the trained model is a “neural network”)
processing a new network input using the task neural network to generate a new network output that includes for each of a new plurality of output positions in the new network output a respective new score distribution over the vocabulary of output tokens (Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where the method is used to perform machine learning “tasks of language modeling and machine translation”, which requires processing new input data to generate a new plurality of output data; see also Shazeer, Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a new respective score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which uses the trained model to perform the method during the “evaluat[ion]” phase; Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a task “neural network”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the new input, “source”, and new output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Shazeer, Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text”), comprises:
for each of the one or more conditional computation layers: receiving a new layer input sequence that is generated from the new network input for the conditional computation layer and that comprises one or more layer inputs (Shazeer, Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the conditional computational layer, “the MoE” layer, receives a new layer input sequence “a given input x” as layer input for “each position in the text” , which is the new network input, and generates an output, “output y”, for each of these “position[s] in the text”, see Shazeer, Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position. The different experts tend to become highly specialized based on syntax and semantics (see Appendix E Table 9). On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost” see also Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the processing steps occur during both training and evaluation);
processing each layer input of the new layer input sequence using the gating sub- layer and in accordance with current values of the gating parameters to generate a respective new set of gating scores (Shazeer, Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the “gating network” sub layer generates a set of new gating scores, which are used in order to determine “G(x)i” for “i” “expert[s]”, for each layer input, “a given input x”, where the “gating function” processes inputs in accordance with a “trainable weight matrix” current values of the gating parameters, see Shazeer, Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”, and where each layer input of the layer input sequence, “The MoE is called once for each position in the text”, is processed using the gating sub-layer, see Shazeer, Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” see also Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the processing steps occur during both training and evaluation);
for each layer input of the new layer input sequence: selecting an expert neural network from the expert sub-layer based at least in part on the respective new set of gating scores for the layer input (Shazeer, Pg. 2, Fig. 1, “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations”, where the arrows entering the “MoE layer[s]” represent the layer input and “select[ing] two experts” requires selecting an expert from the plurality of experts, “Expert 1” to “Expert n”, in the expert sub-layer for the new layer input sequence, “MoE layer”; see also Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example”, where the “experts are themselves neural networks” and the respective new gating scores for the layer input determines “G(x)”, which determines which experts are selected “We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”; and Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the selection steps occur during both training and evaluation);
processing the layer input using the respective selected expert neural network to generate a respective new expert output for the layer input (Shazeer, Pg. 3, Para. 6-7, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [Equation 1.] We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where the layer input, the “given input x”, is processed to generate the new respective expert output for the layer input, “The output y of the MoE module”, and where “Wherever G(x)i = 0, we need not compute Ei(x)”; see also Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the processing steps occur during both training and evaluation); and
generating a new layer output for the conditional computation layer from the expert outputs for the one or more layer inputs (Shazeer, Pg. 2-3, Para. 4-1, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE) . . . The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position . . . On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost”, where the new layer output sequence for the “conditional computation” layer is the output of the “MoE” for “each position in the text”, which “select[s] a potentially different combination of experts at each position” to process the layer inputs; see also Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the processing steps occur during both training and evaluation).
Regarding Claim 14, Shazeer in view of Bengio and Liu teach the method of claim 13, wherein selecting an expert neural network from the expert sub-layer based at least in part on the respective new set of gating scores for each layer input comprises (Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”, where, as discussed above, the new scores, “G(x)i”, determined by the gating scores, “H(x)i”, are used to select an expert for each layer, “Wherever G(x)i = 0, we need not compute Ei(x)”):
selecting the expert neural network for each layer input corresponding to the largest gating score in the respective new set of gating scores for the layer input (Shazeer, Pg. 3, Para. 5-7, “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1, . . . , En, and a “gating network" G whose output is a sparse n-dimensional vector . . . The experts are themselves neural networks . . . We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x)”; where the largest gating score in the respective new set of gating scores will be in the “top k values”, “If we choose k > 1”, and thus, it’s associated new score, “G(x)i” will not be “equal to 0”, see Shazeer, Pg. 4, Para. 2-3, “We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to –[infinity] (which causes the corresponding gate values to equal 0) . . . If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network”).
Regarding Claim 15, Shazeer in view of Bengio and Liu teach the method of claim 13, further comprising, for each output position in the new network output: selecting an output token for the output position from the vocabulary of output tokens in accordance with the respective score new distribution for the output position (Shazeer, Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective new score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”; see also Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a new model output with a plurality of output positions, where an output token is selected in in accordance with the respective score distribution for the output position, see Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016)”, where a person of ordinary skill in the art would understand “beam search” to be a selection technique based on the score values, at least within the context of generating outputs for “language modeling and machine translation”, see Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation”; Shazeer, Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”, which will be selected based on the score generated for the output position; Shazeer, Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text”).
Regarding Claim 19, Shazeer teaches a method performed by one or more data processing apparatus, the method comprising: processing a network input using a task neural network to generate a network output that includes for each of a plurality of output positions in the network output a respective score distribution over the vocabulary of output tokens (Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where the method is used to perform machine learning “tasks of language modeling and machine translation”, which requires processing input data to generate a plurality of output data; see also Pg. 7, Para. 1-2, “table 1 compares the results of these models to the best previously-published result on this dataset . . . Computational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs”, where a “dataset” processing apparatus, “on clusters containing 16-32 Tesla K40 GPUs”, executed the method; Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”, which uses the trained model to perform the method during the “evaluat[ion]” phase; Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”; Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text”),
wherein the task neural network includes one or more conditional computational layers, wherein each conditional computation layer comprises (i) a gating sub-layer comprising a plurality of gating parameters and (ii) an expert sub-layer comprising a plurality of expert neural networks (Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the “Gated Mixture-of-Experts Layer” is a “conditional computation” layer, which comprises a gating sub-layer, “The MoE consists of . . . a trainable gating network”, and an expert sub-layer with a plurality of expert neural networks, “The MoE consists of a number of experts, each a simple feed-forward neural network”; see also Pg. 2, Fig. 1, where both the “Gating network” and “Expert[s]” can reasonably be described as sub-layers; and Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”, where the “gating function” has a “trainable weight matrix” of a plurality of gating parameters), and
wherein the task neural network has been trained by performing operations comprising . . . (Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1)”, where the “approach to conditional computation . . . [using a] general purpose neural network” that is “trainable” is training the “neural network”).
The remaining limitations are substantially the same as limitations of Claim 1, therefore it is rejected under the same rationale.
Regarding Claim 20, Shazeer teaches a system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: . . . (Pg. 7, Para. 2, “Computational Efficiency: We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs”, where the “clusters containing 16-32 Tesla K40 GPUs” is a system comprising one or more computers; see also Pg. Pg. 7, Table 1, “Summary of high-capacity MoE-augmented models with varying computational budgets” and Pg. 9, Para. 4, “We carefully identified the design considerations and challenges of conditional computing and addressed them with a combination of algorithmic and engineering solutions”, where the “clusters containing 16-32 Tesla K40 GPUs” must execute a set of instructions stored in a storage device in order to implement the “algorithmic” “conditional computing” to generate predictive results, as demonstrated by the “Test Perplexity” metric in Table 1).
The remaining limitations are substantially the same as limitations of Claim 19, therefore it is rejected under the same rationale.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Kaplan et al. (hereinafter Kaplan) (“Beating Atari with Natural Language Guided Reinforcement Learning”).
Regarding Claim 3, Shazeer in view of Bengio and Liu teach the method of claim 1, wherein for each gating sub-layer and for each output position in each of the target output sequence, the expected reward . . . [is generated] (Shazeer, Pg. 4, Para. 3, “We train the gating network by simple back-propagation, along with the rest of the model. If we choose k > 1, the gate values for the top k experts have nonzero derivatives with respect to the weights of the gating network . . . Gradients also backpropagate through the gating network to its inputs. Our method differs here from (Bengio et al., 2015) who use boolean gates and a REINFORCE-style approach to train the gating network”, where the gating sub layer, “the gating network”, receives a training value for the output positions, “Gradients . . .to its inputs”, which in view of Bengio, is the expected reward “
E
t
π
θ
”,, see Bengio, Pg. 11, Para. 6, “In reinforcement learning, a sequence of state-action-reward tuples is described as a trajectory t. The objective function of a parameterized policy πθ for the cumulative return of a trajectory t is described as:
PNG
media_image1.png
98
326
media_image1.png
Greyscale
”, where the expected reward, “
E
t
π
θ
”, is received by the gating functionality, see Bengio, Pg. 2, Para. 2-6, “We propose to learn input-dependent activation probabilities for every node (or blocks of nodes), while trying to jointly minimize the prediction errors at the output and the number of participating nodes at every layer, thus reducing the computational load . . . Our model consists in a typical fully-connected neural network model, joined with stochastic per layer policies that activate or deactivate nodes of the neural network in an input-dependent manner, both at train and test time . . .We train a different policy for each layer l”)
from (i) the immediate reward for the output position that depends on the respective score assigned to the ground truth output token at the output position by the score distribution generated by the task neural network for the output position and (ii) respective future immediate rewards for each future output position that follows the output position in the target output sequence that depend on a respective score assigned to the ground truth output token at the future output position by the score distribution generated by the task neural network for the future output position (Liu, Pg. 1646, Col. 1, Para. 2, “Accumulated Reward . . . The Qrefine reward r of each word is the combination of the wording reward and the answer correlation reward,
PNG
media_image2.png
76
402
media_image2.png
Greyscale
. . . between wording reward rlm and answer correlation reward rac; Since we want Qrefine module to have the ability to infer reward even if not reaching the end of the generation process and the future reward will influence the current reward, we adopt the accumulated Qrefine reward R with the discounted factor ỿ and the accumulated Qrefine discounted reward of t-th word is represented as,
PNG
media_image3.png
46
544
media_image3.png
Greyscale
By using the accumulated reward, we are able to infer the answer correlation reward even if we are not reaching the end of the generation process”, where for each output position, “of each word”, in each of the target output sequences, see Liu, Pg. 1644, Col. 2, Para. 3, “Given an ill-formed question consists of x = [x1,x2, …, xn] of an arbitrary-length N, the well-formed question y = [y1,y2, …, yn] of a variable-length M. The aim of question refinement is to refine x to y”, the expected reward is generated from “rw” for the output position, “r(yt)”, which is an immediate reward, see Liu, Pg. 1645, Col. 2, Para. 4, “wording reward rw aims to give an immediate reward to each of words”, which is “influence[d]” by “future reward[s]” for each future output position that follows the output position, “t+1” to “M-t” in the target output sequence, “M”, which are within the broadest reasonable interpretation of future immediate rewards, such that the “expected reward” is generated from the immediate reward and respective immediate future rewards, Liu, Pg. 1646, Col. 2-, Para. 5, Equation 9; see generally Liu, Pg. 1646, Col. 1, Para. 1, “We train the model by maximizing answer correlation reward rac using ground-truth” and Liu, Pg. 1645, Col. 1, Para. 1, “the Seq2Seq model is formed as . . . softmax”, wherein each reward depends on the respective score assigned to the ground truth output token at the output position by the score distribution generated by the task neural network, see Shazeer , Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”; see also Shazeer , Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a model output with a plurality of output positions; Shazeer , Pg. 2, Para. 3, “Our approach to conditional computation is to introduce a new type of general purpose neural network component”, where the model is a “neural network”, which as discussed above, is the task neural network; Shazeer , Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”; Shazeer , Pg. 3, Para. 1, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position”, where the “MoE is called once” for each training example, “each position in the text”).
The reasons for obviousness were discussed in regard to the rejection of Claim 1 above and remain applicable here.
Shazeer in view of Bengio and Liu do not explicitly disclose . . . is a time discounted sum of is a time discounted sum of . . .
However, Kaplan teaches [a reinforcement learning method, where an expected reward] . . . is a time discounted sum of [the respective rewards] . . . (Pg. 3, Para. 5, “For all reinforcement learners there is the notion of the value of a state or action. The objective is to maximize an exponentially time-discounted sum of future rewards”, where “time-discounted sum of future rewards” is a time discounted sum of the respective rewards; see also Pg. 4, Para. 2, “Policy networks maximize the expected, discounted reward of following that policy, which it can be shown equates to gradient descent with step size proportional to the discounted reward R times the log of the probability π assigned to the action”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the receiving of the expected reward for the output positions by the gating sub-layer, wherein each expected reward is generated from the immediate reward for an output position and the respective future immediate rewards for each future output position that follows the output position in a target sequence, and wherein each reward depends on the respective score assigned to the ground truth output token at the output position by the score distribution generated by the task neural network of Shazeer in view of Bengio and Liu with the reinforcement learning method, where an expected reward is a time discounted sum of the respective rewards of Kaplan in order to encourage faster pursuit of the reward (Kaplan, Pg. 3, Para. 5, “The time discounting combats value exploding to infinity over time and encourages faster pursuit of reward”), which contributes to improved performance (Kaplan, Pg. 1, Abstract, “Our agent significantly outperforms Deep Q-Networks (DQNs), Asynchronous Advantage ActorCritic (A3C) agents, and the best agents posted to OpenAI Gym”; Kaplan, Pg. 10, Para. 1, “The agent achieves impressive scores in relatively few frames where traditional agents fail”).
Claims 4 and 5 are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Peralta et al. (hereinafter Peralta) (“Mixture of Experts with Entropic Regularization for
Data Classification”).
Regarding Claim 4, Shazeer in view of Bengio and Liu teach the method of claim 1, wherein the reinforcement learning objective function further comprises . . . (Bengio, Pg. 11, Para. 6, “In reinforcement learning, a sequence of state-action-reward tuples is described as a trajectory t. The objective function of a parameterized policy πθ for the cumulative return of a trajectory t is described as:
PNG
media_image1.png
98
326
media_image1.png
Greyscale
”, where “J(θ)” is the “reinforcement learning” “objective function”).
The reasons for obviousness were discussed in regard to the rejection of Claim 1 above and remain applicable here.
Shazeer in view of Bengio and Liu do not explicitly disclose . . . an entropy term that measures an entropy of the layer input to expert neural network assignments.
However, Peralta teaches . . . [a function comprising] an entropy term that measures an entropy of the layer input to expert neural network assignments (Pg. 5, Para. 2-3, “we maximize the entropy of the gate network output in order to avoid outputs with very few active gates . . . Considering the cost equation associated with the gate network, to which we add the Shannon entropy with the weight λˆ , which represents the entropy degree of a continuous variable”, where “the cost equation” is a function that comprises an entropy term “the Shannon entropy with the weight λ”, which measures the “entropy” of expert assignments, the “outputs” of “active gates”, which measures the entropy of the layer input to expert assignments, because the “gate network” that assigns experts is based on “learnable complex patterns in [input] data”, see Pg. 1, Abstract, ““Mixture-of-experts” is a well-known classification technique; it is a probabilistic model consisting of local expert classifiers weighted by a gate network that is typically based on softmax functions, combined with learnable complex patterns in data”; see also Pg. 1, Para. 3, “This work uses a sparse mixture-of-experts for distributing multiple deep neural networks efficiently”, where the “experts” are “neural networks”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the reinforcement learning objective function of Shazeer in view of Bengio and Liu with the function comprising an entropy term that measures entropy of the layer input to expert neural network assignments of Peralta in order to order to avoid outputs with very few active gates, which will better model complex data patterns (Peralta, Pg. 5, Para. 1-2, “we think that the data patterns can be complex enough that each data input can be better modeled by a set of experts than by just one expert . . . Therefore, we maximize the entropy of the gate network output in order to avoid outputs with very few active gates”) and increase model accuracy on low dimensional data (Peralta, Pg. 13, Para. 2, “Our experiments provide evidence that the EMoE technique improves on the classification accuracy of the classical mixture-of-experts. The results for a diverse set of real datasets indicate a greater average accuracy. In this respect, the proposed technique demonstrates greater utility when the datasets do not have a large number of dimensions”).
Regarding Claim 5, Shazeer in view of Bengio, Liu, and Peralta teach the method of claim 4, wherein the entropy term measures a Shannon Entropy of the layer input to expert neural network assignments (Peralta, Pg. 5, Para. 2-3, “we maximize the entropy of the gate network output in order to avoid outputs with very few active gates . . . Considering the cost equation associated with the gate network, to which we add the Shannon entropy with the weight λˆ , which represents the entropy degree of a continuous variable”, where the entropy term is the “the Shannon entropy with the weight λ”, which measures the “entropy” of expert assignments, the “outputs” of “active gates”, which measures the entropy of the layer input to expert assignments, because the “gate network” that assigns experts is based on “learnable complex patterns in [input] data”, see Peralta, Pg. 1, Abstract, ““Mixture-of-experts” is a well-known classification technique; it is a probabilistic model consisting of local expert classifiers weighted by a gate network that is typically based on softmax functions, combined with learnable complex patterns in data”; see also Peralta, Pg. 1, Para. 3, “This work uses a sparse mixture-of-experts for distributing multiple deep neural networks efficiently”, where the “experts” are “neural networks”).
The reasons for obviousness were discussed in regard to the rejection of Claim 4 above and remain applicable here.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Goswami (“A survey of modeling, rendering and animation of clouds in computer graphics”).
Regarding Claim 8, Shazeer in view of Bengio and Liu teach the method of claim 6, wherein the gating function . . . (Shazeer, Pg. 4, Para. 1, “Softmax Gating: A simple choice of non-sparse gating function (Jordan & Jacobs, 1994) is to multiply the input by a trainable weight matrix Wg and then apply the Softmax function”).
Shazeer in view of Bengio and Liu do not explicitly disclose . . . is an optimal transport function.
However, Goswami teaches [a function that] . . . is an optimal transport function (Pg. 10, Col. 1, Para. 4, “An optimal transport function helps to establish correspondences between the best pair matches in the source and target”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the gating function of Shazeer in view of Bengio and Liu with the function that is an optimal transport function of Goswami in order to generate optimal pairs of tasks and expert neural networks (Goswami, Pg. 10, Col. 1, Para. 4, “An optimal transport function helps to establish correspondences between the best pair matches in the source and target cloudscapes. The animated primitives are created from the interpolation of the identified pairs such that they follow the shortest trajectory”, where “identif[ying]” the “shortest trajectory” could similarity identify the “optimal” experts, “target”, to “pair” with the task, “source”), which has demonstrated utility in simplifying and accelerating complex tasks (compare Goswami, Pg. 9, Col. 1, Para. 3, “[this is] a complex task, which entails . . . setting the right initial conditions . . . [this approach] helps us design specific solutions that are both simplified and accelerated”, where a “simplified” approach is used to “accelerate” “setting the right initial conditions” in “a complex task”, with Bengio, Pg. 1, Abstract, “Deep learning has become the state-of-art tool in many applications, but the evaluation and training of deep models can be time-consuming and computationally expensive. The conditional computation approach has been proposed to tackle this problem . . . by selectively activating only parts of the network at a time . . . We propose a learning scheme motivated by computation speed, capturing the idea of wanting to have parsimonious activations while maintaining prediction accuracy”, where the goal of “conditional computation” is similarly to simplify complex tasks, “selectively activating only parts of the network at a time”, in order to increase speed “We propose a learning scheme motivated by computation speed”).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, Goswami, and Tay et al. (hereinafter Tay) (“Sparse Sinkhorn Attention”).
Regarding Claim 9, Shazeer in view of Bengio, Liu, and Goswami teach the method of claim 8, wherein the gating function applies a[n] . . . algorithm to the gating scores to generate the assignation scores (Shazeer, Pg. 4, Para. 2, “Noisy Top-K Gating: . . .
PNG
media_image4.png
202
704
media_image4.png
Greyscale
”, where, in the context of “Noisy Top-K Gating”, the respective sets of gating scores for the layer inputs, “H(x)i”, are processed by the “gating function” by applying an algorithm “KeepTopK” to generate the respective sets of assignation scores for each layer input, “G(x)i”).
Shazeer in view of Bengio, Liu, and Goswami do not explicitly disclose . . . Sinkhorn . . . .
However, Tay teaches [a model learning method using a] (Pg. 1, col. 1, Abstract, “We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend . . . Via extensive experiments on algorithmic seq2seq sorting, language modeling, pixel-wise image generation, document classification and natural language inference”)
. . . Sinkhorn [algorithm for sorting internal representations] (Pg. 2, Col. 1, Para. 3, “We propose Sparse Sinkhorn Attention, a new attention method based on dynamic, learnable sorting of internal representations”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the gating function employing an algorithm to generate assignation scores based on gating scores of Shazeer in view of Bengio, Liu, and Goswami with the use of a Sinkhorn algorithm for sorting internal representations of Tay in order to generate assignation scores using an algorithm with demonstrated efficiency (Tay, Pg. 1, Abstract, “Via extensive experiments on algorithmic seq2seq sorting, language modeling, pixel-wise image generation, document classification and natural language inference, we demonstrate that our memory efficient Sinkhorn Attention method is competitive with vanilla attention and consistently outperforms recently proposed efficient Transformer models such as Sparse Transformers”) in a transferable field (Tay, Pg. 2, Col. 2, Para. 4, “Learning sparse outputs in attention models has also garnered reasonable interest . . . Along a similar vein, this is also reminiscent of Sparse Mixture of Experts (Shazeer et al., 2017), which performs a sparse selection of outputs (experts) for prediction tasks”; see also Shazeer, Pg. 4, Para. 2, “Noisy Top-K Gating: . . .
PNG
media_image4.png
202
704
media_image4.png
Greyscale
”, where the assignation scores for each layer input, “G(x)i”, are used to sort the internal representations, which are the gating scores for the layer inputs, “H(x)i”).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Du et al. (hereinafter Du) (Pat. Pub. No. US 2020/0327985 A1).
Regarding Claim 10, Shazeer in view of Bengio and Liu teach the method of claim 1, wherein generating a layer output for each conditional computation layer from the respective expert outputs for the conditional computation layer for the layer inputs comprises (Shazeer, Pg. 2-3, Para. 4-1, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE) . . . The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position . . . On both language modeling and machine translation benchmarks, we improve on best published results at a fraction of the computational cost”, where the layer output sequence for the “conditional computation” layer is the output if the “MoE” for “each position in the text”, which “select[s] a potentially different combination of experts at each position” to process the layer inputs; see also Shazeer, Pg 16, Table 8; Shazeer, Pg. 16, Para. 7, “We evaluate our model using perplexity on a holdout dataset. Results are reported in Table 8”, where “evaluation” of a “model”, to achieve the results depicted in “Table 8”, requires that the processing steps occur during both training and evaluation)
. . . [combining] the respective expert outputs for the conditional computation layer for the layer inputs (Shazeer, Pg. 2, Fig. 1, where the respective “Expert” outputs, “x”, which are based on the layer inputs, as indicated by the arrows, and are combined, “+”, as output of the “MoE layer”, which is the conditional computation layer, see also Shazeer, Pg. 3, Para. 6, “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows: [equation 1]”).
Shazeer in view of Bengio and Liu do not explicitly disclose . . . concatenating . . . .
However, Du teaches [a method utilizing neural networks to analyze data, comprising] (Para. [0001], “Recently, more and more approaches adopt deep neural networks, such as a convolutional neural network (CNN) and a recurrent neural network (RNN), and achieve good accuracy for multi-class classification tasks”)
. . . concatenating [expert knowledge] . . . (Para. [0058], “The merging of the learned text information and the learned signal information may include generating the one or more representations including one or more feature vectors, using a concatenated and weighted combination based on model learning or expert knowledge”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the combining of the respective expert outputs for the conditional computation layer for the layer inputs to generate a layer output of Shazeer in view of Bengio and Liu with the use of neural networks to analyze data, comprising concatenating expert knowledge of Du in order to combine the expert outputs using a method proven to enhance model performance by providing comprehensive information for accurate analysis (Du, Para. [0043]-[0044], “the multimodal framework 300 can learn and extract information from both text and signal data to provide a more accurate analysis . . . the multimodal framework 300 can accept different types of . . . data . . . that could provide comprehensive information for better performance of a model”; compare Shazeer, Pg. 14, Para. 2, “The output of the MoE is given by . . . [equation 12]”, where the combining of expert “output[s]” may lead to skewed results if an output is inaccurate and the outputs are averaged or otherwise combined in a manner that does not preserve the individual values, with Du, Para. [0058], “The merging of the learned text information and the learned signal information may include generating the one or more representations including one or more feature vectors, using a concatenated and weighted combination based on model learning or expert knowledge”, where the “expert knowledge” is preserved after “concatenat[ion]”).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Keneshloo et al. (hereinafter Keneshloo) (“Deep Reinforcement Learning for Sequence-to-Sequence Models”).
Regarding Claim 16, Shazeer in view of Bengio and Liu teach the method of claim 15, wherein selecting an output token for the output position from the vocabulary of output tokens in accordance with the respective new score distribution for the output position comprises: selecting the output token from the vocabulary of output tokens that corresponds to . . . [a] score in the respective new score distribution for the output position (Shazeer, Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective new score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”; see also Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a model output with a plurality of output positions, where an output token is selected in in accordance with the respective score distribution for the output position, see Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016)”, where a person of ordinary skill in the art would understand “beam search” to be a selection technique based on the new score values, at least within the context of generating outputs for “language modeling and machine translation”, see Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”, which will be selected based on the score generated for the output position).
Shazeer in view of Bengio and Liu do not explicitly disclose . . . the largest . . . .
However, Keneshloo teaches . . . [selecting the output token that corresponds with] the largest [score in the respective score distribution for the output position] (Pg. 2470, Col. 1, Para. 2, “the model generates an entire sequence as follows: Let yˆt denote the action (output) taken by the model at time t. Then, the next action is generated by yˆt+1 = arg max πθ (y| yˆt,st+1)”, where the “arg max” function “generate[s]” the “output” corresponding with the largest, “max”, of the “distribution over the next output . . . πθ (y| yˆt,st+1)”, see Pg. 2470, Col. 1, Para. 1, “The RNN learns a recursive function to compute st and outputs the distribution over the next output . . . yˆt+1 ∼ πθ (y| yˆt,st+1)”; see also Pg 2471, Col. 2, Para. 2, “In practice, seq2seq model is used as the policy, which generates actions . . . e.g., for a text summarization task, and the action denotes choosing the next token for the summary”, where the output action can be a “token”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the selecting of an output token for an output position from a vocabulary of output tokens in accordance with the score distribution for the output position, wherein the selected token corresponds with a score in the score distribution of Shazeer in view of Bengio and Liu with the selecting an output token that corresponds with the largest score in the respective score distribution for an output position of Keneshloo in order to select the optimal token at each step (Keneshloo, Pg. 2482, Col. 1, Para. 1, “a greedy search algorithm is run to determine the optimal action at each time step, and the policy is trained to predict that action”, where argmax is a “greedy search algorithm”), which leads to improved results when incorporated into the token selection method of Shazeer in view of Bengio (compare Keneshloo, Pg. 2470, Col. 1, Para. 2, “the model generates an entire sequence as follows: Let yˆt denote the action (output) taken by the model at time t. Then, the next action is generated by yˆt+1 = arg max πθ (y| yˆt,st+1). This process could be improved by using beam search to find a reasonable good output sequence”, where use of “arg max” in combination with “beam search” leads to an “improved” “process”, with Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016)”, where Shazeer uses a “beam search” approach).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Olivecrona et al. (hereinafter Olivecrona) (“Molecular de‑novo design through deep reinforcement learning”).
Regarding Claim 17, Shazeer in view of Bengio and Liu teach the method of claim 15, wherein selecting an output token for the output position from the vocabulary of output tokens in accordance with the respective new score distribution for the output position comprises: . . . [selecting] the vocabulary of output tokens in accordance with the respective new score distribution for the output position (Shazeer, Pg. 15, Para. 2-3, “The Softmax output layer was trained efficiently using importance sampling similarly to the models in (Jozefowicz et al., 2016) . . . We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016). We follow the standard procedure and sum over all the words including the end of sentence symbol”, where a person of ordinary skill in the art would understand the use of a “Softmax output layer” to require computing a respective new score distribution over the outputs, as further supported by the reference to “perplexity on the holdout dataset . . . sum[med] over all the words including the end of sentence symbol”; see also Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a model output with a plurality of output positions, where an output token is selected in in accordance with the respective new score distribution for the output position, see Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016)”, where a person of ordinary skill in the art would understand “beam search” to be a selection technique based on the score values, at least within the context of generating outputs for “language modeling and machine translation”, see Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”, which will be selected based on the score generated for the output position).
Shazeer in view of Bengio and Liu do not explicitly disclose . . . sampling the output token from . . . .
However, Olivecrona teaches . . . sampling the output token from [the vocabulary of output tokens in accordance with the respective score distribution] . . . (Pg. 3, Col. 1, Para. 3, “Once an RNN has been trained on target sequences, it can then be used to generate new sequences that follow the conditional probability distributions learned from the training set . . . we sample an output token xt
from the predicted probability distribution P(Xt ) over our vocabulary X”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the selecting of an output token for an output position from a vocabulary of output tokens in accordance with the score distribution for the output position, wherein the selected token corresponds with a score in the score distribution of Shazeer in view of Bengio and Liu with the sampling of an output token from the vocabulary of output tokens in accordance with the respective score distribution of Olivecrona in order to focus token searching over a subset of data without imposing rigid rules, which contributes to high accuracy (Olivecrona, Pg. 2, col. 1, para. 2, “By using a data-driven method that attempts to learn the underlying probability distribution over a large set of [data] . . . the search over the [data] . . . can be reduced . . . without introducing the rigidity of rule based approaches . . . which generates a high fraction of predicted actives”, where, in the context, “active” is based on “accuracy”, see Olivecrona, Pg. 11, Col. 2, Para. 3, “Whether or not these compounds are active will be dependent on the accuracy of the target activity model”).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Shazeer in view of Bengio, Liu, and Gomez et al. (hereinafter Gomez) (Pat. Pub. No. US 2020/0089772 A1).
Regarding Claim 18, Shazeer in view of Bengio and Liu teach the method of claim 15, wherein the task neural network . . . generates the output tokens in the new network output . . . (Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora”, where both “language modeling and machine translation” require processing a model input to generate a new model output with a plurality of output positions, where an output token is selected in in accordance with the respective score distribution for the output position, see Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces. We also used the same beam search technique as proposed in (Wu et al., 2016)”, where a person of ordinary skill in the art would understand “beam search” to be a selection technique based on generated score values, at least within the context of generating outputs for “language modeling and machine translation”, see Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation”; Shazeer, Pg. 17, Para. 3, “We use a shared source and target vocabulary of 32K wordpieces”, where both the input, “source”, and target output, “target”, are vocabulary tokens, “vocabulary of 32K wordpieces”, which will be selected based on the score generated for the output position).
Shazeer in view of Bengio do not explicitly disclose . . . autoregressively . . . by processing a combined sequence comprising at least a concatenation of the new network input and any output tokens at output positions in the new network output preceding the output token.
However, Gomez teaches . . . [a neural network that] autoregressively [generates network outputs] (Abstract, “processing the encoder neural network output using an autoregressive decoder neural network to generate a decoder neural network output”)
. . . by processing a combined sequence comprising at least a concatenation of the new network input and any output tokens at output positions in the new network output preceding the output token (Para. [0033], “Text segment inputs in an input natural language and text segment outputs in a target natural language are embedded into the same feature depth, encoded by two separate sub-networks and concatenated before being fed into a decoder that autoregressively generates each element of the output. At each step, the autoregressive decoder produces a new output prediction given the encoded inputs and the encoding of the existing predicted outputs”, where “segment[s]” of “text” are within the broadest reasonable interpretation of tokens and the “Text segment inputs” and “Text segment outputs”, which are preceding “existing predicted outputs”, are “concatenated” and processed by being “fed into a decoder that autoregressively generates . . . the output”).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the use of a neural network to generate output tokens of Shazeer in view of Bengio with the use of an autoregressive neural network to generate output tokens by processing network inputs that are concatenated with preceding network outputs of Gomez in order to jointly process the input and output tokens (Gomez, Para. [0033], “Text segment inputs . . . and text segment outputs . . . [are] concatenated before being fed into a decoder that autoregressively generates each element of the output”), which contributes to an improved machine translation method with consistent accuracy and reduced computational complexity (compare Gomez, Para. [0023], “The machine translation system described in this specification represents a significant technical improvement to state of the art machine translation systems . . . [the system] enables machine translation tasks to be performed at similar accuracy whilst requiring less system parameters and reduced computational costs compared to conventional machine translation systems” with Shazeer, Pg. 1, Abstract, “We apply the MoE to the tasks of language modeling and machine translation”).
Response to Amendment
Applicant's arguments filed on March 3rd, 2026 have been fully considered. Each argument is addressed in detail below.
I. Applicant indicates the objections to the claims should be withdrawn (Applicant’s Remarks, 03/03/2026, Pg. 1).
Applicant’s amendments to the claims have overcome each and every objection to the claims, as previously set forth in the December 3rd, 2025 Office Action. As a result, these objections have been withdrawn.
However, Applicant’s amendments introduce new informalities, which necessitate the new grounds for objection discussed in detail above.
II. Applicant argues the rejections of the claims, under 35 USC § 103, should be withdrawn (Applicant’s Remarks, 03/03/2026, Section “Rejections under 35 USC § 103”, Pg. 1-2).
In response to Applicant’s amendments, the previously communicated rejections under 35 U.S.C. § 103, have been withdrawn. However, Applicants arguments are not persuasive in light of the new grounds for rejection, under 35 U.S.C. § 103, discussed in detail above. The new grounds of rejection rely on new prior art of record to teach the new combination of elements in the amended independent claims, which were not presented in this arrangement in any of the previously presented claims. As a result, Applicant’s arguments are rendered moot.
III. Applicant argues the rejections of the claims, under 35 USC § 112(b), should be withdrawn (Applicant’s Remarks, 03/03/2026, Section “Rejections under 35 USC § 112, Second Paragraph”, Pg. 2).
Applicant’s amendments to the claims have overcome each and every rejection to the claims, under 35 USC § 112(b), as previously set forth in the December 3rd, 2025 Office Action. As a result, these objections have been withdrawn.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW BRYCE GOLAN whose telephone number is (571)272-5159. The examiner can normally be reached Monday through Friday, 8:00 AM to 5:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached at (571) 270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW BRYCE GOLAN/Examiner, Art Unit 2123
/ALEXEY SHMATOV/Supervisory Patent Examiner, Art Unit 2123
Prosecution Insights