DETAILED ACTION
This action is in response to the application filed 05/05/2025. Claims 1-21 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 10/31/2025 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered.
Claim Objections
Claims 1, 9, and 13 are objected to because of the following informalities:
Regarding claim 1, “that is same across all of the one or more positions in the input sequence” in line 11-12 should read “that is the same across all of the one or more positions in the input sequence”.
Regarding claim 9, “that is same across all of the one or more positions in the input sequence” in line 11-12 should read “that is the same across all of the one or more positions in the input sequence”.
Regarding claim 13, “that is same across all of the one or more positions in the input sequence” in line 11-12 should read “that is the same across all of the one or more positions in the input sequence”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Specifically, the claim limitation “a shared query-key-value (QKV) attention mechanism that is the same across all of the one or more positions in the input sequence” in claims 1, 9, and 13 is not supported by the specification. Page 7 of the specification describes “Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention using the queries, keys, and values to generate an output. When there are multiple attention heads, the sub-layer 120 then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer.” The specification does not describe how this QKV attention mechanism is shared, nor does the specification describe a shared QKV attention mechanism that is the same across all the positions in an input sequence.
Regarding claims 2-8 and 21, claims 2-8 and 21 are rejected for at least the same reasons as claim 1 since claims 2-8 and 21 depend on claim 1.
Regarding claims 10-12, claims 10-12 are rejected for at least the same reasons as claim 9 since claims 10-12 depend on claim 9.
Regarding claims 14-20, claims 14-20 are rejected for at least the same reasons as claim 13 since claims 14-20 depend on claim 13.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-21 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 14, 22, and 23 of copending Application No. 18/009841 in view of Vaswani et al. (“Attention Is All You Need”) (hereafter referred to as Vaswani) in further view of Shazeer et al. (“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of Experts Layer”) (hereafter referred to as Shazeer).
This is a provisional nonstatutory double patenting rejection.
Regarding claim 1:
Instant Application Claim 1
US Application No. 18/009841 Claim 1
A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to process a network input to generate a network output for a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub- layer configured to:
A system for performing a machine learning task on a network input to generate a network output, the system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to perform the machine learning task, the attention neural network comprising a plurality of layers arranged in a sequence, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
and generate an attended input sequence
and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer,
the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to: receive the attended input sequence;
the attended input sequence comprising a respective attended layer input at each of the one or more positions, and the feed-forward sub-layer configured to: receive the attended input sequence;
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions, wherein, for each layer in a subset of the plurality of layers in the sequence, the feed- forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
receiving the respective attended layer input at the position
receiving the respective attended layer input at the position;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores, processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset, wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores;
selecting, from the plurality of expert feed-forward neural networks, a proper subset of expert feed-forward neural networks based at least on the respective gate scores; processing the respective attended layer input at the position using each of the expert feed-forward neural networks in the proper subset to generate a respective expert output for each of the expert feed-forward neural networks;
combining the respective expert outputs to generate a combined expert output;
combining the respective expert outputs to generate a combined expert output;
and generating the respective layer output at the position from the combined expert output.
and generating the respective layer output at the position from the combined expert output, and wherein, for each layer that is not in the subset of the plurality of layers, the feed-forward sub-layer is configured to generate the output sequence for the layer by processing each respective attended layer input at each of the positions in the input sequence for the layer using a single feed-forward neural network, and wherein the input sequence received by at least one layer that is not in the subset comprises the output sequence generated by a preceding layer in the subset.
Claim 1 of the ‘841 application does not explicitly disclose but Vaswani discloses:
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
The disclosures by claim 1 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Claim 1 of the ‘841 application in view of Vaswani do not disclose, but Shazeer does disclose:
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
The disclosures by claim 1 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 2:
Instant Application Claim 2
US Application No. 18/009841 Claim 14
The system of claim 1, wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises: selecting k expert feed-forward neural networks with k highest respective gate scores among the plurality of expert feed-forward neural networks.
9. The system of claim 1, wherein selecting, from the plurality of expert feed-forward neural networks, a proper subset based at least on the respective gate scores comprises: selecting at most k of a total number E of expert feed-forward neural networks in the plurality of expert feed-forward neural networks
14. The system of claim 9, wherein after training the attention neural network, selecting, from the plurality of expert feed-forward neural networks, a proper subset comprises: selecting the k experts with the k highest gating scores.
Regarding claim 3, claim 14 of the ‘841 application discloses all the limitations of claim 2 as shown above. Claim 14 of the ‘841 application in does not explicitly disclose, but claim 14 in view of Vaswani and Shazeer further discloses:
wherein k = 1 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that since the Noisy-Top-K Gating chooses 2 experts, the Noisy-Top-K Gating must first choose 1 expert and thus k=1.).
The disclosures by claim 14 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 4, claim 14 of the ‘841 application discloses all the limitations of claim 2 as shown above. Claim 14 of the ‘841 application in does not explicitly disclose, but claim 14 in view of Vaswani and Shazeer further discloses:
wherein k = 2 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers.” Examiner notes that k=2.).
The disclosures by claim 14 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 5:
Instant Application Claim 5
US Application No. 18/009841 Claim 1
The system of claim 1, wherein, for a second subset of the plurality of layers, the feed-forward sub-layer includes a single feed forward neural network and is configured to generate the output sequence for the layer by processing the respective attended layer input at each of the one or more positions in the attended input sequence using the single feed forward neural network.
for each layer that is not in the subset of the plurality of layers, the feed-forward sub-layer is configured to generate the output sequence for the layer by processing each respective attended layer input at each of the positions in the input sequence for the layer using a single feed-forward neural network, and wherein the input sequence received by at least one layer that is not in the subset comprises the output sequence generated by a preceding layer in the subset.
Regarding claim 6, claim 1 of the ‘841 application discloses all the limitations of claim 1 as shown above. Claim 1 of the ‘841 application in does not explicitly disclose, but claim 1 in view of Vaswani and Shazeer further discloses:
wherein the plurality of expert feed-forward neural networks have a same architecture but different parameter values (Shazeer, page 3, 5th paragraph, “the experts are themselves neural networks, each with their own parameters. Although in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.”).
The disclosures by claim 1 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 7, claim 1 of the ‘841 application discloses all the limitations of claim 1 as shown above. Claim 1 of the ‘841 application in does not explicitly disclose, but claim 1 in view of Vaswani further discloses:
wherein the attention mechanism comprises a multi-head self-attention mechanism (Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
and “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network” (Vaswani, page 2 last paragraph – page 3 first paragraph).).
The disclosures by claim 1 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Regarding claim 8, claim 1 of the ‘841 application discloses all the limitations of claim 1 as shown above. Claim 1 of the ‘841 application in does not explicitly disclose, but claim 1 in view of Vaswani further discloses:
wherein the machine learning task comprises one of: a text generation task where the network output comprises a sequence of text; or an image generation task where the network output comprises a sequence of intensity values for pixels of an image (Vaswani, page 1, abstract, “Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train” where “Most competitive neural sequence transduction models have an encoder-decoder structure [5,2,29]. Here the encoder maps an input sequence of symbol representations (x1, …, xn) to a sequence of continuous representations z = (z1,…,zn). Given z, the decoder then generates an output sequence (y1, …, ym) of symbols one element at a time…. The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1 respectively” (Vaswani, page 2, 8th paragraph), and where, “On the WMT 2014 English-to-German translations task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (include ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4” (Vaswani, page 8, 1st paragraph). Examiner notes that the machine learning task is generating a translation of text. Examiner further notes that the output sequence or translation of text is the sequence of text).
The disclosures by claim 1 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Regarding claim 9:
Instant Application Claim 9
US Application No. 18/009841 Claim 22
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:an attention neural network configured to process a network input to generate a network output for a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub- layer configured to:
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:an attention neural network configured to perform a machine learning task on a network input to generate a network output,, the attention neural network comprising a plurality of layers arranged in a sequence, each layer comprising an attention sub-layer and a feed-forward sub- layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
and generate an attended input sequence
and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer,
the attended input sequence comprising a respective attended layer input at each of the one or more positions, andthe feed-forward sub-layer configured to:receive the attended input sequence;
the attended input sequence comprising a respective attended layer input at each of the one or more positions, andthe feed-forward sub-layer configured to:receive the attended input sequence;
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions,wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions,wherein, for each layer in a subset of the plurality of layers in the sequence, the feed- forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
receiving the respective attended layer input at the position
receiving the respective attended layer input at the position;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores, processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset, wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores;
selecting, from the plurality of expert feed-forward neural networks, a proper subset of expert feed-forward neural networks based at least on the respective gate scores;processing the respective attended layer input at the position using each of the expert feed-forward neural networks in the proper subset to generate a respective expert output for each of the expert feed-forward neural networks;
combining the respective expert outputs to generate a combined expert output;
combining the respective expert outputs to generate a combined expert output;
and generating the respective layer output at the position from the combined expert output.
and generating the respective layer output at the position from the combined expert output, andwherein, for each layer that is not in the subset of the plurality of layers, the feed-forward sub-layer is configured to generate the output sequence for the layer by processing each respective attended layer input at each of the positions in the input sequence for the layer using a single feed-forward neural network, and wherein the input sequence received by at least one layer that is not in the subset comprises the output sequence generated by a preceding layer in the subset.
Claim 22 of the ‘841 application does not explicitly disclose but Vaswani discloses:
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
The disclosures by claim 22 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Claim 22 of the ‘841 application in view of Vaswani do not disclose, but Shazeer does disclose:
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
The disclosures by claim 22 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 10, claim 22 of the ‘841 application discloses all the limitations of claim 9 as shown above. Claim 22 of the ‘841 application in does not explicitly disclose, but claim 9 in view of Vaswani and Shazeer further discloses:
wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises: selecting k expert feed-forward neural networks with k highest respective gate scores among the plurality of expert feed-forward neural networks (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” where “Noisy Top-K Gating: we add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to - ∞ (which causes the corresponding gate values to equal 0)” (Shazeer, page 4, 2nd paragraph). ).
The disclosures by claim 22 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 11, claim 22 of the ‘841 application, Vaswani, and Shazeer discloses all the limitations of claim 10 as shown above. Claim 22 in view of Vaswani and Shazeer further discloses:
wherein k = 1 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that since the Noisy-Top-K Gating chooses 2 experts, the Noisy-Top-K Gating must first choose 1 expert and thus k=1.).
The disclosures by claim 22 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 12, claim 22 of the ‘841 application, Vaswani, and Shazeer discloses all the limitations of claim 10 as shown above. Claim 22 in view of Vaswani and Shazeer further discloses:
wherein k = 2 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers.” Examiner notes that k=2.).
The disclosures by claim 22 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 13:
Instant Application Claim 13
US Application No. 18/009841 Claim 23
A method performed by one or more computers, the method comprising:receiving a network input; andprocessing the network input using an attention neural network to generate a network output for a machine learning task, the attention neural network comprising a plurality of layers, each layer comprising an attention sub-layer and a feed-forward sub-layer, the attention sub- layer configured to:
A method performed by one or more computers, the method comprising:receiving a network input; andprocessing the network input using an attention neural network to generate a network output for the network input, the attention neural network comprising a plurality of layers arranged in a sequence, each layer comprising an attention sub-layer and a feed-forward sub- layer, the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
receive an input sequence for the layer comprising a respective layer input at each of one or more positions;
and generate an attended input sequence
and generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer,
the attended input sequence comprising a respective attended layer input at each of the one or more positions, andthe feed-forward sub-layer configured to:receive the attended input sequence;
the attended input sequence comprising a respective attended layer input at each of the one or more positions, andthe feed-forward sub-layer configured to:receive the attended input sequence;
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions,wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions,wherein, for each layer in a subset of the plurality of layers in the sequence, the feed- forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations comprising, for each of the positions in the input sequence for the layer:
receiving the respective attended layer input at the position
receiving the respective attended layer input at the position;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks;
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores, processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset, wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores;
selecting, from the plurality of expert feed-forward neural networks, a proper subset of expert feed-forward neural networks based at least on the respective gate scores;processing the respective attended layer input at the position using each of the expert feed-forward neural networks in the proper subset to generate a respective expert output for each of the expert feed-forward neural networks;
combining the respective expert outputs to generate a combined expert output;
combining the respective expert outputs to generate a combined expert output;
and generating the respective layer output at the position from the combined expert output.
and generating the respective layer output at the position from the combined expert output, andwherein, for each layer that is not in the subset of the plurality of layers, the feed-forward sub-layer is configured to generate the output sequence for the layer by processing each respective attended layer input at each of the positions in the input sequence for the layer using a single feed-forward neural network, and wherein the input sequence received by at least one layer that is not in the subset comprises the output sequence generated by a preceding layer in the subset.
Claim 23 of the ‘841 application does not explicitly disclose but Vaswani discloses:
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
The disclosures by claim 23 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Claim 23 of the ‘841 application in view of Vaswani do not disclose, but Shazeer does disclose:
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
The disclosures by claim 23 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 14, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 23 in view of Vaswani and Shazeer further discloses:
wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises: selecting k expert feed-forward neural networks with k highest respective gate scores among the plurality of expert feed-forward neural networks (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” where “Noisy Top-K Gating: we add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to - ∞ (which causes the corresponding gate values to equal 0)” (Shazeer, page 4, 2nd paragraph). ).
The disclosures by claim 23 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 15, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 23 in view of Vaswani and Shazeer further discloses:
wherein k = 1 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that since the Noisy-Top-K Gating chooses 2 experts, the Noisy-Top-K Gating must first choose 1 expert and thus k=1.).
The disclosures by claim 23 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 16, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 23 in view of Vaswani and Shazeer further discloses:
wherein k = 2 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers.” Examiner notes that k=2.).
The disclosures by claim 23 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 5:
Instant Application Claim 17
US Application No. 18/009841 Claim 23
The system of claim 1, wherein, for a second subset of the plurality of layers, the feed-forward sub-layer includes a single feed forward neural network and is configured to generate the output sequence for the layer by processing the respective attended layer input at each of the one or more positions in the attended input sequence using the single feed forward neural network.
for each layer that is not in the subset of the plurality of layers, the feed-forward sub-layer is configured to generate the output sequence for the layer by processing each respective attended layer input at each of the positions in the input sequence for the layer using a single feed-forward neural network, and wherein the input sequence received by at least one layer that is not in the subset comprises the output sequence generated by a preceding layer in the subset.
Regarding claim 18, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 23 in view of Vaswani and Shazeer further discloses:
wherein the plurality of expert feed-forward neural networks have a same architecture but different parameter values (Shazeer, page 3, 5th paragraph, “the experts are themselves neural networks, each with their own parameters. Although in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.”).
The disclosures by claim 23 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 19, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 23 in view of Vaswani further discloses:
wherein the attention mechanism comprises a multi-head self-attention mechanism (Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
and “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network” (Vaswani, page 2 last paragraph – page 3 first paragraph).).
The disclosures by claim 23 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Regarding claim 20, claim 23 of the ‘841 application discloses all the limitations of claim 13 as shown above. Claim 23 of the ‘841 application in does not explicitly disclose, but claim 13 in view of Vaswani further discloses:
wherein the machine learning task comprises one of: a text generation task where the network output comprises a sequence of text; or an image generation task where the network output comprises a sequence of intensity values for pixels of an image (Vaswani, page 1, abstract, “Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train” where “Most competitive neural sequence transduction models have an encoder-decoder structure [5,2,29]. Here the encoder maps an input sequence of symbol representations (x1, …, xn) to a sequence of continuous representations z = (z1,…,zn). Given z, the decoder then generates an output sequence (y1, …, ym) of symbols one element at a time…. The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1 respectively” (Vaswani, page 2, 8th paragraph), and where, “On the WMT 2014 English-to-German translations task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (include ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4” (Vaswani, page 8, 1st paragraph). Examiner notes that the machine learning task is generating a translation of text. Examiner further notes that the output sequence or translation of text is the sequence of text).
The disclosures by claim 23 of application No. 18/009841 and Vaswani are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified application 18/009841 to use the architecture in Vaswani. Doing so is advantageous because “the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers” (Vaswani, page 9, Conclusion, 2nd paragraph).
Regarding claim 21, claim 14 of the ‘841 application discloses all the limitations of claim 2 as shown above. Claim 14 of the ‘841 application in does not explicitly disclose, but claim 14 in view of Vaswani and Shazeer further discloses:
for each of the k expert feed-forward neural networks: determining whether the expert feed-forward neural network has already been selected a maximum number of times during processing of a group of attended layer inputs (Shazeer, page 19, Batchwise Mask section, "To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X,M), which operates over batches of input vectors. Instead of keeping the top k values per example, we keep the top m values per expert across the training batch…. As our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask…. To learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.” Examiner notes that the threshold determines whether the expert has been selected a maximum number of times.);
and selecting the expert feed-forward neural network for inclusion in the proper subset only when the expert feed-forward neural network has not already been selected the maximum number of times (Shazeer, page 13, 1st paragraph, "as discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples." Examiner notes that by having equal numbers of training examples, the experts are selected when the experts have not been selected a maximum number of times.)
The disclosures by claim 14 of application No. 18/009841, Vaswani and Shazeer are considered analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified No. 18/009841 and Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-8 and 13-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani et al. (“Attention Is All You Need”) (hereafter referred to as Vaswani) in view of Shazeer et al. (“Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of Experts Layer”) (hereafter referred to as Shazeer).
Regarding claim 1, Vaswani teaches
A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement: an attention neural network configured to process a network input to generate a network output for a machine learning task (Vaswani, page 7, Section 5.2 Hardware and Schedule, “ We trained our models on one machine with 8 NVIDIA P100 GPUs” and Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
Examiner notes that the inputs in Figure 1 is the network input and the output probabilities in Figure 1 is the network output.)
the attention neural network comprising a plurality of layers, each layer comprising an attention sub- layer and a feed-forward sub-layer, (Vaswani, page 2 last paragraph – page 3 first paragraph, “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network.” Examiner notes that the multi-head self-attention layer is the attention sub-layer and the position-wise fully connected feed-forward network is the feed-forward sub-layer) the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions (Vaswani, page 5, 3.2.3 Applications of Attention in our Model section, “The Transformer uses multi-head attention in three different ways: In ‘encoder-decoder attention’ layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence.”);
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
and the feed-forward sub-layer configured to: receive the attended input sequence (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer.”);
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is dmodel=512, and the inner-layer has dimensionality dff=2048m”),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
Vaswani does not teach, but Shazeer does teach
wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations (Shazeer, page 2, last paragraph, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input.” Examiner notes that the MOE layer is the first subset of the plurality of layers and the feed-forward sub-layer) comprising,
for each of the positions in the input sequence for the layer: receiving the respective attended layer input at the position (Shazeer, page 3, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” where “We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and the decoder, with the first decoder LSTM receiving output from and providing input for the attention” (Shazeer, page 17, Section E Machine Translation – Experimental Details). Examiner notes that the attended layer is created by the attention mechanism. );
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks (Shazeer, page 5, Section 4 Balancing Expert Utilization, 1st paragraph, “We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the gating network is the gating function and the gate values are the gate scores. Examiner further notes that the attended layer input is x in Figure 1.);
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores (Shazeer, page 2, Figure 1: “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network.” Examiner notes that the two experts are the one or more expert feed-forward neural networks.);
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are each a proper subset in the plurality of expert feed-forward neural networks and each generate an output. Examiner further notes that the attended layer input is x in Figure 1.);
wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are both proper subsets within the plurality of expert feed-forward neural networks after being selected by the gating function.);
combining the respective expert outputs to generate a combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are concatenated.);
and generating the respective layer output at the position from the combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the concatenation of Expert 2 and Expert n-1 is the respective layer output from the combined expert output.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 2, Vaswani in view of Shazeer teach the system of claim 1. Vaswani does not teach, but Shazeer does teach
wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises: selecting k expert feed-forward neural networks with k highest respective gate scores among the plurality of expert feed-forward neural networks (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” where “Noisy Top-K Gating: we add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to - ∞ (which causes the corresponding gate values to equal 0)” (Shazeer, page 4, 2nd paragraph). ).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 3, Vaswani in view of Shazeer teach the system of claim 2. Vaswani does not teach, but Shazeer does teach
wherein k = 1 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that since the Noisy-Top-K Gating chooses 2 experts, the Noisy-Top-K Gating must first choose 1 expert and thus k=1.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 4, Vaswani in view of Shazeer teach the system of claim 2. Vaswani does not teach, but Shazeer does teach
wherein k = 2 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers.” Examiner notes that k=2.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 5, Vaswani in view of Shazeer teach the system of claim 1. Vaswani further teaches,
wherein, for a second subset of the plurality of layers, the feed-forward sub-layer includes a single feed forward neural network and is configured to generate the output sequence for the layer by processing the respective attended layer input at each of the one or more positions in the attended input sequence using the single feed forward neural network (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically.” and Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
Examiner notes that the second subset of the plurality of layers is the multi-head attention layer and feed-forward layer. Examiner further notes that the output sequence is the output probabilities and each position is the respective attended layer input.).
Regarding claim 6, Vaswani in view of Shazeer teach the system of claim 1. Vaswani does not teach, but Shazeer does teach
wherein the plurality of expert feed-forward neural networks have a same architecture but different parameter values (Shazeer, page 3, 5th paragraph, “the experts are themselves neural networks, each with their own parameters. Although in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.”).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 7, Vaswani in view of Shazeer teach the system of claim 1. Vaswani further teaches,
wherein the attention mechanism comprises a multi-head self-attention mechanism (Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
and “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network” (Vaswani, page 2 last paragraph – page 3 first paragraph).).
Regarding claim 8, Vaswani in view of Shazeer teach the system of claim 1. Vaswani further teaches,
wherein the machine learning task comprises one of: a text generation task where the network output comprises a sequence of text; or an image generation task where the network output comprises a sequence of intensity values for pixels of an image (Vaswani, page 1, abstract, “Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train” where “Most competitive neural sequence transduction models have an encoder-decoder structure [5,2,29]. Here the encoder maps an input sequence of symbol representations (x1, …, xn) to a sequence of continuous representations z = (z1,…,zn). Given z, the decoder then generates an output sequence (y1, …, ym) of symbols one element at a time…. The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1 respectively” (Vaswani, page 2, 8th paragraph), and where, “On the WMT 2014 English-to-German translations task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (include ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4” (Vaswani, page 8, 1st paragraph). Examiner notes that the machine learning task is generating a translation of text. Examiner further notes that the output sequence or translation of text is the sequence of text).
Regarding claim 13, Vaswani teaches
A method performed by one or more computers, the method comprising: receiving a network input; and processing the network input using an attention neural network to generate a network output for a machine learning task (Vaswani, page 7, Section 5.2 Hardware and Schedule, “ We trained our models on one machine with 8 NVIDIA P100 GPUs” and Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
Examiner notes that the inputs in Figure 1 is the network input and the output probabilities in Figure 1 is the network output.)
the attention neural network comprising a plurality of layers, each layer comprising an attention sub- layer and a feed-forward sub-layer, (Vaswani, page 2 last paragraph – page 3 first paragraph, “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network.” Examiner notes that the multi-head self-attention layer is the attention sub-layer and the position-wise fully connected feed-forward network is the feed-forward sub-layer) the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions (Vaswani, page 5, 3.2.3 Applications of Attention in our Model section, “The Transformer uses multi-head attention in three different ways: In ‘encoder-decoder attention’ layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence.”);
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
and the feed-forward sub-layer configured to: receive the attended input sequence (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer.”);
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is dmodel=512, and the inner-layer has dimensionality dff=2048m”),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
Vaswani does not teach, but Shazeer does teach
wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations (Shazeer, page 2, last paragraph, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input.”) comprising,
for each of the positions in the input sequence for the layer: receiving the respective attended layer input at the position (Shazeer, page 3, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” where “We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and the decoder, with the first decoder LSTM receiving output from and providing input for the attention” (Shazeer, page 17, Section E Machine Translation – Experimental Details). Examiner notes that the attended layer is created by the attention mechanism. );
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks (Shazeer, page 5, Section 4 Balancing Expert Utilization, 1st paragraph, “We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the gating network is the gating function and the gate values are the gate scores. Examiner further notes that the attended layer input is x in Figure 1.);
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores (Shazeer, page 2, Figure 1: “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network.” Examiner notes that the two experts are the one or more expert feed-forward neural networks.);
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are each a proper subset in the plurality of expert feed-forward neural networks and each generate an output. Examiner further notes that the attended layer input is x in Figure 1.);
wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are both proper subsets within the plurality of expert feed-forward neural networks after being selected by the gating function.);
combining the respective expert outputs to generate a combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are concatenated.);
and generating the respective layer output at the position from the combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the concatenation of Expert 2 and Expert n-1 is the respective layer output from the combined expert output.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 14, claim 14 recites substantially similar limitations to claim 2, and is therefore rejected under the same analysis.
Regarding claim 15, claim 15 recites substantially similar limitations to claim 3, and is therefore rejected under the same analysis.
Regarding claim 16, claim 16 recites substantially similar limitations to claim 4, and is therefore rejected under the same analysis.
Regarding claim 17, claim 17 recites substantially similar limitations to claim 5, and is therefore rejected under the same analysis.
Regarding claim 18, claim 18 recites substantially similar limitations to claim 6, and is therefore rejected under the same analysis.
Regarding claim 19, claim 19 recites substantially similar limitations to claim 7, and is therefore rejected under the same analysis.
Regarding claim 20, claim 20 recites substantially similar limitations to claim 8, and is therefore rejected under the same analysis.
Regarding claim 21, Vaswani in view of Shazeer teach the system of claim 2, wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises. Vaswani does not teach, but Shazeer does teach
for each of the k expert feed-forward neural networks: determining whether the expert feed-forward neural network has already been selected a maximum number of times during processing of a group of attended layer inputs (Shazeer, page 19, Batchwise Mask section, "To force each expert to receive the exact same number of examples, we introduce an alternative mask function, Mbatchwise(X,M), which operates over batches of input vectors. Instead of keeping the top k values per example, we keep the top m values per expert across the training batch…. As our experiments suggest and also observed in (Ioffe & Szegedy, 2015), using a batchwise function during training (such as Mbatchwise) requires modifications to the inference when we may not have a large batch of examples. Our solution to this is to train a vector T of per-expert threshold values to approximate the effects of the batchwise mask…. To learn the threshold values, we apply an additional loss at training time which is minimized when the batchwise mask and the threshold mask are identical.” Examiner notes that the threshold determines whether the expert has been selected a maximum number of times.);
and selecting the expert feed-forward neural network for inclusion in the proper subset only when the expert feed-forward neural network has not already been selected the maximum number of times (Shazeer, page 13, 1st paragraph, "as discussed in section 4, for load-balancing purposes, we want to define an additional loss function to encourage experts to receive roughly equal numbers of training examples." Examiner notes that by having equal numbers of training examples, the experts are selected when the experts have not been selected a maximum number of times.)
Vaswani and Shazeer are considered analogous to the claimed invention because they both
use machine learning and attention mechanisms to perform machine translation. It would have
been obvious to one having ordinary skill in the art prior to the effective filing date to have
modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous
because "[they] obtain greater than 1000x improvements in model capacity with only minor
losses in computational efficiency and significantly advance the state-of-the-art results on public
language modeling and translation data sets" (Shazeer, page 2, 2ⁿᵈ to last paragraph)
Claim(s) 9-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Vaswani in view of Shazeer in further view of Shazeer et al. (US 2018/0341860 A1) (hereafter referred to as Shazeer ‘860).
Regarding claim 9, Vaswani teaches
an attention neural network configured to process a network input to generate a network output for a machine learning task (Vaswani, page 7, Section 5.2 Hardware and Schedule, “ We trained our models on one machine with 8 NVIDIA P100 GPUs” and Vaswani, page 3, Figure 1,
PNG
media_image2.png
555
353
media_image2.png
Greyscale
Examiner notes that the inputs in Figure 1 is the network input and the output probabilities in Figure 1 is the network output.)
the attention neural network comprising a plurality of layers, each layer comprising an attention sub- layer and a feed-forward sub-layer, (Vaswani, page 2 last paragraph – page 3 first paragraph, “the encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network.” Examiner notes that the multi-head self-attention layer is the attention sub-layer and the position-wise fully connected feed-forward network is the feed-forward sub-layer) the attention sub-layer configured to:
receive an input sequence for the layer comprising a respective layer input at each of one or more positions (Vaswani, page 5, 3.2.3 Applications of Attention in our Model section, “The Transformer uses multi-head attention in three different ways: In ‘encoder-decoder attention’ layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence.”);
and generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence and that uses a set of queries, a set of keys, and a set of values generated from the input sequence for the layer, the attended input sequence comprising a respective attended layer input at each of the one or more positions (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism.),
and the feed-forward sub-layer configured to: receive the attended input sequence (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer.”);
and generate an output sequence for the layer from the attended input sequence, the output sequence comprising a respective layer output at each of the one or more positions (Vaswani, page 5, 3.3 Position-wise Feed-Forward Networks, “In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically. This consists of two linear transformations with a ReLU activation in between…While the linear transformations are the same across different positions, they use different parameters from layer to layer. Another way of describing this is as two convolutions with kernel size 1. The dimensionality of input and output is dmodel=512, and the inner-layer has dimensionality dff=2048m”),
receiving the respective attended layer input at the position generated by the attention sub-layer in part by applying the QKV attention mechanism (Vaswani, page 4, 3.2.2 Multi-Head Attention section, “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values and is thus the respective attended layer input.)
Vaswani does not teach, but Shazeer does teach
wherein, for a first subset of the plurality of layers, the feed-forward sub-layer is a conditional computation sub-layer that (i) comprises a plurality of expert feed-forward neural networks and (ii) is configured to generate the output sequence for the layer by performing operations (Shazeer, page 2, last paragraph, “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input.”) comprising,
for each of the positions in the input sequence for the layer: receiving the respective attended layer input at the position (Shazeer, page 3, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” where “We insert MoE layers in both the encoder (between layers 2 and 3) and the decoder (between layers 1 and 2). We use an attention mechanism between the encoder and the decoder, with the first decoder LSTM receiving output from and providing input for the attention” (Shazeer, page 17, Section E Machine Translation – Experimental Details). Examiner notes that the attended layer is created by the attention mechanism. );
applying a gating function to the respective attended layer input at the position to generate a respective gate score for each of the plurality of expert feed-forward neural networks (Shazeer, page 5, Section 4 Balancing Expert Utilization, 1st paragraph, “We take a soft constraint approach. We define the importance of an expert relative to a batch of training examples to be the batchwise sum of the gate values for that expert” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the gating network is the gating function and the gate values are the gate scores. Examiner further notes that the attended layer input is x in Figure 1.);
selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores (Shazeer, page 2, Figure 1: “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network.” Examiner notes that the two experts are the one or more expert feed-forward neural networks.);
wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.)
processing the respective attended layer input at the position using each expert feed-forward neural network in a proper subset of the plurality of expert feed-forward neural networks to generate a respective expert output for the expert feed-forward neural network in the proper subset (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are each a proper subset in the plurality of expert feed-forward neural networks and each generate an output. Examiner further notes that the attended layer input is x in Figure 1.);
wherein the proper subset of the plurality of expert feed-forward neural networks comprises the one or more expert feed-forward neural networks that have been selected based at least on the respective gate scores (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are both proper subsets within the plurality of expert feed-forward neural networks after being selected by the gating function.);
combining the respective expert outputs to generate a combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that Expert 2 and Expert n-1 are concatenated.);
and generating the respective layer output at the position from the combined expert output (Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that the concatenation of Expert 2 and Expert n-1 is the respective layer output from the combined expert output.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Vaswani and Shazeer do not teach, but Shazeer ‘860 does teach
One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to implement an attention network (Shazeer ‘860, page 11, paragraph 0088, “Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.”)
Vaswani, Shazeer, and Shazeer ‘860 are analogous to the claimed invention because they use machine learning and attention mechanisms to perform machine translation. It would have been obvious to a person having ordinary skill in the art prior to the effective filing date to have implemented Vaswani in view of Shazeer on Shazeer ‘860. Thus, this would be applying a known technique (using machine learning and attention mechanisms) to a known device (non-transitory computer-readable storage media) ready for improvement to yield predictable results (machine translation) (MPEP 2143 I. (C) Use of known technique to improve similar devices (methods, or products) in the same way).
Regarding claim 10, Vaswani in view of Shazeer and Shazeer ‘860 teach the computer-readable storage media of claim 9. Vaswani does not teach, but Shazeer does teach
wherein selecting the one or more expert feed-forward neural networks based at least on the respective gate scores comprises: selecting k expert feed-forward neural networks with k highest respective gate scores among the plurality of expert feed-forward neural networks (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” where “Noisy Top-K Gating: we add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to - ∞ (which causes the corresponding gate values to equal 0)” (Shazeer, page 4, 2nd paragraph). ).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 11, Vaswani in view of Shazeer and Shazeer ‘860 teach the computer-readable storage media of claim 10. Vaswani does not teach, but Shazeer does teach
wherein k = 1 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
Examiner notes that since the Noisy-Top-K Gating chooses 2 experts, the Noisy-Top-K Gating must first choose 1 expert and thus k=1.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Regarding claim 12, Vaswani in view of Shazeer and Shazeer ‘860 teach the computer-readable storage media of claim 10. Vaswani does not teach, but Shazeer does teach
wherein k = 2 (Shazeer, page 14, last paragraph, “We varied the number of experts between models, using ordinary MoE layers with 4, 32 and 256 experts and hierarchical MoE layers with 256, 1024 and 4096 experts. We call the resulting models MoE-4, MoE-32, MoE-256, MoE-256-h, MoE-1024-h and MoE-4096- h. For the hierarchical MoE layers, the first level branching factor was 16, corresponding to the number of GPUs in our cluster. We use Noisy-Top-K Gating (see Section 2.1) with k = 4 for the ordinary MoE layers and k = 2 at each level of the hierarchical MoE layers.” Examiner notes that k=2.).
Vaswani and Shazeer are considered analogous to the claimed invention because they both use machine learning and attention mechanisms to perform machine translation. It would have been obvious to one having ordinary skill in the art prior to the effective filing date to have modified Vaswani to use the mixture of experts like in Shazeer. Doing so is advantageous because “[they] obtain greater than 1000x improvements in model capacity with only minor losses in computational efficiency and significantly advance the state-of-the-art results on public language modeling and translation data sets” (Shazeer, page 2, 2nd to last paragraph).
Response to Arguments
On pages 11-12, Applicant argues:
Applicant respectfully submits that the cited portions of Vaswani and Non-Patent
Literature Shazeer have not been shown to disclose or suggest this combination of features recited in amended claim 1.
In particular, the cited portions of Vaswani and Non-Patent Literature Shazeer have not been shown to disclose or suggest applying a same query-key-value (QKV) attention mechanism to different positions in the input sequence and selecting different subsets of expert feed-forward neural networks for the different positions in the input sequence.
Regarding the Applicant’s argument that the prior art does not disclose “generate an attended input sequence at least in part by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence”, Examiner respectfully disagrees. Specifically, Examiner notes that Vaswani in view of Shazeer teaches this with “On each of these projected versions of queries, keys, and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions” (Vaswani, page 4, 3.2.2 Multi-Head Attention section) where “the encoder contains self-attention layers. In a self-attention layer all of the keys, values, and queries come from the same place, in this case, the output of the previous layer in the encoder. Each position in the encoder can attend to all positions in the previous layer of the encoder” (Vaswani, page 5, 3.2.3 Applications of Attention in our Model, 2nd bullet point). Examiner notes that the attended input sequence is the output values. Examiner further notes that the encoder is the shared attention mechanism since the encoder can attend to all positions in the previous layer of the encoder.
On page 12, Applicant argues:
Regarding features that relate to selecting expert feed-forward neural networks, the Office Action acknowledges on page 7 that Vaswani does not recite these features and instead cites a portion of Non-Patent Literature Shazeer that states "Figure I: A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network." On page 2.
While this cited portion of Non-Patent Literature Shazeer recites "select[ing] two experts to perform computations," it has not been shown to disclose or suggest selecting different subsets of expert feed-forward neural networks for "a first attended layer input" and "a second attended layer input," where the attended layer inputs are generated by applying a shared query-key-value (QKV) attention mechanism that is same across all of the one or more positions in the input sequence.
Rather, Non-Patent Literature Shazeer merely teaches that "The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position." Id, on page 3.
Regarding the Applicant’s argument that the prior art of record does not disclose “selecting, from the plurality of expert feed-forward neural networks, one or more expert feed-forward neural networks based at least on the respective gate scores, wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence”, Examiner respectfully disagrees. Specifically, Examiner notes that Vaswani in view of Shazeer teaches this. Vaswani specifically teaches the QKV attention mechanism and Shazeer teaches an attention mechanism that selects experts.
Further, Shazeer discloses “wherein a first proper subset of expert feed-forward neural networks selected for a first attended layer input at a first position in the input sequence is different from a second proper subset of expert feed-forward neural networks selected for a second attended layer input at a second position in the input sequence” (Shazeer, page 2, 1st paragraph, “The MoE is called once for each position in the text, selecting a potentially different combination of experts at each position” and Shazeer, page 2, Figure 1,
PNG
media_image1.png
343
624
media_image1.png
Greyscale
)
Examiner notes that within Figure 1, Expert 2 is the first proper subset and G(x)2 is the first attended layer input at a first position in the input sequence. Expert n-1 is the second proper subset and G(x)(n-1) is the second attended layer input at a second position in the input sequence. Examiner further notes that the attended layer input is x in Figure 1.
On page 12, Applicant argues:
Accordingly, Applicant respectfully submits that claim 1 and its dependent claims are in condition for allowance. Independent claims 9 and 13 and their respective dependent claims are allowable for corresponding reasons.
Regarding the Applicant’s argument that the dependent claims are allowable at least due in part to their dependency on the independent claims, the Examiner respectfully disagrees and notes the instant rejections and response to arguments regarding the independent claims above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Aljundi et al. (“Expert Gate: Lifelong Learning with a Network of Experts”) also describes expert neural networks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KAITLYN R LAU whose telephone number is (571)272-1429. The examiner can normally be reached Monday - Thursday: 8:00 am - 6:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.L./ Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
