Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
Providing supporting paragraph(s) for each limitation of amended/new claim(s) in Remarks is strongly requested for clear and definite claim interpretations by Examiner (e.g., to avoid rejections under 35 U.S.C § 112(a) “Lack of written description”)
Applicant can schedule interviews (via Automated Interview Request (AIR)) at any stage of the prosecution (e.g., Non-Final, Final, and After-Final) to discuss any issues related to, for example, rejections under 35 U.S.C § 101 and § 102/103, for moving toward allowance.
Priority
Acknowledgment is made of applicant's claim for the present application filed on 01/29/2024.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 3, 7, 11, 15, 18, 20 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim(s) 3 recite(s) the limitation “the number of batches” (line 1). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “a number of batches”, or something else. For the purposes of examination, “a number of batches” is used. In addition, claim(s) 11, 18 is/are rejected for the same reason.
Claim(s) 3 recite(s) the limitation “the number of computing units” (line 2). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “a number of computing units”, or something else. For the purposes of examination, “a number of computing units” is used. In addition, claim(s) 11, 18 is/are rejected for the same reason.
Claim(s) 7 recite(s) the limitation “the same chip package” (line 2). There is insufficient antecedent basis for this limitation in the claim. It is not clear what it is referring to. It appears it may need to read “a same chip package”, or something else. For the purposes of examination, “a same chip package” is used. In addition, claim(s) 15, 20 is/are rejected for the same reason.
Claim(s) 3, 7, 11, 15, 18, 20 each recite(s) limitations that raise issues of indefiniteness as set forth above, and their dependent claims are rejected at least based on their direct and/or indirect dependency from the claims listed above. Appropriate explanation and/or amendment is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1
Step 1: “Is the claim to a process, machine, manufacture, or composition of matter?”
The claim is directed to a method. Therefore, yes.
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?”
determining, …, a split for the batches of data across a plurality of computing units associated with activated experts based on the MoE index; (i.e., mental process)
aggregating, …, the processed batches of data to generate a response for the MoE request (i.e., mental process)
The claim is directed to an abstract idea. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
receiving, by one or more processors, an MoE request comprising a plurality of batches of data and an MoE index; (insignificant extra-solution activity of receiving data, see MPEP 2106.05(g)) (well-understood, routine, and conventional generic computer and/or model, see MPEP 2106.05(f))
by the one or more processors (well-understood, routine, and conventional generic computer and/or model, see MPEP 2106.05(f))
distributing in parallel, by the one or more processors, the batches of data across the plurality of computing units based on the split; (insignificant extra-solution activity of transmitting data, see MPEP 2106.05(g))
receiving, by the one or more processors, processed batches of data from the plurality of computing units; and (insignificant extra-solution activity of receiving data, see MPEP 2106.05(g)) (well-understood, routine, and conventional generic computer and/or model, see MPEP 2106.05(f))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Specifically, the claimed inventions simply append well-understood, routine and conventional activities previously known to the industry, both when viewed independently and as an ordered combination, specified at a high level of generality, to the judicial exception, (e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry). Therefore, no.
Regarding claim 2
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
outputting, by the one or more processors, the response for the MoE request (insignificant extra-solution activity of mere data output, see MPEP 2106.05(g)) (well-understood, routine, and conventional generic computer and/or model, see MPEP 2106.05(f))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 3
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein the split comprises the number of batches in the plurality of batches of data divided by the number of computing units in the plurality of computing units (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 4
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein the split is based on at least one of respective sizes of the batches of data or respective performances of the computing units (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 5
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein the MoE index indicates which expert networks to activate when processing the MoE request (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 6
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein each processed batch of data comprises a result from processing by the respective computing unit (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 7
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein the one or more processors are implemented on the same chip package as the plurality of computing units (well-understood, routine, and conventional generic computer and/or model, see MPEP 2106.05(f))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 8
Step 2A Prong 1: “Does the claim recite an abstract idea, law of nature, or natural phenomenon?” The claim recites the abstract idea identified above regarding claim 1. Therefore, yes.
Step 2A Prong 2: “Does the claim recite additional elements that integrate the judicial exception into a practical application?” The following elements are directed to additional elements:
wherein the one or more processors are optically connected to the plurality of computing units (a particular type or source of model/data, Field of Use and Technological Environment, see MPEP 2106.05(h))
Therefore, no.
Step 2B: “Does the claim recite additional elements that amount to significantly more than the judicial exception?”
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Therefore, no.
Regarding claim 9
The claim is rejected for the reasons set forth in the rejection of Claim 1 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 10
The claim is rejected for the reasons set forth in the rejection of Claim 2 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 11
The claim is rejected for the reasons set forth in the rejection of Claim 3 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 12
The claim is rejected for the reasons set forth in the rejection of Claim 4 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 13
The claim is rejected for the reasons set forth in the rejection of Claim 5 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 14
The claim is rejected for the reasons set forth in the rejection of Claim 6 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 15
The claim is rejected for the reasons set forth in the rejection of Claim 7 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 16
The claim is rejected for the reasons set forth in the rejection of Claim 8 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 17
The claim is rejected for the reasons set forth in the rejection of Claim 1 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 18
The claim is rejected for the reasons set forth in the rejection of Claim 3 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 19
The claim is rejected for the reasons set forth in the rejection of Claim 4 under 35 U.S.C. 101, mutatis mutandis.
Regarding claim 20
The claim is rejected for the reasons set forth in the rejection of Claim 7 or claim 8 under 35 U.S.C. 101, mutatis mutandis.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-6, 9-14, 17-19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Shazeer et al. (OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER)
Regarding claim 1
Shazeer teaches
A method for serving mixture of expert (MoE) models comprising:
(Shazeer [fig(s) 1] “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network” [sec(s) Abs] “We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks.” [sec(s) 1] “Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure 1). All parts of the network are trained jointly by back-propagation.” [sec(s) 2] “The Mixture-of-Experts (MoE) layer consists of a set of n “expert networks" E1,··· ,En, and a “gating network" G whose output is a sparse n-dimensional vector.”;)
receiving, by one or more processors, an MoE request comprising a plurality of batches of data and an MoE index;
(Shazeer [fig(s) 1] “A Mixture of Experts (MoE) layer embedded within a recurrent language model. In this case, the sparse gating function selects two experts to perform computations. Their outputs are modulated by the outputs of the gating network” [sec(s) Abs] “A trainable gating network determines a sparse combination of these experts to use for each example.” [sec(s) 2] “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:
PNG
media_image1.png
188
553
media_image1.png
Greyscale
(1)” [sec(s) 2.1] “We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.
PNG
media_image2.png
300
1211
media_image2.png
Greyscale
” [sec(s) 3.1] “On modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates.” [sec(s) 5.1] “We trained our models using TensorFlow (Abadi et al., 2016) on clusters containing 16-32 Tesla K40 GPUs. For each of our models, we determine computational efficiency in TFLOPS/GPU by dividing the number of floating point operations required to process one training batch by the observed step time and the number of GPUs in the cluster.”;)
determining, by the one or more processors, a split for the batches of data across a plurality of computing units associated with activated experts based on the MoE index;
(Shazeer [sec(s) 3] “We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts). If the model is distributed over d devices, and each device processes a batch of size b, each expert receives a batch of approximately kbd/n examples. Thus, we achieve a factor of d improvement in expert batch size.”;)
distributing in parallel, by the one or more processors, the batches of data across the plurality of computing units based on the split;
(Shazeer [sec(s) 3.1] “In our technique, these different batches run synchronously so that they can be combined for the MoE layer. We distribute the standard layers of the model and the gating network according to conventional data-parallel schemes, but keep only one shared copy of each expert. Each expert in the MoE layer receives a combined batch consisting of the relevant examples from all of the data-parallel input batches. The same set of devices function as data-parallel replicas (for the standard layers and the gating networks) and as model-parallel shards (each hosting a subset of the experts).” [sec(s) 3.2] “Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert’s computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size×output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.”;)
receiving, by the one or more processors, processed batches of data from the plurality of computing units; and
(Shazeer [sec(s) 3.2] “Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert’s computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size×output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.”;)
aggregating, by the one or more processors, the processed batches of data to generate a response for the MoE request.
(Shazeer [sec(s) 2] “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:
PNG
media_image1.png
188
553
media_image1.png
Greyscale
(1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network.” [sec(s) 2.1] “We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.
PNG
media_image2.png
300
1211
media_image2.png
Greyscale
”;)
Regarding claim 2
The combination of Shazeer teaches claim 1.
Shazeer further teaches
further comprising outputting, by the one or more processors, the response for the MoE request.
(Shazeer [sec(s) 2] “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:
PNG
media_image1.png
188
553
media_image1.png
Greyscale
(1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network.” [sec(s) 3.2] “Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert’s computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size×output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.”;)
Regarding claim 3
The combination of Shazeer teaches claim 1.
Shazeer further teaches
wherein the split comprises the number of batches in the plurality of batches of data divided by the number of computing units in the plurality of computing units.
(Shazeer [sec(s) 3.1] “On modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb/n << b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size.”;)
Regarding claim 4
The combination of Shazeer teaches claim 1.
Shazeer further teaches
wherein the split is based on at least one of respective sizes of the batches of data or respective performances of the computing units.
(Shazeer [sec(s) 3.2] “Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert’s computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device. For GPUs, this may be thousands to one. In our experiments, we use experts with one hidden layer containing thousands of RELU-activated units. Since the weight matrices in the expert have sizes input_size×hidden_size and hidden_size×output_size, the ratio of computation to input and output is equal to the size of the hidden layer. Conveniently, we can increase computational efficiency simply by using a larger hidden layer, or more hidden layers.” [sec(s) 3.1] “On modern CPUs and GPUs, large batch sizes are necessary for computational efficiency, so as to amortize the overhead of parameter loads and updates. If the gating network chooses k out of n experts for each example, then for a batch of b examples, each expert receives a much smaller batch of approximately kb/n << b examples. This causes a naive MoE implementation to become very inefficient as the number of experts increases. The solution to this shrinking batch problem is to make the original batch size as large as possible. However, batch size tends to be limited by the memory necessary to store activations between the forwards and backwards passes. We propose the following techniques for increasing the batch size.”;)
Regarding claim 5
The combination of Shazeer teaches claim 1.
Shazeer further teaches
wherein the MoE index indicates which expert networks to activate when processing the MoE request.
(Shazeer [sec(s) 2] “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:
PNG
media_image1.png
188
553
media_image1.png
Greyscale
(1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network.” [sec(s) 2.1] “We add two components to the Softmax gating network: sparsity and noise. Before taking the softmax function, we add tunable Gaussian noise, then keep only the top k values, setting the rest to −∞ (which causes the corresponding gate values to equal 0). The sparsity serves to save computation, as described above. While this form of sparsity creates some theoretically scary discontinuities in the output of gating function, we have not yet observed this to be a problem in practice. The noise term helps with load balancing, as will be discussed in Appendix A. The amount of noise per component is controlled by a second trainable weight matrix Wnoise.”;)
Regarding claim 6
The combination of Shazeer teaches claim 1.
Shazeer further teaches
wherein each processed batch of data comprises a result from processing by the respective computing unit.
(Shazeer [sec(s) 2] “Let us denote by G(x) and Ei(x) the output of the gating network and the output of the i-th expert network for a given input x. The output y of the MoE module can be written as follows:
PNG
media_image1.png
188
553
media_image1.png
Greyscale
(1) We save computation based on the sparsity of the output of G(x). Wherever G(x)i = 0, we need not compute Ei(x). In our experiments, we have up to thousands of experts, but only need to evaluate a handful of them for every example. If the number of experts is very large, we can reduce the branching factor by using a two-level hierarchical MoE. In a hierarchical MoE, a primary gating network chooses a sparse weighted combination of “experts", each of which is itself a secondary mixture-of-experts with its own gating network.” [sec(s) 3.2] “Since the experts are stationary (see above) and the number of gating parameters is small, most of the communication involves sending the inputs and outputs of the experts across the network. To maintain computational efficiency, the ratio of an expert’s computation to the size of its input and output must exceed the ratio of computational to network capacity of the computing device.”;)
Regarding claim 9
The claim is rejected for the reasons set forth in the rejection of Claim 1.
Regarding claim 10
The claim is rejected for the reasons set forth in the rejection of Claim 2.
Regarding claim 11
The claim is rejected for the reasons set forth in the rejection of Claim 3.
Regarding claim 12
The claim is rejected for the reasons set forth in the rejection of Claim 4.
Regarding claim 13
The claim is rejected for the reasons set forth in the rejection of Claim 5.
Regarding claim 14
The claim is rejected for the reasons set forth in the rejection of Claim 6.
Regarding claim 17
The claim is rejected for the reasons set forth in the rejection of Claim 1.
Regarding claim 18
The claim is rejected for the reasons set forth in the rejection of Claim 3.
Regarding claim 19
The claim is rejected for the reasons set forth in the rejection of Claim 4.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 7, 15, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer et al. (OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER) in view of Arunkumar et al. (MCM-GPU: Multi-chip-module GPUs for continued performance scalability)
Regarding claim 7
The combination of Shazeer teaches claim 1.
However, the combination of Shazeer does not appear to explicitly teach:
wherein the one or more processors are implemented on the same chip package as the plurality of computing units.
Arunkumar teaches
wherein the one or more processors are implemented on the same chip package as the plurality of computing units.
(Arunkumar [sec(s) Abs] “To address this need, in this paper we demonstrate that package-level integration of multiple GPU modules to build larger logical GPUs can enable continuous performance scaling beyond Moore’s law.” [sec(s) 1] “Our proposal aggregates multiple GPU Modules (GPMs) within a single package as illustrated in Figure 1.” [fig(s) 1] “MCM-GPU: Aggregating GPU modules and DRAM on a single package” [sec(s) 3.1] “In this paper we propose the MCM-GPU as a collection of GPMs that share resources and are presented to software and programmers as a single monolithic GPU. Pooled hardware resources, and shared I/O are concentrated in a shared on-package module (the SYS + I/O module shown in Figure 1).”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Shazeer with the same chip package of Arunkumar.
One of ordinary skill in the art would have been motived to combine in order to improve GPU performance scaling at a package level, beyond what is possible today.
(Arunkumar [sec(s) 8] “Many of today’s important GPU applications scale well with GPU compute capabilities and future progress in many fields such as exascale computing and artificial intelligence will depend on continued GPU performance growth. The greatest challenge towards building more powerful GPUs comes from reaching the end of transistor density scaling, combined with the inability to further grow the area of a single monolithic GPU die. In this paper we propose MCM-GPU, a novel GPU architecture that extends GPU performance scaling at a package level, beyond what is possible today.”)
Regarding claim 15
The claim is rejected for the reasons set forth in the rejection of Claim 1.
Regarding claim 20
The claim is rejected for the reasons set forth in the rejection of Claim 7 or claim 8.
Claim(s) 8, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shazeer et al. (OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER) in view of Beausoleil et al. (Photonic Architectures for High-Performance Data Centers)
Regarding claim 8
The combination of Shazeer teaches claim 1.
However, the combination of Shazeer does not appear to explicitly teach:
wherein the one or more processors are optically connected to the plurality of computing units.
Beausoleil teaches
wherein the one or more processors are optically connected to the plurality of computing units.
(Beausoleil [sec(s) II.A] “The DOE exascale requirement specifies a minimum node bandwidth of 400 Gbytes/s, which might equate to eight parallel networks assuming 400-Gb/s optical-fiber-based communication ports. The networks will exploit high-radix routers connected in a HyperX topology [11] to yield a low-diameter network.” [sec(s) II.B] “An additional advantage is that such links have very high bandwidth density; up to 640 Gb/s per fiber assuming 64 wavelengths at 10 Gb/s.” [sec(s) IV.A] “Systems and applications requiring bandwidths approaching 1 Tb/s per optomechanical connector (e.g., a single fiber optical core affixed to a chip or board) will require DWDM and, therefore, resonant filters such as microrings or microdisks. The small size (and correspondingly low capacitance and resistance) of these resonant devices allows them to be operated as electrooptic modulators at the low voltages and low driving power required by integrated circuit technologies beyond 22-nm CMOS.”;)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Shazeer with the optical connection of Beausoleil.
One of ordinary skill in the art would have been motived to combine in order to enable improvements in the efficiency of large-scale computing systems, particularly to support applications that require exascale algorithmic performance.
(Beausoleil [sec(s) Abs] “Over the next decade, significant progress must be made in research on computer architectures that enable unprecedented improvements in the efficiency of large-scale computing systems, particularly to support applications that require exascale algorithmic performance. Here, we review the performance requirements for both high-performance computing systems and data centers, and show that it will be critical to exploit photonic devices for interconnect applications to meet these expectations. In the long term, CMOS-compatible fabrication technologies promise a “Moore’s Law for photonics” that could completely change the economics of integrated optics and high-performance computing for defense, security, scientific, and consumer applications”)
Regarding claim 16
The claim is rejected for the reasons set forth in the rejection of Claim 2.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SEHWAN KIM whose telephone number is (571)270-7409. The examiner can normally be reached Mon - Fri 9:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SEHWAN KIM/Examiner, Art Unit 2129
Prosecution Insights