CTNF 18/477,546 CTNF 94181 DETAILED ACTION This action is in response to the Applicant Response filed 28 September 2023 for application 18/477,546 filed 28 September 2023. Claim(s) 1-21 is/are pending. Claim(s) 1-20 is/are rejected. 12-151-07 AIA 07-97 12-51-07 Claim (s) 21 is/are allowed. Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Objections 07-29-01 AIA Claim (s) 1-20 is/are objected to because of the following informalities: Claim 1, line 10, sublayer should, for consistency, read “sub-layer” Claim 5, line 3, the period at the end of the line should be a comma Claim 12, line 3, conditional computation sub-layer should read “conditional computational layer” Claim 14, line 4, conditional computation sub-layer should read “conditional computational layer” Claim 2-20 are objected to, due to their dependence, either directly or indirectly, on claims 1, 5, 12, 14 Appropriate correction is required. Claim Rejections - 35 USC § 112 07-30-02 AIA The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. 07-34-01 Claim 14 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 14 recites wherein each expert neural network while failing to provide a proper antecedent basis for the term. Neither claim 14 nor its parent claim, claim 1, recite claim language for “expert neural networks.” It is suggested that the claim be amended to depend from either claim 12 or claim 13. Clarification or correction is required. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claim(s) 1-20 is/are rejected under 35 U.S.C. 101, because the claim(s) is/are directed to an abstract idea, and because the claim elements, whether considered individually or in combination, do not amount to significantly more than the abstract idea, see Alice Corporation Pty. Ltd. V. CLS Bank International et al. , 573 US 208 (2014). Regarding claim 1 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 1 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of generating a plurality of candidate neural networks for performing the machine learning task, wherein each candidate neural network comprises a plurality of instances of a layer block, wherein the layer block comprises a sequence of a plurality of layers that each update each token in the input sequence , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. The limitation of wherein the generating comprises, for each candidate neural network, selecting a respective type for each of the plurality of layers from a set of layer types that comprises: (i) A temporal mixture sublayer that captures relationships among the tokens in the input sequence, (ii) A dense feed-forward sub-layer that operates independently on each token in the input sequence, and (iii) A conditional computational layer that operates independently on each token in the input sequence and that selects, for each token, one or more of a plurality of neural networks for processing the token , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. The limitation of determining a performance score for the trained candidate neural network that characterizes the performance of the trained candidate neural network on the machine learning task , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. The limitation of determining a final neural network for performing the machine learning task based at least on the performance scores for the candidate neural networks , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites additional element(s) – machine learning task, plurality of (trained) candidate neural networks, temporal mixture sublayer, dense feed-forward sub-layer, conditional computational layer, one or more of a plurality of neural networks, final neural network . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). The claim recites receiving training data for a machine learning task that requires processing an input sequence of tokens to generate a network output for the input sequence of tokens , which is simply receiving data recited at a high level of generality. This is nothing more than insignificant extra-solution activity (MPEP 2106.05(g)). The claim recites for each of the candidate neural networks, training the candidate neural network on at least a portion of the training data to generate a trained candidate neural network which is simply generic training to perform the abstract idea of model generation and amounts to mere instructions to apply the exception (MPEP 2106.05(f)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: receiving data amount(s) to no more than insignificant extra-solution activity (MPEP 2106.05(g)), wherein the insignificant extra-solution activity is the well-understood routine and conventional activit(y/ies) of receiving or transmitting data over a network and/or storing and retrieving information in memory (MPEP 2016.05(d)) machine learning task, plurality of (trained) candidate neural networks, temporal mixture sublayer, dense feed-forward sub-layer, conditional computational layer, one or more of a plurality of neural networks, final neural network amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 2 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 2 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein generating the plurality of candidate neural networks comprises performing an evolutionary search over a search space over candidate neural networks , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites additional element(s) – evolutionary search . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: evolutionary search amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 3 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 3 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 2 is applicable here since claim 3 carries out the method of claim 2 but for the recitation of additional element(s) of wherein the evolutionary search is guided by the performance scores for the candidate neural network subject to one or more constraints . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the evolutionary search and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the evolutionary search do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 4 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 4 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 4 carries out the method of claim 1 but for the recitation of additional element(s) of wherein training the candidate neural network on at least a portion of the training data comprises: training each candidate neural network on a same set of one or more first target hardware devices for a same amount of wall clock time . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites wherein training the candidate neural network on at least a portion of the training data comprises: training each candidate neural network on a same set of one or more first target hardware devices for a same amount of wall clock time which is simply additional information regarding the training, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). The claim recites additional element(s) – one or more first target hardware devices . The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: one or more first target hardware devices amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)) additional information regarding the training do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 5 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 5 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of determining, for each candidate neural network, whether a step time of the candidate neural network satisfies one or more criteria , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. The limitation of wherein determining a final neural network for performing the machine learning task based at least on the performance scores for the candidate neural networks comprises: selecting, from the candidate neural networks that have step times that satisfy the one or more criteria, the candidate neural network having a best performance on the machine learning task , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements which integrate the abstract idea into a practical application and, therefore, does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the claim does not recite any additional elements which provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 6 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 6 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 5 is applicable here since claim 6 carries out the method of claim 5 but for the recitation of additional element(s) of wherein the step time of the candidate neural network measures a time required for the candidate neural network to generate a respective network output for each input sequence in a set of one or more input sequences when deployed on a set one or more second target hardware devices . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites wherein the step time of the candidate neural network measures a time required for the candidate neural network to generate a respective network output for each input sequence in a set of one or more input sequences when deployed on a set one or more second target hardware devices which is simply additional information regarding the step time, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). The claim recites additional element(s) – one or more second target hardware devices . The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: one or more second target hardware devices amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)) additional information regarding the step time do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 7 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 7 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 5 is applicable here since claim 7 carries out the method of claim 5 but for the recitation of additional element(s) of wherein the one or more criteria include a first criterion requiring that the step time of the candidate neural network be less than a baseline step time of a baseline neural network for the machine learning task by at least a threshold amount of time . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the step time and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the step time do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 8 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 8 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 7 is applicable here since claim 8 carries out the method of claim 7 but for the recitation of additional element(s) of wherein the threshold amount of time is zero . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the step time and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the step time do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 9 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 9 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein determining a performance score for the trained candidate neural network comprises: determining a validation accuracy of the trained candidate neural network on a set of validation data for the machine learning task , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements which integrate the abstract idea into a practical application and, therefore, does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the claim does not recite any additional elements which provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 10 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 10 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein the generating comprises, for each candidate neural network, selecting, for each of the plurality of layers one or more respective dimension values that specify a dimensionality of the tokens when processed by each of the components of the layer , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements which integrate the abstract idea into a practical application and, therefore, does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the claim does not recite any additional elements which provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 11 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 11 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 10 is applicable here since claim 11 carries out the method of claim 10 but for the recitation of additional element(s) of wherein different layer types in the set of layer types have different sets of possible values . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. In particular, the claim recites additional information regarding the layers and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of additional information regarding the layers do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Not applying the exception in a meaningful way does not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 12 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 12 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein the generating comprises, for each candidate neural network, selecting, for each of the plurality of layers for which the selected layer type is the conditional computation sub-layer, a respective routing scheme for routing tokens to expert neural networks from a set of possible routing schemes , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites additional element(s) – expert neural networks . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: expert neural networks amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 13 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 13 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 12 is applicable here since claim 13 carries out the method of claim 12 but for the recitation of additional element(s) of wherein the set of possible routing schemes comprises token-based routing and expert-based routing . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites wherein the set of possible routing schemes comprises token-based routing and expert-based routing which is simply additional information regarding the routing scheme, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). The claim recites additional element(s) – token-based routing, expert-based routing . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: token-based routing, expert-based routing amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) additional information regarding the routing scheme do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 14 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 14 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein the generating comprises, for each candidate neural network, selecting, for each of the plurality of layers for which the selected layer type is the conditional computation sub-layer, a capacity factor from a set of possible capacity factors that each specify a different maximum number of tokens from the input sequence that can be routed to any one of the plurality of devices , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites additional element(s) – plurality of devices . The additional element(s) is/are recited at a high-level of generality (i.e., as generic computer components performing generic computer functions of executing instructions on the computers) such that it amounts to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)). The claim recites wherein each expert neural network is deployed on a respective device from a plurality of devices which is simply additional information regarding the expert neural networks, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: plurality of devices amount(s) to no more than mere instructions to apply the exception using generic computer components (MPEP 2106.05(b)) additional information regarding the expert neural networks do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 15 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 15 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein the generating comprises, for each candidate neural network, selecting, for each of the plurality of layers, a respective activation function to be applied within the layer from a set of possible activation functions , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites additional element(s) – set of possible activation functions . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: set of possible activation functions amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 16 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 16 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of providing data specifying the final neural network for use in performing the machine learning task , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements which integrate the abstract idea into a practical application and, therefore, does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the claim does not recite any additional elements which provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 17 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 17 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 17 carries out the method of claim 1 but for the recitation of additional element(s) of receiving a new input sequence; and performing the machine learning task on the new input sequence by processing the new input sequence using the final neural network . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites receiving a new input sequence , which is simply receiving data recited at a high level of generality. This is nothing more than insignificant extra-solution activity (MPEP 2106.05(g)). The claim recites performing the machine learning task on the new input sequence by processing the new input sequence using the final neural network which is simply applying the model recited at a high level of generality and amounts to the recitation of the words “apply it” (or an equivalent) or amounts to no more than mere instructions to implement an abstract idea or other exception on a computer (MPEP 2106.05(f)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: applying the model amount(s) to no more than mere instructions to apply the exception (MPEP 2106.05(f)) receiving data amount(s) to no more than insignificant extra-solution activity (MPEP 2106.05(g)), wherein the insignificant extra-solution activity is the well-understood routine and conventional activit(y/ies) of receiving or transmitting data over a network and/or storing and retrieving information in memory (MPEP 2016.05(d)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 18 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 18 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 1 is applicable here since claim 18 carries out the method of claim 1 but for the recitation of additional element(s) of wherein the temporal mixture sub-layer is a self-attention sub-layer that applies self-attention over the tokens . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites wherein the temporal mixture sub-layer is a self-attention sub-layer that applies self-attention over the tokens which is simply additional information regarding the model, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). The claim recites additional element(s) – self-attention sub-layer, self-attention . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: self-attention sub-layer, self-attention amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) additional information regarding the model do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 19 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 19 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The Step 2A Prong One Analysis for claim 18 is applicable here since claim 19 carries out the method of claim 18 but for the recitation of additional element(s) of wherein the self-attention is causal self-attention . Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim recites wherein the self-attention is causal self-attention which is simply additional information regarding the model, and the element(s) do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)). The claim recites additional element(s) – causal self-attention . The additional element(s) is/are recited at a high-level of generality such that it amounts to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)). Accordingly, the additional element(s) do(es) not integrate the abstract idea into a practical application because the additional element(s) do(es) not impose any meaningful limits on practicing the abstract idea, and, therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element(s) of: causal self-attention amount(s) to no more than indicating a field of use or technological environment in which to apply the judicial exception (MPEP 2106.05(h)) additional information regarding the model do(es) not apply the exception in a meaningful way (MPEP 2106.05(e)) The additional element(s) do(es) not provide an inventive concept, and, therefore, the claim is not patent eligible. Regarding claim 20 , the claim is rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. Step 1 Analysis: Claim 20 is directed to a method, which is directed to a process, one of the statutory categories. Step 2A Prong One Analysis: The claim recites a(n) method. The limitation of wherein the generating comprises, for each candidate neural network, selecting, for each of the plurality of layers for which the selected layer type is the temporal mixture sub-layer, a number of attention heads to be included in the temporal mixture sub-layer from a set of possible numbers , as drafted, is a process that, under its broadest reasonable interpretation, covers a mental process. The limitation is directed to observation, evaluation, judgment and opinion and is a process capable of being performed by a human mentally or using pen and paper. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind, then it falls within the "Mental Processes" grouping. Accordingly, the claim recites an abstract idea. Step 2A Prong Two Analysis: With respect to the abstract idea, the judicial exception is not integrated into a practical application. The claim does not recite any additional elements which integrate the abstract idea into a practical application and, therefore, does not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea. Step 2B Analysis: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the claim does not recite any additional elements which provide an inventive concept, and, therefore, the claim is not patent eligible. Allowable Subject Matter 12-151-07 AIA 07-97 12-51-07 Claim (s) 21 is/are allowed. The following is an examiner’s statement of reasons for allowance: in view of claim(s) 21 and further search, claim(s) 21 is/are considered allowable since when reading the claim(s) in light of the specification, as per MPEP 2111.01, none of the references of record either alone or in combination fairly disclose or suggest the combination of limitations specified in the independent claim(s). Regarding the limitation(s) of claim(s) 21 which do(es) not appear to be taught by the prior art: Zhang et al. teaches conditional language specific routing. Lepikhin et al. (NPL) teaches scaling multilingual transformer models. Fedus et al. teaches simplifying MoE model routing. Darvish Rouhani et al. teaches a model comprising a sequence of transformer blocks comprising attention and MoE layers. Lepikhin et al. (Patent) teaches attention neural networks with conditional computation. Wierstra et al., Gesmundo, Li et al., and Tan teach neural architecture search, including with sequences of blocks comprising sequences of layers. Kitaev et al and Shazeer teach attention neural networks. However, the claim(s) in the application is/are deemed to be directed to a nonobvious improvement over the prior art of record. As noted above, the independent claim(s) comprise(s) a model comprising a plurality of blocks, each block having a sequence of layers comprising a temporal layer, a dense feed-forward layer and a conditional computational layer. Therefore, the cited/applied prior art fails to teach or suggest each and every feature of each of the combination of features recited in independent claim(s) 21. When taken into context, the claim(s) as a whole were not uncovered in the prior art; i.e., all dependent claims which depend from claim(s) 21 are allowed as they depend upon an allowable independent claim. Conclusion Any inquiry concerning this communication or earlier communication from the examiner should be directed to MARSHALL WERNER whose telephone number is (469) 295-9143. The examiner can normally be reached on Monday – Thursday 7:30 AM – 4:30 PM ET. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar, can be reached at (571) 272-7796. The fax number for the organization where this application or proceeding is assigned is (571) 273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MARSHALL L WERNER/ Primary Examiner, Art Unit 2125 Application/Control Number: 18/477,546 Page 2 Art Unit: 2125 Application/Control Number: 18/477,546 Page 3 Art Unit: 2125 Application/Control Number: 18/477,546 Page 4 Art Unit: 2125 Application/Control Number: 18/477,546 Page 5 Art Unit: 2125 Application/Control Number: 18/477,546 Page 6 Art Unit: 2125 Application/Control Number: 18/477,546 Page 7 Art Unit: 2125 Application/Control Number: 18/477,546 Page 8 Art Unit: 2125 Application/Control Number: 18/477,546 Page 10 Art Unit: 2125 Application/Control Number: 18/477,546 Page 11 Art Unit: 2125 Application/Control Number: 18/477,546 Page 12 Art Unit: 2125 Application/Control Number: 18/477,546 Page 13 Art Unit: 2125 Application/Control Number: 18/477,546 Page 14 Art Unit: 2125 Application/Control Number: 18/477,546 Page 15 Art Unit: 2125 Application/Control Number: 18/477,546 Page 16 Art Unit: 2125 Application/Control Number: 18/477,546 Page 17 Art Unit: 2125 Application/Control Number: 18/477,546 Page 18 Art Unit: 2125 Application/Control Number: 18/477,546 Page 20 Art Unit: 2125 Application/Control Number: 18/477,546 Page 21 Art Unit: 2125 Application/Control Number: 18/477,546 Page 23 Art Unit: 2125 Application/Control Number: 18/477,546 Page 24 Art Unit: 2125 Application/Control Number: 18/477,546 Page 25 Art Unit: 2125 Application/Control Number: 18/477,546 Page 26 Art Unit: 2125 Application/Control Number: 18/477,546 Page 28 Art Unit: 2125 Application/Control Number: 18/477,546 Page 29 Art Unit: 2125 Application/Control Number: 18/477,546 Page 30 Art Unit: 2125
Prosecution Insights