DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 01/26/2026 has been entered. Claims 1-2, 4-9, 11-16, and 18-20 are pending. The amendments have overcome the 101 rejection on claims 15-16, 18-20 for being directed to non-statutory subject matter.
Response to Arguments
Applicant's arguments with respect to 35 U.S.C § 101 filed 01/26/2026 have been fully
considered but they are not persuasive.
Applicant argues that the claimed invention do not recite a mental process (pages 7-9 of applicant’s arguments). Applicant states that the claimed invention cannot be performed mentally or with pen and paper due to the sheer volume of data requiring computer processing. The examiner respectfully disagrees. As the claims are currently written, in their broadest reasonable interpretation, the limitations “evaluating, by the compute device, TBALM architectures to identify architectures that satisfy the maximum latency constraint resulting in potential TBALM architectures;” and “automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy, a TBALM architecture of the potential TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture” could both be performed in the human mind or with pen and paper. These limitations recite choosing architectures that satisfy a constraint and of those architectures picking the on with greatest number of decoder parameters. This, in its broadest reasonable interpretation, can be performed in the human mind or with pen and paper. Applicant also argues that the claimed invention recite specific technical operation in neural architecture search, not abstract concepts. The invention using technical operation does not make the limitations above any less of a mental process.
Applicant argues that the claimed invention are integrated into a practical application because they provide a technological improvement to computer functionality (pages 9-12). Applicant states the improvement is the use of decoder parameters as a proxy for TBALM accuracy “enables faster architecture selection such that architectures can be evaluated based on parameter count without requiring full training of each candidate”. The applicant also states the Examiner is insufficient and misapplies MPEP 2106.05(a). The examiner respectfully disagrees. MPEP 2106.05(a) states “It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements.” The improvements mentioned above are coming from the mental process (the judicial exception) and alone cannot provide the improvement. The additional elements state receiving a request for a TBALM architecture, training a neural network which includes the identified TBALM and providing the trained neural network. None of these show the improvements mentioned above.
Finally the applicant argues that the claims include significantly more than any purported abstract idea (page 12-13). The applicant argues that the claimed invention amounts to significantly more because it is an improvement. The examiner disagrees for the same reasons listed above.
Thus the 101 rejection is maintained.
Applicant’s arguments with respect to 35 U.S.C § 103 filed 01/26/2026 have been
considered but are moot because the new ground of rejection does not rely on any reference applied in
the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2, 4-9, 11-16 & 18-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
101 Subject Matter Eligibility analysis
Step 1: Claims 1-14 are within the four statutory categories (a process, machine, manufacture or composition of matter). Claims 1-7 describe a process and claims 8-14 describe a machine.
With respect to claim 1:
Step 2A Prong 1: The claim recites an abstract idea enumerated in the 2019 PEG.
Evaluating,…, TBALM architectures to identify architectures that satisfy the maximum latency constraint resulting in potential TBALM architectures; (This is an abstract idea of a "Mental Process." The "evaluating" & “identify” step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The evaluation could be made manually by an individual).
Automatically selecting,… using a total number of decoder parameters of the architecture as a proxy for architecture accuracy, a TBALM architecture of the potential TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architecture (This is an abstract idea of a "Mental Process." The "selecting" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The selection could be made manually by an individual).
Step 2a Prong 2: The judicial exception is not integrated into a practical application
Additional elements:
receiving, at a compute device, a request for a transformer-based autoregressive language model (TBALM), the request specifying a maximum latency constraint; (This amounts to no more than mere instructions to “apply” the exception using a generic computer component).
By the compute device (This amounts to no more than mere instructions to “apply” the exception using a generic computer component).
Training a neural network (NN) model that includes the identified TBALM architecture; resulting in a trained NN (this limitation merely limits the judicial exception to a particular field of use).
Providing the trained NN to the requester (This amounts to no more than mere instructions to “apply” the exception using a generic computer component.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
As explained above the additional elements “receiving…”, “by the compute device” and “providing...” is recited in a generic level and they represent generic computer components to apply the abstract idea. Mere instructions to apply an exception cannot provide an inventive concept.
The additional element of “training a neural network…” is merely providing a particular field of use and cannot provide an inventive concept.
When considered in combination, these additional elements represent mere instructions to apply an exception, insignificant extra-solution activity and field of use, which do not provide an inventive concept.
When considered in combination, these additional elements represent insignificant extra-solution activity and mere instructions to apply an expectation, which do not provide an inventive concept.
Therefore, claim 1 is ineligible.
With respect to claim 2:
Step 2A Prong 1: claim 2, which incorporates the rejection of claim 1, recites an additional abstract idea:
The evaluating and automatically selecting includes identifying the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, (ii) satisfies the maximum amount of memory consumed, and (iii) has a greatest number of decoder parameters for the architectures that satisfy both the maximum latency constraint and maximum amount of memory consumed resulting in the identified TBALM architecture; (This is an abstract idea of a "Mental Process." The "evaluating" and “selecting” step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The evaluation could be made manually by an individual).
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the request further specifies a maximum amount of memory consumed by the TBALM during inference; (this limitation merely limits the judicial exception to a particular field of use.)
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element of “the request…” is merely providing a particular field of use and cannot provide an inventive concept.
Therefore, claim 2 is ineligible.
With respect to claim 4:
Step 2A Prong 1: claim 4, which incorporates the rejection of claim 1, recites an additional abstract idea:
the total number of decoder parameters includes weights and biases of only the decoder of the TBALM architecture; (This is an abstract idea of a "Mental Process." This step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind.)
Step 2a Prong 2: claim 4 does not recite any additional elements and thus cannot be integrated into a practical application
Step 2B: claim 4 does not recite an additional element.
Therefore, claim 4 is ineligible.
With respect to claim 5:
Step 2A Prong 1: claim 5, which incorporates the rejection of claim 4, recites an additional abstract idea:
the decoder parameters include, of the identified TBALM architecture: weights of attention heads; model dimensions; inner dimension of a feed forward network (FFN); and number of decoder layers; (This is an abstract idea of a "Mental Process." This step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind.)
Step 2a Prong 2: claim 5 does not recite any additional elements and thus cannot be integrated into a practical application
Step 2B: claim 5 does not recite an additional element.
Therefore, claim 5 is ineligible.
With respect to claim 6:
Step 2A Prong 1: claim 6, which incorporates the rejection of claim 1, recites an additional abstract idea:
generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve; (This is an abstract idea of a “Mental Process”. The “generating” in its broadest reasonable interpretation is capable of being performable in the human mind or on paper. For example, without reciting any specifics of the generating process, a person can create a pareto curve, and therefore, the generating is directed towards an abstract idea).
Step 2a Prong 2: The judicial exception is not integrated into a practical application.
the compute device is a client device; (this limitation merely limits the judicial exception to a particular field of use).
Step 2B: the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception
The additional element of “the compute device is a client device” is merely providing a particular field of use and cannot provide an inventive concept.
Therefore, claim 6 is ineligible.
With respect to claim 7:
Step 2A Prong 1: claim 7, which incorporates the rejection of claim 6, recites an additional abstract idea:
The automatically selecting the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency and (ii) has a greatest number of decoder parameters for the architectures that satisfy the maximum latency constraint includes selecting the TBALM corresponding to a point at a boundary of the generated pareto curve; (This is an abstract idea of a "Mental Process." The "selecting" step under its broadest reasonable interpretation, covers concepts that can be practically performed in the human mind. The selecting could be made manually by an individual).
Step 2a Prong 2: claim 7 does not recite any additional elements and thus cannot be integrated into a practical application.
Step 2B: claim 7 does not recite any additional elements.
Therefore, claim 7 is ineligible.
With respect to claim 8:
The claim recites similar limitations as corresponding to claim 1. Therefore, the same subject matter analysis that was utilized for claim 1, as described above, is equally applicable to claim 8.
Therefore, claim 8 is ineligible.
With respect to claim 9:
The claim recites similar limitations as corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 9.
Therefore, claim 9 is ineligible.
With respect to claim 11:
The claim recites similar limitations as corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 11.
Therefore, claim 11 is ineligible.
With respect to claim 12:
The claim recites similar limitations as corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 12.
Therefore, claim 12 is ineligible.
With respect to claim 13:
The claim recites similar limitations as corresponding to claim 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 13.
Therefore, claim 13 is ineligible.
With respect to claim 14:
The claim recites similar limitations as corresponding to claim 7. Therefore, the same subject matter analysis that was utilized for claim 7, as described above, is equally applicable to claim 14.
Therefore, claim 14 is ineligible.
With respect to claim 15:
The claim recites similar limitations as corresponding to claim 1. Therefore, the same subject matter analysis that was utilized for claim 1, as described above, is equally applicable to claim 15.
Therefore, claim 15 is ineligible.
With respect to claim 16:
The claim recites similar limitations as corresponding to claim 2. Therefore, the same subject matter analysis that was utilized for claim 2, as described above, is equally applicable to claim 16.
Therefore, claim 16 is ineligible.
With respect to claim 18:
The claim recites similar limitations as corresponding to claim 4. Therefore, the same subject matter analysis that was utilized for claim 4, as described above, is equally applicable to claim 18.
Therefore, claim 18 is ineligible.
With respect to claim 19:
The claim recites similar limitations as corresponding to claim 5. Therefore, the same subject matter analysis that was utilized for claim 5, as described above, is equally applicable to claim 19.
Therefore, claim 19 is ineligible.
With respect to claim 20:
The claim recites similar limitations as corresponding to claim 6. Therefore, the same subject matter analysis that was utilized for claim 6, as described above, is equally applicable to claim 20.
Therefore, claim 20 is ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 8-9, 11-12, 15-16 & 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Arikawa (US 20230385603 A1) in view of Henighan (NPL: ‘Scaling Laws for Autoregressive Generative Modeling’ (2020)).
Regarding claim 1, Arikawa teaches:
A computer-implemented method; ([0021] “A neural architecture search method”).
receiving, at a compute device, a request for a neural network, the request specifying a maximum latency constraint; ([0049] “One feature of the neural architecture search system of this embodiment is that the architecture of the neural network is searched for so as to satisfy a constraint condition specified by a user as an external setting parameter P1 ” and [0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
evaluating, by the computer device neural network architectures to identify architectures that satisfy the maximum latency constraint resulting in potential neural network architectures; ([0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
training a neural network (NN) model that includes the identified TBALM architecture resulting in a trained NN ([0083] ”Next, the learning engine unit 3 inputs the training data D1 to the model of the neural network prescribed by the model information M2 and performs learning of the model under the search condition specified by the search parameter P3 (step S103).”)
providing the identified neural network architecture; (“[0013] Embodiments of the present invention can solve the above-described problem, and an embodiment thereof is to provide a neural architecture search system and search method capable of searching for an architecture of a neural network that can satisfy constraints of a processing device and a communication network.”)
Arikawa does not teach:
transformer-based autoregressive language model (TBALM)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures.
However, Henighan does:
transformer-based autoregressive language model (TBALM) (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities”)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget”)
Arikawa and Henighan are considered analogous art to the claimed invention because they are in the same field of endeavor being neural architecture searching. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the neural network of Arikawa with the transformer language model of Henighan to obtain the predictable result of an autoregressive model search method. Also Henighan already selects optimal models based on a certain constraint (not latency) so one would want to combine the latency constraints of Arikawa with the selection process of Henighan.
Regarding claim 2, Arikawa in view of Henighan teaches claim 1 as outlined above. Arikawa further teaches:
the request further specifies a maximum amount of memory consumed by the TBALM during inference; ([0061] “The memory capacity required for the processing of the neural network needs to be lower than available memory capacities of the processing devices 100-1-1 to 100-1-N, 100-2-1 to 100-2-M, . . . , and 100-P-1 to 100-P-L.”)
Henighan further teaches:
the evaluating and automatically selecting includes identifying the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, (ii) satisfies the maximum amount of memory consumed and (iii) has a greatest number of decoder parameters for the architectures that satisfy both the maximum latency constraint and maximum amount of memory consumed resulting in the identified TBALM architecture. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget” using the latency and memory constraints from Arikawa)
Regarding claim 4, Arikawa in view of Henighan teaches claim 1 as outlined above. Henighan further teaches
the total number of decoder parameters includes weights and biases of only the decoder of the TBALM architecture. (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities” in their broadest reasonable interpretation parameters includes weights.)
Regarding claim 5, Arikawa in view of Henighan teaches claim 4 as outlined above. Henighan further teaches:
wherein the decoder parameters include, of the identified TBALM architecture: weights of attention heads; model dimensions; inner dimension of a feed forward network (FFN); and number of decoder layers (Page 34 “All models used a learning rate schedule with a 3000 step linear warm-up followed by a linear decay to 1=10 of the maximum learning rate. Model hyperparmeters and learning rates are shown in tables 4 and 5. The number of attention heads was always chosen to be max(2; dmodel=64). Most models were trained with roughly 5 _ 105 tokens per batch; differences from this are noted in the captions of the tables below. ‘Parameters’ always refers to the non-embedding parameter counts” and page 6 “The transformers used for language and multimodal modeling have fully connected layers of size 4dmodel and attention layers of size dmodel, in the notation of [KMH+20, BMR+20]. For math, image, and video modeling we scale the FC layers to dmodel and the attention layers to dmodel=4. We use an aspect ratio dmodel=nlayer ~ 10 for math, images, and videos as we find that this is approximately optimal, meaning that these domains prefer much deeper models as compared to language [KMH+20], where the optimal aspect ratio ~ 100. Thus our math, image, and video models are essentially identical, differing only in context length. For math alone we used a weight decay [LH17] of 0:05. We provide more detailed hyperparameter settings in appendix F.”)
Regarding claim 8, Arikawa teaches:
A device comprising: processing circuitry; a memory including instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations comprising ([0159] “The neural architecture search systems described in the first to fourth embodiments can be implemented by a computer including a central processing unit (CPU), a storage device, and an interface and a program for controlling those hardware resources. A configuration example of the computer is illustrated in FIG. 10.”)
receiving, at a compute device, a request for a neural network, the request specifying a maximum latency constraint; ([0049] “One feature of the neural architecture search system of this embodiment is that the architecture of the neural network is searched for so as to satisfy a constraint condition specified by a user as an external setting parameter P1 ” and [0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
evaluating, by the computer device neural network architectures to identify architectures that satisfy the maximum latency constraint resulting in potential neural network architectures; ([0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
training a neural network (NN) model that includes the identified TBALM architecture resulting in a trained NN ([0083] ”Next, the learning engine unit 3 inputs the training data D1 to the model of the neural network prescribed by the model information M2 and performs learning of the model under the search condition specified by the search parameter P3 (step S103).”)
providing the identified neural network architecture; (“[0013] Embodiments of the present invention can solve the above-described problem, and an embodiment thereof is to provide a neural architecture search system and search method capable of searching for an architecture of a neural network that can satisfy constraints of a processing device and a communication network.”)
Arikawa does not teach:
transformer-based autoregressive language model (TBALM)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures.
However, Henighan does:
transformer-based autoregressive language model (TBALM) (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities”)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget”)
Arikawa and Henighan are considered analogous art to the claimed invention because they are in the same field of endeavor being neural architecture searching. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the neural network of Arikawa with the transformer language model of Henighan to obtain the predictable result of an autoregressive model search method. Also Henighan already selects optimal models based on a certain constraint (not latency) so one would want to combine the latency constraints of Arikawa with the selection process of Henighan.
Regarding claim 9, Arikawa in view of Henighan teaches claim 8 as outlined above. Arikawa further teaches:
the request further specifies a maximum amount of memory consumed by the TBALM during inference; ([0061] “The memory capacity required for the processing of the neural network needs to be lower than available memory capacities of the processing devices 100-1-1 to 100-1-N, 100-2-1 to 100-2-M, . . . , and 100-P-1 to 100-P-L.”)
Henighan further teaches:
the evaluating and automatically selecting includes identifying the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, (ii) satisfies the maximum amount of memory consumed and (iii) has a greatest number of decoder parameters for the architectures that satisfy both the maximum latency constraint and maximum amount of memory consumed resulting in the identified TBALM architecture. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget” using the latency and memory constraints from Arikawa)
Regarding claim 11, Arikawa in view of Henighan teaches claim 8 as outlined above. Henighan further teaches
the total number of decoder parameters includes weights and biases of only the decoder of the TBALM architecture. (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities” in their broadest reasonable interpretation parameters includes weights.)
Regarding claim 12, Arikawa in view of Henighan teaches claim 11 as outlined above. Henighan further teaches:
wherein the decoder parameters include, of the identified TBALM architecture: weights of attention heads; model dimensions; inner dimension of a feed forward network (FFN); and number of decoder layers (Page 34 “All models used a learning rate schedule with a 3000 step linear warm-up followed by a linear decay to 1=10 of the maximum learning rate. Model hyperparmeters and learning rates are shown in tables 4 and 5. The number of attention heads was always chosen to be max(2; dmodel=64). Most models were trained with roughly 5 _ 105 tokens per batch; differences from this are noted in the captions of the tables below. ‘Parameters’ always refers to the non-embedding parameter counts” and page 6 “The transformers used for language and multimodal modeling have fully connected layers of size 4dmodel and attention layers of size dmodel, in the notation of [KMH+20, BMR+20]. For math, image, and video modeling we scale the FC layers to dmodel and the attention layers to dmodel=4. We use an aspect ratio dmodel=nlayer ~ 10 for math, images, and videos as we find that this is approximately optimal, meaning that these domains prefer much deeper models as compared to language [KMH+20], where the optimal aspect ratio ~ 100. Thus our math, image, and video models are essentially identical, differing only in context length. For math alone we used a weight decay [LH17] of 0:05. We provide more detailed hyperparameter settings in appendix F.”)
Regarding claim 15, Arikawa teaches:
A non-transitory machine-readable medium including instructions that, when executed by a machine, cause the machine to perform operations ([0159] “The neural architecture search systems described in the first to fourth embodiments can be implemented by a computer including a central processing unit (CPU), a storage device, and an interface and a program for controlling those hardware resources. A configuration example of the computer is illustrated in FIG. 10.” Thus it must have some sort of computer readable medium to execute the method.)
receiving, at a compute device, a request for a neural network, the request specifying a maximum latency constraint; ([0049] “One feature of the neural architecture search system of this embodiment is that the architecture of the neural network is searched for so as to satisfy a constraint condition specified by a user as an external setting parameter P1 ” and [0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
evaluating, by the computer device neural network architectures to identify architectures that satisfy the maximum latency constraint resulting in potential neural network architectures; ([0050] “For example, in a case where a processing time from when data is input to the neural network until when an inference result of the neural network is transmitted to a predetermined device, that is, end-to-end latency, is defined as a constraint condition, it is necessary to search for the architecture of the neural network not only having high inference accuracy but also satisfying the constraint condition.”)
training a neural network (NN) model that includes the identified TBALM architecture resulting in a trained NN ([0083] ”Next, the learning engine unit 3 inputs the training data D1 to the model of the neural network prescribed by the model information M2 and performs learning of the model under the search condition specified by the search parameter P3 (step S103).”)
providing the identified neural network architecture; (“[0013] Embodiments of the present invention can solve the above-described problem, and an embodiment thereof is to provide a neural architecture search system and search method capable of searching for an architecture of a neural network that can satisfy constraints of a processing device and a communication network.”)
Arikawa does not teach:
transformer-based autoregressive language model (TBALM)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures.
However, Henighan does:
transformer-based autoregressive language model (TBALM) (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities”)
automatically selecting, by the compute device and using a total number of decoder parameters of the architecture as a proxy for architecture accuracy a TBALM architecture of the potential identified TBALM architectures that has a greatest number of decoder parameters resulting in an identified TBALM architecture, the total number of decoder parameters excludes parameters of an embedding layer of the TBALM architectures. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget”)
Arikawa and Henighan are considered analogous art to the claimed invention because they are in the same field of endeavor being neural architecture searching. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the neural network of Arikawa with the transformer language model of Henighan to obtain the predictable result of an autoregressive model search method. Also Henighan already selects optimal models based on a certain constraint (not latency) so one would want to combine the latency constraints of Arikawa with the selection process of Henighan.
Regarding claim 16, Arikawa in view of Henighan teaches claim 15 as outlined above. Arikawa further teaches:
the request further specifies a maximum amount of memory consumed by the TBALM during inference; ([0061] “The memory capacity required for the processing of the neural network needs to be lower than available memory capacities of the processing devices 100-1-1 to 100-1-N, 100-2-1 to 100-2-M, . . . , and 100-P-1 to 100-P-L.”)
Henighan further teaches:
the evaluating and automatically selecting includes identifying the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, (ii) satisfies the maximum amount of memory consumed and (iii) has a greatest number of decoder parameters for the architectures that satisfy both the maximum latency constraint and maximum amount of memory consumed resulting in the identified TBALM architecture. (Figure 3 on page 5 shows multiple graphs comparing parameters to loss and it all cases the more parameters the loss goes down. As mentioned above, this paper is using decoder-only models and thus all the parameters must be decoder parameters. This allows them to select an optimal model. Page 4 “The scaling of loss with compute makes it possible to estimate the optimal model size for a given compute budget” using the latency and memory constraints from Arikawa)
Regarding claim 18, Arikawa in view of Henighan teaches claim 15 as outlined above. Henighan further teaches
the total number of decoder parameters includes weights and biases of only the decoder of the TBALM architecture. (Page 5 section 1.1 “We apply autoregressive decoder-only Transformer models to all data modalities” in their broadest reasonable interpretation parameters includes weights.)
Regarding claim 19, Arikawa in view of Henighan teaches claim 15 as outlined above. Henighan further teaches:
wherein the decoder parameters include, of the identified TBALM architecture: weights of attention heads; model dimensions; inner dimension of a feed forward network (FFN); and number of decoder layers (Page 34 “All models used a learning rate schedule with a 3000 step linear warm-up followed by a linear decay to 1=10 of the maximum learning rate. Model hyperparmeters and learning rates are shown in tables 4 and 5. The number of attention heads was always chosen to be max(2; dmodel=64). Most models were trained with roughly 5 _ 105 tokens per batch; differences from this are noted in the captions of the tables below. ‘Parameters’ always refers to the non-embedding parameter counts” and page 6 “The transformers used for language and multimodal modeling have fully connected layers of size 4dmodel and attention layers of size dmodel, in the notation of [KMH+20, BMR+20]. For math, image, and video modeling we scale the FC layers to dmodel and the attention layers to dmodel=4. We use an aspect ratio dmodel=nlayer ~ 10 for math, images, and videos as we find that this is approximately optimal, meaning that these domains prefer much deeper models as compared to language [KMH+20], where the optimal aspect ratio ~ 100. Thus our math, image, and video models are essentially identical, differing only in context length. For math alone we used a weight decay [LH17] of 0:05. We provide more detailed hyperparameter settings in appendix F.”)
Claims 6-7, 13-14, & 20 are rejected under 35 U.S.C. 103 as being unpatentable over Arikawa in view of Henighan and Yang (NPL: ‘Efficient Resource-Aware Convolutional Neural Architecture Search for Edge Computing with Pareto-Bayesian Optimization’ (Jan 2021)).
Regarding claim 6, Arikawa in view of Henighan teaches claim 1 as outlined above. Neither of them teach:
the method further comprises generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve.
However, Yang does:
the method further comprises generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve. (page 8 “By finding the Pareto Optimal Front, the search direction is guided, and the CNN with a trade-off between accuracy and inference latency for the edge devices is also obtained. Moreover, since inference latency is the additional search target, the search space will be further narrowed down, and the search efficiency will also be enhanced.”)
Arikawa, Henighan and Yang are considered analogous art to the claimed invention because they are in the same field of endeavor neural architecture search. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall structure of Arikawa with the transformer based language model and selection criteria of Henighan with the pareto optimal of Yang. Yang already uses accuracy and inference for their trade-off so one would want to replace the accuracy of Yang with the accuracy used in Henighan.
Regarding claim 7, Arikawa in view of Henighan and Yang teaches claim 6 as outlined above. Yang further teaches
the automatically selecting the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, and (ii) has a greatest number of decoder parameters for the architectures that satisfy the maximum latency constraint includes selecting the TBALM corresponding to a point at a boundary of the generated pareto curve (Page 10 “When applying the Pareto optimal to the search process, at each round of the search process, the new morphed networks are trained, and through the profiling model, their accuracy and inference latency are obtained. The accuracy and latency are saved as a tuple, by finding the Pareto front to get the results of Pareto optimal” where Henighan teaches the TBALM architecture.)
Regarding claim 13, Arikawa in view of Henighan teaches claim 8 as outlined above. Neither of them teach:
generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve.
However, Yang does:
generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve. (page 8 “By finding the Pareto Optimal Front, the search direction is guided, and the CNN with a trade-off between accuracy and inference latency for the edge devices is also obtained. Moreover, since inference latency is the additional search target, the search space will be further narrowed down, and the search efficiency will also be enhanced.”)
Arikawa, Henighan and Yang are considered analogous art to the claimed invention because they are in the same field of endeavor neural architecture search. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall structure of Arikawa with the transformer based language model and selection criteria of Henighan with the pareto optimal of Yang. Yang already uses accuracy and inference for their trade-off so one would want to replace the accuracy of Yang with the accuracy used in Henighan.
Regarding claim 14, Arikawa in view of Henighan and Yang teaches claim 13 as outlined above. Yang further teaches
the automatically selecting the TBALM architecture of the TBALM architectures that (i) satisfies the maximum latency constraint, and (ii) has a greatest number of decoder parameters for the architectures that satisfy the maximum latency constraint includes selecting the TBALM corresponding to a point at a boundary of the generated pareto curve (Page 10 “When applying the Pareto optimal to the search process, at each round of the search process, the new morphed networks are trained, and through the profiling model, their accuracy and inference latency are obtained. The accuracy and latency are saved as a tuple, by finding the Pareto front to get the results of Pareto optimal” where Henighan teaches the TBALM architecture.)
Regarding claim 20, Arikawa in view of Henighan teaches claim 15 as outlined above. Neither of them teach:
generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve.
However, Yang does:
generating, by the compute device, a pareto curve of number of decoder parameters versus latency for a variety of TBALM architectures, based on a processor of the compute device resulting in a generated pareto curve. (page 8 “By finding the Pareto Optimal Front, the search direction is guided, and the CNN with a trade-off between accuracy and inference latency for the edge devices is also obtained. Moreover, since inference latency is the additional search target, the search space will be further narrowed down, and the search efficiency will also be enhanced.”)
Arikawa, Henighan and Yang are considered analogous art to the claimed invention because they are in the same field of endeavor neural architecture search. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the overall structure of Arikawa with the transformer based language model and selection criteria of Henighan with the pareto optimal of Yang. Yang already uses accuracy and inference for their trade-off so one would want to replace the accuracy of Yang with the accuracy used in Henighan.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ghorbani (NPL: ‘Scaling Laws for Neural Machine Translation’ (Sep 2021)) teaches the affects of encoder and decoder parameters have on accuracy for neural machine translation.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL P GRUSZKA whose telephone number is (571)272-5259. The examiner can normally be reached M-F 9:00 AM - 6:00 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li Zhen can be reached at (571) 272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL GRUSZKA/Examiner, Art Unit 2121
/Li B. Zhen/Supervisory Patent Examiner, Art Unit 2121
Prosecution Insights