TRAINING MACHINE LEARNING TRANSFORMER ARCHITECTURES WITH HYBRID MOVING AVERAGE

§102 §103 §112

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This office action is in response to submission of application on 3/19/2024. Claims 1-20 are presented for examination. Drawings The Drawings filed on 3/19/2024 are acceptable for examination purposes. Specification The Specification filed on 3/19/2024 is acceptable for examination purposes. Drawings The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because: reference characters "325 [¶0035]" and "328 [¶0037 & Fig. 3]" have both been used to designate the workspace module. reference character “325” has been used to designate both the interface module [¶0034 & Fig. 3] and the workspace module [¶0035]. reference characters "410 [¶0060]" and "450 [¶0060, ¶0061 & Fig. 4]" have both been used to designate the driver node. reference character “355” has been used to designate both the file management module [¶0077] and the query processing module [¶0034, ¶0040, Fig 3]. reference character “355” has been used to designate both the file management module [¶0077] and the query processing module [¶0034, ¶0040, Fig 3]. Reference character “812” has been used to designate both the Input/Output (IO) systems [¶0079-80] and the Input Device(s) [Fig 8]. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: [¶0060] “driver node 410". Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance. Claim Rejections - 35 USC § 112 The following is a quotation of 35 U.S.C. 112(b): (b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention. The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph: The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention. Claim 7 recites the limitation "the smoothing term" in “further comprising determining the first number of iterations based on a total number of iterations and the smoothing term”. There is insufficient antecedent basis for this limitation in the claim, as “smoothing term” is not previously recited in claim 7 or claim 1, from which claim 7 depends. Appropriate correction is required. The following references are relied upon for the art rejection set forth below: Busbridge et al. How to Scale Your EMA, September 2023, hereinafter Busbridge Zhou et al. Text to Image Generation in DCGAN and Stable Diffusion Model, January 2024, hereinafter Zhou Li et al. FedTP: Federated Learning by Transformer Personalization, April 2023, hereinafter Li Karras et al. Analyzing and improving the Training Dynamics of Diffusion Models, December 2023, hereinafter Karras Claim Rejections - 35 USC § 102 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. Claims 1-3, 6, 8-10, 13, 15-17 & 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Busbridge. Regarding Claim 1, Busbridge discloses A method of training a transformer model, comprising: obtaining a set of training examples; [Busbridge: § 3.2] “We train a transformer (Vaswani et al., 2017) using the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) and Adam optimizer on the train-clean-100 subset (100h) of LibriSpeech (Panayotov et al., 2015).” Busbridge teaches obtaining the train-clean-100 subset of the LibriSpeech dataset as a set of training examples for training the transformer model. dividing the set of training examples across one or more batches for one or more iterations of training parameters of the transformer model; [Busbridge: §3.2] “and use dynamic batching (minibatch size × sequence length = const = 290s, and s indicates audio duration in seconds).” [Busbridge: §2.1] “Iterating over a sequence of independent minibatches 𝔹0, 𝔹1, . . . , 𝔹k-1 produces model parameters…” Busbridge teaches using dynamic batching during transformer training and iterating over a sequence of minibatches to train the model parameters. training the transformer model for a first number of iterations by repeatedly updating parameters of the transformer model; [Busbridge: §2.1] “Iterating over a sequence of independent minibatches 𝔹0, 𝔹1, . . . , 𝔹k-1 produces model parameters…” [Busbridge: Appendix G] “The pre-training phase, where we train model on labeled data for 20k steps with model EMA accumulation starting after 19k steps” [Busbridge: Table 6, p33] “Base start of EMA accumulation (steps) | N/A | 19k”. Busbridge teaches training the model for 20k steps by repeatedly updating model parameters over independent minibatches, with the first 19k training steps occurring before the start of Exponential Moving Average (EMA) accumulation corresponding to the first number of iterations. training the transformer model for a second number of iterations, for each iteration in the second number of iterations: [Busbridge: Appendix G] “The pre-training phase, where we train model on labeled data for 20k steps with model EMA accumulation starting after 19k steps” and [Busbridge: §3.3] “We first pre-train a target model (student) on a limited labeled set for a short period … Concurrently, the student updates a model EMA (teacher).” Busbridge teaches training the target model (student) for 20k steps, with model EMA (teacher) accumulation starting after 19k steps, corresponding to a second number of training iterations in which model EMA accumulation is performed. obtaining a set of moving average parameters for a current iteration from a cache, Regarding the claimed “cache”, under the broadest reasonable interpretation consistent with the specification, “cache” is interpreted to include storage for retaining moving average parameters for retrieval during a subsequent iteration. The specification states [¶0049] “The training module 350 generates moving average parameters for the next iteration (e.g., using Equation 1) and stores the moving average parameters (e.g., in a cache or in data store 270), such that the stored moving average parameters are now used for the next iteration.” [Busbridge: §1, Definition 1.1] “The EMA update for the model EMA parameters ζ t following target model parameters θ t at iteration t with momentum ρ ≡ 1 - β ρ is ζ t + 1 =   ρ ζ t + 1 - ρ θ t ≡ 1 - β ρ ζ t +   β ρ θ t .” [Busbridge: Appendix I] “all experiments presented in the paper store weights for target model and EMA in FP32...” [Busbridge: Algorithm 1, p.22] “Require: Initial target model parameters θ and model EMA parameters ζ ” … “ ζ ← ρ ^ ζ + 1 - ρ ^ θ   [ E M A   U p d a t e ] "   Busbridge teaches storing the EMA weights during training and maintaining the EMA parameters ζ as persistent state through successive iterations. For each EMA update, the current stored EMA parameters ζ accessed and used to compute updated EMA parameters ζ which are retained as the EMA state for the subsequent iteration, corresponding to obtaining a set of moving average parameters for a current iteration from a cache. generating a set of parameters for the current iteration, comprising: [Busbridge: Algorithm 1, p22] “ θ ← θ - η ^ B ^ ∑ x ∈ X ∇ L x ; θ ,   ζ   [ S G D   U p d a t e ] " Busbridge teaches generating an updated set of model parameters θ for the current iteration by performing the SGD update. generating a set of predictions for a batch of training examples by applying a set of parameters of the transformer model for a previous iteration to the batch of training examples for the current iteration, computing a loss function for the batch of training examples based on the set of predictions, [Busbridge: §2.1, Definition 2.1] “The SGD update for a model with parameters θt at iteration 𝑡 given a minibatch 𝔹… with learning rate 𝜂 is θ t + 1 =   θ t - η × 1 B ∑ x ∈ B ∇ θ L x ; θ t , where L is the loss function, ∇ θ L x ; θ t is the parameter gradient for the sample 𝑥 at iteration 𝑡.” [Busbridge: Appendix G] “We employ a vanilla encoder-based only transformer model trained with the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006).” [Busbridge: Appendix G, Definition G.1] “For a sequence of frames, select the most probable token per frame, removing repetitions and the CTC blank token” Busbridge teaches applying the model parameters θ t at iteration t to each sample x of the current minibatch to generate model outputs from which predictions are selected, and computing the CTC loss function for the minibatch using the model outputs/predictions. computing the set of parameters for the current iteration, wherein the set of parameters are computed to reduce the loss function, [Busbridge: Algorithm 1, p22] “ θ ← θ - η ^ B ^ ∑ x ∈ X ∇ L x ; θ ,   ζ   [ S G D   U p d a t e ] " Busbridge teaches computing an updated set of model parameters θ for the current iteration using the gradient of the loss function L , thereby computing the parameters to reduce the loss function. computing a set of moving average parameters for a next iteration by combining the set of parameters for the current iteration and the set of moving average parameters for the current iteration, and updating the set moving average parameters of the transformer model for the next iteration as the set of moving average parameters for the current iteration. [Busbridge: Algorithm 1, p. 22] “ ζ ← ρ ^ ζ + 1 - ρ ^ θ   [ E M A   U p d a t e ] "   Busbridge teaches that, following the SGD update of the model parameters θ for the current iteration, the EMA update combines the updated model parameters θ with the current EMA parameters ζ to compute updated EMA parameters ζ , which are maintained as the EMA parameters for the next iteration. Regarding Claim 2, Busbridge discloses the method according to claim 1, wherein training the transformer model for the first number of iterations to repeatedly update parameters of the transformer model comprises, for each iteration in the first number of iterations: [Busbridge: Appendix G] “The pre-training phase, where we train model on labeled data for 20k steps with model EMA accumulation starting after 19k steps…” [Busbridge: §2.1] “Iterating over a sequence of independent minibatches… produces model parameters…” Busbridge teaches repeatedly updating the model parameters over the first 19k steps, before the EMA accumulation begins, corresponding to the training the transformer model for the first number of iterations as claimed. generating a set of predictions for a batch of training examples by applying a set of parameters of the transformer model for a previous iteration to the batch of training examples for the current iteration; “[Busbridge: §2.1] “The SGD update for a model with parameters θ t at iteration t given a minibatch B = x b ~ P x : b = 1,2 , … , B   o f   B = | B | samples with learning rate η is θ t + 1 =   θ t - η × 1 B ∑ x ∈ B ∇ θ L ( x ; θ t )   ” [Busbridge: §3.2] “We train a transformer (Vaswani et al., 2017) using the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) and Adam optimizer”. Busbridge teaches training with a transformer model and CTC loss while applying the model parameters θ t for training of the next model parameters θ t + 1 . The use of CTC necessarily generates network predictions from the input training examples. The SGD update formula further teaches applying the model parameters θ t , resulting from the previous iteration, to the batch of training examples for the current iteration. Therefore, Busbridge teaches generating a set of predictions for the batch of training examples by applying the parameters from the previous iteration, as claimed. computing a loss function for the batch of training examples based on the set of predictions; and [Busbridge: §3.2] “We train a transformer (Vaswani et al., 2017) using the Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) and Adam optimizer”. The CTC loss is necessarily computed based on the network predictions generated from the training examples. Therefore, Busbridge teaches computing a loss function for the batch of training examples based on the set of predictions, as claimed. computing the set of parameters for the current iteration, wherein the set of parameters are computed to reduce the loss function. [Busbridge: §2.1] “The SGD update for a model with parameters θ t at iteration t given a minibatch 𝔹… with learning rate η is θ t + 1 =   θ t - η × 1 B ∑ x ∈ B ∇ θ L ( x ; θ t ) , where L is the loss function, ∇ θ L x ; θ t is the parameter gradient for the sample 𝑥 at iteration 𝑡.” Busbridge teaches computing the updated model parameters θ t + 1 using the gradient of the loss function, corresponding to computing the set of parameters for the current iteration to reduce the loss function. Regarding Claim 3, Busbridge discloses the method according to claim 2, wherein training the transformer model for the first number of iterations comprises training the transformer model without computing a moving average of the parameters of the transformer model. [Busbridge: Appendix G] “The pre-training phase, where we train model on labeled data for 20k steps with model EMA accumulation starting after 19k steps…” Busbridge teaches training the model for the first 19k steps before model EMA accumulation begins, corresponding to training the transformer model for the first number of iterations without computing the set of moving average parameters, as claimed. Regarding Claim 6, Busbridge discloses the method according to claim 1, wherein combining the set of parameters for the current iteration and the set of moving average parameters for the current iteration comprises: scaling the set of moving average parameters for the current iteration by a smoothing term; scaling the set of parameters for the current iteration by one minus the smoothing term; and summing the scaled set of moving average parameters for the current iteration and the scaled set of parameters for the current iteration. [Busbridge: §1] “The EMA update for the model EMA parameters ζ t following target model parameters θ t at iteration t with momentum ρ ≡ 1 - β ρ is ζ t + 1 =   ρ ζ t + 1 - ρ θ t ≡ 1 - β ρ ζ t +   β ρ θ t .” Busbridge teaches scaling the current EMA parameters ζ t by the smoothing term ρ , scaling the current model parameters θ t by one minus the smoothing term ρ , and summing the resulting scaled parameters to compute the EMA parameters ζ t + 1 for the next iteration as claimed. Claims 8-10, & 13 are substantially similar in scope and spirit as claims 1-3, & 6, respectively, differing only in reciting A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented form that Busbridge discloses as implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore, the rejections of claims 1-3, & 6 are applied accordingly. Claims 15-17, & 20 are substantially similar in scope and spirit as claims 1-3, & 6, respectively, differing only in reciting A computer system, comprising: a computer processor; and a non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented system that Busbridge discloses as being implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore the rejections of claims 1-3, & 6 are applied accordingly. Claim Rejections - 35 USC § 103 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention. Claims 4, 11, & 18 are rejected under 35 U.S.C. 103 as being unpatentable over Busbridge in view of Zhou Regarding Claim 4, Busbridge discloses the method according to claim 1, however Busbridge does not expressly disclose, but in combination with Zhou does teach: wherein the transformer model is a stable diffusion model coupled to receive input as text and generate an output as an image. [Zhou: §1.2] “we also use the stable diffusion model [8] to implement the text-to-image generation process. We chose the U-net model [9] to generate images in the stable diffusion model…” and further describes the approach “In the MNIST dataset [11], we use numbers 0 to 9 as text input and generate related images.” Therefore, Zhou teaches a stable diffusion model coupled to receive input as text and generate an output as an image. Zhou further discloses [Zhou: §2.1.5] “In the transformer block, we combine the cross-attention layer and feed-forward process.” and teaches that “the U-Net transformer represents a time dependent score-based model built upon the U-Net architecture with added attention layers” Therefore, Zhou teaches implementing the stable diffusion model using a U-Net transformer architecture including transformer blocks, corresponding to the claimed transformer model being a stable diffusion model. Busbridge and Zhou are analogous art because they are from the same field of endeavor, specifically training machine-learning models using Transformer-based architectures, with Busbridge teaching model training using an exponential moving average of model parameters and Zhou teaching training a Stable Diffusion U-Net transformer for text-to-image generation. It would have been obvious to a person having ordinary skill in the art (PHOSITA), before the effective filing date of the claimed invention, to apply the model Busbridge’s EMA training techniques to Zhou’s stable diffusion U-net transformer to reduce the effect of variation in the model parameters over training iterations and provide a more stable set of trained model parameters. The suggestion/motivation for doing so would have been provided by Busbridge itself, which teaches that model EMA [Busbridge: Abstract] “can improve the robustness and generalization of supervised learning” and demonstrates its EMA technique is effective “across a range of architectures, optimizers, and data modalities.” Busbridge further notes that [Busbridge: §1] “model EMA has a number of desirable properties: i) the model EMA inhabits wider minima than the target model, reducing overfitting and improving generalization … the model EMA is relatively cheap to compute, whilst providing a valid model but different to the target model.” A PHOSITA therefore would have been motivated to apply Busbridge’s model EMA technique when training Zhou’s stable diffusion U-net transformer, combining prior-art elements according to known methods to obtain the predictable result of improved robustness and generalization while reducing overfitting. (MPEP § 2143.01(A)). Claims 11 & 18 are substantially similar in scope and spirit as claim 4, differing only in reciting A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented form that Busbridge discloses as implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore, the rejection of claim 4 is applied accordingly. Claims 11 & 18 are substantially similar in scope and spirit as claim 4, differing only in reciting A computer system, comprising: a computer processor; and a non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented system that Busbridge discloses as being implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore the rejection of claim 4 is applied accordingly. Claims 5, 12, & 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Busbridge and Zhou in further view of Li. Regarding Claim 5, the combination of Busbridge and Zhou disclose the method according to claim 4, wherein the stable diffusion model includes one or more layers configured as a U-net, [Zhou: §2.1.4] “In the stable diffusion model, we implement the cross-attention layer to connect text-embedding and generated images” [Zhou: §2.1.5] “In the transformer block, we combine the cross-attention layer and feed-forward process… Moreover, the U-Net transformer represents a time-dependent score-based model built upon the U-Net architecture with added attention layers.” Zhou discloses a stable diffusion model with cross-attention layers configured within a U-Net architecture, corresponding to the U-net as claimed. However, the combination of Busbridge and Zhou does not disclose, but in combination with Li does teach wherein the computing of the set of moving average parameters for the next iteration excludes a subset of parameters for attention layers of the U-net. [Li: §III.B] “we first develop a personalized self-attention mechanism that maintains the self-attention layers of each client locally so as to personalize the model, while averaging other parameters to learn the common information.” [Li: §III.B] “Let W i denote the projection parameters of the self-attention layer, and let ξ denote the parameters of other layers… where W i t , k is retained locally to store the personalized information of client i and ξ - i t , k is aggregated across the clients.” Li teaches excluding the self-attention layer parameters from an averaging operation while averaging the remaining model parameters. Applying Li’s selective averaging technique to the attention layers of Zhou’s U-Net in the moving-average computation of Busbridge results in excluding the attention-layer parameters from the computation of the moving-average parameters, corresponding to computing the set of moving average parameters for the next iteration while excluding a subset of parameters for attention layers of the U-Net. Busbridge, Zhou, and Li are analogous art because they relate to training and updating parameters of neural-network models employing attention-based architectures. Busbridge expressly applies its model-EMA technique to Vision Transformer architectures, Li concerns the treatment of self-attention parameters in Transformer models, and Zhou employs attention layers within a U-Net transformer for a stable diffusion model. Further, Li is reasonably pertinent to the problem of treatment of parameters in attention-based neural networks during model-parameter averaging. It would have been obvious to a PHOSITA before the effective filing date of the claimed invention to incorporate the selective parameter-averaging technique taught by Li into the stable diffusion model of Zhou, as modified by Busbridge, in order to preserve the information represented by the attention mechanism while still obtaining the benefit of applying a moving average to the remaining parameters. The suggestion/motivation to do so would have been provided by Li itself, which notes that [Li: Abstract] “federated averaging algorithms actually have a negative impact on self-attention in cases of data heterogeneity, which limits the capabilities of the Transformer model”, Li further explains [Li: §I Introduction] “aggregating the self-attention of clients with heterogeneous data may ruin the client-specific representations, potentially degrading the model’s performance.” Li’s addresses this issue by creating a framework that provides benefits [Li: §I Introduction] “In addition to high accuracy, FedTP is also scalable to the increase of client number and enjoys good generalization capability to novel clients.” A PHOSITA would have been motivated to incorporate Li’s selective parameter-averaging technique into Zhou’s stable diffusion U-net transformer, as modified by Busbridge, combining prior-art elements according to known methods to obtain the predictable result of improved model performance, scalability, and generalization. (MPEP § 2143.01(A)). Claim 12 is substantially similar in scope and spirit as claim 5, differing only in reciting A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented form that Busbridge discloses as implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore, the rejection of claim 5 is applied accordingly. Claim 19 is substantially similar in scope and spirit as claim 5, differing only in reciting A computer system, comprising: a computer processor; and a non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented system that Busbridge discloses as being implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore the rejection of claim 5 is applied accordingly. Claims 7 & 14 are rejected under 35 U.S.C. 103 as being unpatentable over Busbridge in view of Karras. Regarding Claim 7, Busbridge discloses the method according to claim 1, however Busbridge does not expressly disclose, but in combination with Karras does teach: further comprising determining the first number of iterations based on a total number of iterations and the smoothing term, such that a degree of decay of the parameters of the transformer model computed in the first number of training iterations is less than a predetermined threshold after training the model for the second number of iterations. [Karras: §3.1] “Traditional EMA maintains a running weighted average θ ^ β of the network parameters alongside the parameters θ that are being trained. At each training step, the average is updated by θ ^ β t = β θ ^ β t - 1 + ( 1 - β ) θ ( t ) , where t indicates the current training step, yielding an exponential decay profile in the contributions of earlier training steps. The rate of decay is determined by the constant β …” Karras further teaches that [Karras: §3.1] “very long exponential EMA puts non-negligible weight on initial stages of training where network parameters are mostly random.… longer training runs benefit from longer EMA decay, and thus the averaging profile should ideally scale automatically with training time.” Karras teaches that the constant β , corresponding to the claimed smoothing term, determines the rate at which the contribution of parameters from earlier training iterations decays. Karras additionally identifies the residual contribution of parameters from earlier training iterations and teaches reducing that contribution by selecting the EMA decay relative to the duration of training. Karras further teaches [Karras: Appendix C.3] “In the case of exponential EMA, β ( t n ) is constant and … β t n = … = 2 - Δ t / λ ”, expressly relating the EMA decay constant β , the duration Δt of a training iteration, and the EMA half-life λ. Therefore, Karras teaches that the contribution of parameters from earlier training iterations decays at a rate determined by the smoothing term, that the extent of that decay should be selected relative to the duration of training, and provides the relationship between the smoothing term and the duration over which that decay occurs. Under the broadest reasonable interpretation, selecting the training-iteration boundary at which the residual contribution of the earlier parameters has decayed below a chosen level constitutes determining, based on the total number of iterations and the smoothing term, the first number of iterations such that the decay of parameters from the first number of iterations is below a predetermined threshold after the second number of iterations, as claimed. Busbridge and Karras are analogous art because they are from the same field of endeavor, specifically that of training machine-learning models using moving averages of model parameters during training, including controlling the EMA decay applied to parameters over successive training iterations. It would have been obvious to a PHOSITA before the effective filing date of the claimed invention to apply the EMA decay and training-duration teachings of Karras to the EMA training procedure of Busbridge in order to appropriately select the EMA decay relative to the duration of training and improve the effectiveness of EMA over the course of training. The suggestion/motivation for doing so would have been provided by Karras itself, which teaches that [Karras: §3] “the choice of its [EMA] decay parameter has a significant impact on results“ and that [Karras: §3.1] “longer training runs benefit from longer EMA decay, and thus the averaging profile should ideally scale automatically with training time.” Karras further demonstrates that model performance varies as a function of EMA length, identifying an optimum EMA length and teaching that deviation from that optimum is detrimental [Karras: Figure 5]. A PHOSITA therefore would have been motivated to apply Karras’s EMA decay and training duration teachings to Busbridge’s EMA training procedures, combining prior-art elements according to known methods to obtain the predictable results of improved model performance through more effective application of EMA over the course of training (MPEP § 2143.01(A)). Claim 14 is substantially similar in scope and spirit as claim 7, differing only in reciting A non-transitory computer readable storage medium comprising stored program code, the program code comprising instructions, the instructions when executed causes a processor system to perform the recited operations – a computer-implemented form that Busbridge discloses as implemented in software and executed on processing hardware [Busbridge: Appendix J] “Investigations carried out in two distributed, scalable frameworks: Jax … and PyTorch…” [Busbridge: Appendix G] “All experiments are done using A100 80GB 8GPU nodes with bfloat16 precision training.” Therefore, the rejection of claim 7 is applied accordingly. Conclusion The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Wang et al. “Gradient Vaccine: Investigating and Improving Multi-Task Optimization in Massively Multilingual Models” Chefer et al. “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models” Gandikota et al. “Erasing Concepts from Diffusion Models” Zhang et al. “Enhancing Semantic Fidelity in Text-to-Image Synthesis: Attention Regulation in Diffusion Models” Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRANT F FITCH whose telephone number is (571)270-0621. The examiner can normally be reached Monday-Thursday 7-3. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached at (571) 270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /G.F.F./Examiner, Art Unit 2124 /ALAN CHEN/ Primary Examiner, Art Unit 2125