DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR
1.17(e), was filed in this application after final rejection. Since this application is eligible for continued
examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the
finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's
submission filed on 30 March 2026 has been entered.
Response to Amendment
The amendment filed on 30 March 2026 has been entered.
Claims 1-19 are pending.
Claims 1-2, 4, 6-7, 9-12 are amended.
Claims 13-19 are cancelled.
Claims 1-12 will be pending.
Response to Arguments
Applicant's arguments filed on 30 March 2026 have been fully considered, but they are not persuasive.
Applicant’s remarks, regarding the rejections of claims under 35 USC 103, have been fully considered.
Applicant submits neither Ravishankar, Towal, Aghdasi, the remaining references, nor any combination thereof discloses, teaches, or suggests each and every claimed feature of independent Claim 1. Applicant submits neither Ravishankar, Towal, Aghdasi, the remaining references, nor any combination thereof discloses, teaches, or suggests each and every claimed feature of independent Claim 11.
Applicant’s arguments have been considered, but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 4, 8-12 are rejected under 35 U.S.C. 103 as being unpatentable over Ravishankar et al. (U.S. Pre-Grant Publication No. 20190130247, hereinafter ‘Ravishankar'), in view of Fu et al. (NPL: "Learn-to-Share: A Hardware-friendly Transfer Learning Framework Exploiting Computation and Parameter Sharing", hereinafter 'Fu').
Regarding claim 1 and analogous claim 11, Ravishankar teaches A processor-implemented method with multi-task processing, the method comprising ([0046] The one or more processors 152 are, in certain implementations, microprocessors configured to execute instructions stored in the memory 156 or other accessible locations. Alternatively, the one or more processors 152 may be implemented as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and/or other devices designed to perform functions discussed herein in a dedicated manner. As will be appreciated, multiple processors 152 or processing components may be used to perform functions discussed herein in a distributed or parallel manner.; [0047] The memory 156 may encompass any tangible, non-transitory medium for storing data or executable routines, including volatile memory, non-volatile memory, or any combination thereof. Although shown for convenience as a single block in FIG. 8, the memory 156 may actually encompass various discrete media in the same or different physical locations. The one or more processors 152 may access data in the memory 156 via one or more busses 154.):
obtaining a first output data of a first layer of a second neural network, a second output data of a first layer of a third neural network and weights of a second layer of a first neural network ([0017] In general, the processing from one level or abstraction to the next can be considered as one ‘stage’ of the analysis process. obtaining a first output data of a first layer of a second neural network, a second output data of a first layer of a third neural network Each stage of the analysis can be performed by separate neural networks or by different parts of one larger neural network. For example, as discussed herein, a single deep learning network may cover all stages in an analytic process (e.g., from an initial input to an output data set). Alternatively, separate distinct deep learning network(s) may each cover only one stage (or a subset of stages) of the overall analysis process.; [0023] Turning to FIGS. 3 and 4, an example of multi-task feature ranking network 116 is provided. In this example, the multi-task feature ranking network 116 is in the form of a multi-task deep learning algorithm in which input features 102 (input via input layer 54) are processed by K neural networks or branches 120 of a neural network for predicting the K tasks 122. To facilitate the feature ranking approach discussed herein, a broadcast layer 126 is provided which selects and/or filters the input features 102 before they propagate through the separate, parallel branches 120 of the neural network.; [0024] By way of explanation, let the input samples {X} be represented by N features 102. Let and weights of a second layer of a first neural network W be the collection of weights of the feature weighting layer (i.e., broadcast layer 126). For a K-task learning problem, W has N*K weights, where for every task, N scaling coefficients Wk are learned.),
Ravishankar fails to teach wherein the second neural network is fine-tuned from the first neural network based on a first task, wherein the third neural network is fine-tuned from the first neural network based on a second task, and in response to obtaining the first output data and the second output data, obtaining, from memory, first delta weights and second delta weights previously computed and stored, wherein the first delta weights are associated with a second layer of the second neural network, wherein the second delta weights are associated with a second layer of the third neural network; performing an operation of the second neural network on the first output data, based on sums of the weights of the second layer of the first neural network and the first delta weights; and performing an operation of the third neural network on the second output data, based on sums of the weights of the second layer of the first neural network and the second delta weights, wherein the first delta weights comprise difference values in the weights of the second layer of the first neural network and weights of the second layer of the second neural network, and the second delta weights comprises difference values in the weights of the second layer of the first neural network and weights of the second layer of the third neural network.
Fu teaches wherein the second neural network is fine-tuned from the first neural network based on a first task ([Wδ pruning.] In this work, we treat the final fine-tuning weight W f as the addition between pre-trained weight Wp and a weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can wherein the second neural network is fine-tuned from the first neural network based on a first task obtain the deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.),
wherein the third neural network is fine-tuned from the first neural network based on a second task ([Wδ pruning.] In this work, we treat the final fine-tuning weight W f as the addition between pre-trained weight Wp and a weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can wherein the third neural network is fine-tuned from the first neural network based on a second task obtain the deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.), and
in response to obtaining the first output data and the second output data, obtaining, from memory, first delta weights and second delta weights previously computed and stored ([Search space.] As shown in Figure 4, we decouple data dependency across layers by using a pooling layer, a linear layer, and a Bi-LSTM to aggregate all layers’ output for final classification. The pooling layer uses the first hidden vector corresponding to the first token (i.e., [CLS] token) (Devlin et al., 2019) as the layer presentation. The pooling output vectors are then fed into a linear layer and a Bi-LSTM. LeTS first builds up a stochastic super network Wτ for the searching phase. Before searching, we copy the weights from Wp to Wτ trainable layers and disable the gradient computation based on cτ which is the weight difference mask obtained from the Delta-Pruning. Two decisions should be made in each attention layer Wτ j : (i) the input to the trainable layer. It can be either the cached result (xp j−1) or the in response to obtaining the first output data and the second output data output from the previous trainable layer (xf j−1) (ii) the output to the pooling layer from layer j. It can be either xp j or xf j . The total size of the search space would be 4N where N is the layer number in the pre-training mode (≈ 1015 for BERTLARGE).; [(i) Bypass self-attention layers.] When the cached result xp j is used by the final pooling layer and next trainable layer, the obtaining, from memory computation and parameters of the entire layer can be saved. This can be applied at layer Wp j where j ∈ {0,1,2,6}.; [(ii) Exploit the sparsity of Wδ.] LeTS can leverage the sparsity feature of Wδ j . More specifically, when the input to the trainable attention layer is xp j−1, LeTS computes xp j−1 · first delta weights and second delta weights previously computed and stored Wδ j and adds it to a cached result.),
wherein the first delta weights are associated with a second layer of the second neural network ([3.1. Delta-Pruning in Early Stage] Delta-Pruning is motivated by SNIP (Lee et al., 2019) which targets to generate weight sparsity before training. We decompose the final fine-tuned weight (W f) into two parts as shown in Eq. (1). Wf =Wp+c Wδ (1) Here, Wδ ∈ Rd is the fine-tuning weight difference, c ∈ {0,1}d is the generated mask for Wδ. To resolve (ii), we first learn the weight difference initialization by warm-up the fine-tuning using D for steps Nsteps and get W f. We then wherein the first delta weights are associated with a second layer of the second neural network approximate Wδ using a task-specific initialization as Wδ = W f −Wp. Our ablation study shows that using task-specific warm-up shows better results compared to random initialization as this accumulation of gradients can better reflect the final weight difference distribution.),
wherein the second delta weights are associated with a second layer of the third neural network ([3.1. Delta-Pruning in Early Stage] Delta-Pruning is motivated by SNIP (Lee et al., 2019) which targets to generate weight sparsity before training. We de compose the final fine-tuned weight (W f) into two parts as shown in Eq. (1). Wf =Wp+c Wδ (1) Here, Wδ ∈ Rd is the fine-tuning weight difference, c ∈ {0,1}d is the generated mask for Wδ. To resolve (ii), we first learn the weight difference initialization by warm-up the fine-tuning using D for steps Nsteps and get W f. We then wherein the second delta weights are associated with a second layer of the third neural network approximate Wδ using a task-specific initialization as Wδ = W f −Wp. Our ablation study shows that using task-specific warm-up shows better results compared to random initialization as this accumulation of gradients can better reflect the final weight difference distribution.);
performing an operation of the second neural network on the first output data ([1. Introduction] (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the computation can be reduced into a sparse matrix multiplication by leveraging the sparsity produced from (iii).),
based on sums of the weights of the second layer of the first neural network and the first delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on sums of the weights of the second layer of the first neural network and the first delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.); and
performing an operation of the third neural network on the second output data ([1. Introduction] (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the computation can be reduced into a sparse matrix multiplication by leveraging the sparsity produced from (iii).),
based on sums of the weights of the second layer of the first neural network and the second delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on sums of the weights of the second layer of the first neural network and the second delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.),
wherein the first delta weights comprise difference values in the weights of the second layer of the first neural network and weights of the second layer of the second neural network ([Wδ pruning.] the addition between pre-trained weight Wp and a wherein the first delta weights comprise difference values weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can obtain the deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.; To resolve (ii), we first learn the weight difference initialization by warm-up the fine-tuning using D for steps Nsteps and get W f. We then approximate in the weights of the second layer of the first neural network and weights of the second layer of the second neural network Wδ using a task-specific initialization as Wδ = W f −Wp. Our ablation study shows that using task-specific warm-up shows better results compared to random initialization as this accumulation of gradients can better reflect the final weight difference distribution.), and
the second delta weights comprises difference values in the weights of the second layer of the first neural network and weights of the second layer of the third neural network ([Wδ pruning.] the addition between pre-trained weight Wp and a the second delta weights comprises difference values weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can obtain the deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.; To resolve (ii), we first learn the weight difference initialization by warm-up the fine-tuning using D for steps Nsteps and get W f. We then approximate in the weights of the second layer of the first neural network and weights of the second layer of the third neural network Wδ using a task-specific initialization as Wδ = W f −Wp. Our ablation study shows that using task-specific warm-up shows better results compared to random initialization as this accumulation of gradients can better reflect the final weight difference distribution.).
Ravishankar and Fu are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Ravishankar, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Fu to Ravishankar before the effective filing date of the claimed invention in order to leverage both computation and parameter sharing across multiple tasks (cf. Fu, [Abstract] Task-specific fine-tuning on pre-trained transformers has achieved performance breakthroughs in multiple NLP tasks. Yet, as both computation and parameter size grows linearly with the number of sub-tasks, it is increasingly difficult to adopt such methods to the real world due to unrealistic memory and computation overhead on computing devices. Previous works on fine-tuning focus on reducing the growing parameter size to save storage cost by parameter sharing. However, compared to storage, the constraint of computation is a more critical issue with the fine-tuning models in modern computing environments. In this work, we propose LeTS, a framework that leverages both computation and parameter sharing across multiple tasks. Compared to traditional fine-tuning, LeTS proposes a novel neural architecture that contains a fixed pre-trained transformer model, plus learnable additive components for sub-tasks. The learnable components reuse the intermediate activations in the fixed pre-trained model, decoupling computation dependency. Differentiable neural architecture search is used to determine a task-specific computation sharing scheme, and a novel early stage pruning is applied to additive components for sparsity to achieve parameter sharing. Extensive experiments show that with 1.4% of extra parameters per task, LeTS reduces the computation by 49.5% on GLUE benchmarks with only 0.2% accuracy loss compared to full fine-tuning.).
Regarding claim 2, Ravishankar, as modified by Fu, teaches The method of claim 1.
Fu teaches wherein the obtaining of the first delta weights and the second delta weights comprises: obtaining first compressed data stored corresponding to the second layer of the second neural network; and obtaining the first delta weights by decoding the first compressed data ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask. (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the obtaining first compressed data stored corresponding to the second layer of the second neural network computation can be reduced into a sparse matrix multiplication by obtaining the first delta weights by decoding the first compressed data leveraging the sparsity produced from (iii).).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 4, Ravishankar, as modified by Fu, teaches The method of claim 1.
Fu teaches wherein the obtaining of the first delta weights and the second delta weights comprises: obtaining second compressed data stored corresponding to the second layer of the third neural network; and obtaining the second delta weights by decoding the second compressed data ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask. (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the obtaining second compressed data stored corresponding to the second layer of the third neural network computation can be reduced into a sparse matrix multiplication by obtaining the second delta weights by decoding the second compressed data leveraging the sparsity produced from (iii).).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 8, Ravishankar teaches A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors ([0047] The memory 156 may encompass any tangible, non-transitory computer-readable storage medium non-transitory medium for storing instructions storing data or executable routines, including volatile memory, non-volatile memory, or any combination thereof. Although shown for convenience as a single block in FIG. 8, the memory 156 may actually encompass various discrete media in the same or different physical locations. The executed by one or more processors one or more processors 152 may access data in the memory 156 via one or more busses 154.),
configure the one or more processors to perform the method of claim 1 (see rejection of claim 1).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 9, Ravishankar, as modified by Fu, teaches The method of claim 1.
Fu teaches further comprising: in response to a request to change the first task to a third task, obtaining, from memory, third delta weights previously computed and stored, wherein the third delta weights are associated with a second layer of a fourth neural network ([Search space.] As shown in Figure 4, we decouple data dependency across layers by using a pooling layer, a linear layer, and a Bi-LSTM to aggregate all layers’ output for final classification. The pooling layer uses the first hidden vector corresponding to the first token (i.e., [CLS] token) (Devlin et al., 2019) as the layer presentation. The pooling output vectors are then fed into a linear layer and a Bi-LSTM. LeTS first builds up a stochastic super network Wτ for the searching phase. Before searching, we copy the weights from Wp to Wτ trainable layers and disable the gradient computation based on cτ which is the weight difference mask obtained from the Delta-Pruning. Two decisions should be made in each attention layer Wτ j : (i) the input to the trainable layer. It can be either the cached result (xp j−1) or the output from the previous trainable layer (xf j−1) (ii) the output to the pooling layer from layer j. It can be either xp j or xf j . The total size of the search space would be 4N where N is the layer number in the pre-training mode (≈ 1015 for BERTLARGE).; [(i) Bypass self-attention layers.] When the cached result xp j is used by the final pooling layer and next trainable layer, the obtaining, from memory computation and parameters of the entire layer can be saved. This can be applied at layer Wp j where j ∈ {0,1,2,6}.; [(ii) Exploit the sparsity of Wδ.] LeTS can leverage the sparsity feature of Wδ j . More specifically, when the input to the trainable attention layer is xp j−1, LeTS computes xp j−1 · third delta weights previously computed and stored Wδ j and adds it to a cached result.; [3.1. Delta-Pruning in Early Stage] Delta-Pruning is motivated by SNIP (Lee et al., 2019) which targets to generate weight sparsity before training. We decompose the final fine-tuned weight (W f) into two parts as shown in Eq. (1). Wf =Wp+c Wδ (1) Here, Wδ ∈ Rd is the fine-tuning weight difference, c ∈ {0,1}d is the generated mask for Wδ. To resolve (ii), we first learn the weight difference initialization by warm-up the fine-tuning using D for steps Nsteps and get W f. We then wherein the third delta weights are associated with a second layer of a fourth neural network approximate Wδ using a task-specific initialization as Wδ = W f −Wp. Our ablation study shows that using task-specific warm-up shows better results compared to random initialization as this accumulation of gradients can better reflect the final weight difference distribution.; [6. Conclusion] We propose LeTS, a transfer learning framework that achieves computation and parameter sharing in response to a request to change the first task to a third task when multiple tasks arriving in stream. LeTS proposes a novel architecture space that can reuse computed results to reduce computation. By leveraging NAS with a computation-aware loss function, LeTS can find models with high task performance and low computation overhead. By treating the fine-tuned weight as the sum of pre-trained weight and weight difference, we present a early-stage pruning algorithm to compress weight difference without task performance decrease.),
wherein the fourth neural network is fine-tuned from the first neural network based on the third task ([Wδ pruning.] In this work, we treat the final fine-tuning weight W f as the addition between pre-trained weight Wp and a weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can wherein the fourth neural network is fine-tuned from the first neural network based on the third task obtain the deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.);
replacing the first delta weights with the third delta weights ([Wδ pruning.] In this work, we treat the final fine-tuning weight W f as the addition between pre-trained weight Wp and a weight difference (Wδ). By proposing an early-stage pruning approach, called Delta-Pruning, we compute the connection sensitivity of Wδ, which reveals the important connections in the Wδ for a given task (See Sec. 3.1). In this way, we can obtain the replacing the first delta weights with the third delta weights deterministic task-specific mask at the beginning of fine-tuning and use the generated mask to guide NAS.); and
performing an operation of the fourth neural network ([1. Introduction] (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the computation can be reduced into a sparse matrix multiplication by leveraging the sparsity produced from (iii).)
based on sums of the weights of the second layer of the first neural network and the third delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on sums of the weights of the second layer of the first neural network and the third delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 10, Ravishankar, as modified by Fu, teaches The method of claim 9.
Fu teaches wherein the performing of the operation of the fourth neural network comprises: performing the operation of the fourth neural network on a third output data ([1. Introduction] (iv) We systematically integrate (ii) and (iii) to produce fine tuning models with high task performance and low computation and storage cost. During NAS, a generated mask from (ii) on the trainable parameters can better characterize the model performance. Also, during the online prototyping, when the input and output of a given linear layer is already computed, the computation can be reduced into a sparse matrix multiplication by leveraging the sparsity produced from (iii).)
based on sums of the weights of the second layer of the first neural network and the third delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on sums of the weights of the second layer of the first neural network and the third delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.),
wherein the third output data is obtained from the first output data or is an output of a first layer of the fourth neural network ([LeTS design.] Motivated by these observations, we pro pose a novel fine-tuning architecture that can reduce computation by reusing computed results. Also, the new architecture decouples the data dependency of different layers to enable speedup. Given input query xin (Figure 2(b)), LeTS first caches all N attention layers’ output (xp j , j ∈ {0,1,...,N −1}) computed from input query xin and pre-trained model Wp. For a given layer j in sub-task s, the wherein the third output data is obtained input to the trainable layer W f j can be chosen from from the first output data cached result xp j−1 or is an output of a first layer of the fourth neural network or the computed result xf j−1 from the previous trainable layer. The attention output to the pooling layer can be chosen from (i) xp j or (ii) xf j . LeTS uses pooling and Bi-LSTM to aggregate the outputs from attention layers to generate the final result. For each trainable layer W f j , we treat W f j as Wp j +Wδ j and make Wδ j sparse using our proposed delta pruning algorithm.).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Regarding claim 12, Ravishankar, as modified by Fu, teaches The apparatus of claim 11.
Fu teaches further comprising: a memory configured to store weights of one or more layers included in the first neural network and delta weights corresponding to the second neural network and the third neural network ([LeTS design.] Given input query xin (Figure 2(b)), LeTS first caches all N attention layers’ output (xp j , j ∈ {0,1,...,N −1}) computed from input query xin and pre-trained model Wp. For a given layer j in sub-task s, the input to the trainable layer W f j can be chosen from cached result xp j−1 or the computed result xf j−1 from the previous trainable layer. The attention output to the pooling layer can be chosen from (i) xp j or (ii) xf j . LeTS uses pooling and Bi-LSTM to aggregate the outputs from attention layers to generate the final result. For each trainable layer W f j , we treat W f j as Wp j +Wδ j and make Wδ j sparse using our proposed delta pruning algorithm. We use an example architecture in Figure 2(b) to illustrate the advantages of the new architecture: (i) Bypass self-attention layers. When the a memory configured to store weights of one or more layers included in the first neural network and delta weights corresponding to the second neural network and the third neural network cached result xp j is used by the final pooling layer and next trainable layer, the computation and parameters of the entire layer can be saved. This can be applied at layer Wp j where j ∈ {0,1,2,6}.).
Ravishankar and Fu are combinable for the same rationale as set forth above with respect to claim 1.
Claims 3, 5 are rejected under 35 U.S.C. 103 as being unpatentable over Ravishankar, Fu, and further in view of Ribalta et al. (U.S. Pre-Grant Publication No. 20210397943, hereinafter 'Ribalta').
Regarding claim 3, Ravishankar, as modified by Fu, teaches The method of claim 2.
Ravishankar, as modified by Fu, fails to teach wherein the first compressed data comprises metadata storing a position of a non-zero weight and a value of the non-zero weight, of the first delta weights.
Ribalta teaches wherein the first compressed data comprises metadata storing a position of a non-zero weight and a value of the non-zero weight, of the first delta weights ([0169] Embodiments described herein allow for multiple neural networks to be performed simultaneously and/or sequentially, and for results to be combined together to enable Level 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN executing on a DLA or a discrete GPU (e.g., GPU(s) 1020) may include text and word recognition, allowing reading and understanding of traffic signs, including signs for which a neural network has not been specifically trained. In at least one embodiment, a DLA may further include a neural network that is able to identify, interpret, and provide semantic understanding of a sign, and to pass that semantic understanding to path planning modules running on a CPU Complex.; [0061] In at least one embodiment, neural network training logic 128 is used to train an untrained neural network 130 using the first compressed data comprises compressed training data 116 to generate a trained neural network 132.; [0084] In at least one embodiment, inference and/or training logic 715 may include, without limitation, a code and/or data storage 705 to metadata storing a position of a non-zero weight and a value of the non-zero weight, of the first delta weights store backward and/or output weight and/or input/output data corresponding to neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments. In at least one embodiment, code and/or data storage 705 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input/output data and/or weight parameters during training and/or inferencing using aspects of one or more embodiments. In at least one embodiment, training logic 715 may include, or be coupled to code and/or data storage 705 to store graph code or other software to control timing and/or order, in which weight and/or other parameter information is to be loaded to configure, logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs).; [0085] In at least one embodiment, code, such as graph code, causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, any portion of code and/or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and/or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, a choice of whether code and/or data storage 705 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash memory or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors.).
Ravishankar, Fu, and Ribalta are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Ravishankar and Fu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Ribalta to Ravishankar before the effective filing date of the claimed invention in order to improve the amount of memory, time, or computing resources used to train neural networks and perform inferencing with neural networks (cf. Ribalta, [0002] Training neural networks and performing inferencing with neural networks can use significant memory, time, or computing resources. The amount of memory, time, or computing resources used to train neural networks and perform inferencing with neural networks can be improved.).
Regarding claim 5, Ravishankar, as modified by Fu, teaches The method of claim 4.
Ravishankar, as modified by Fu, fails to teach wherein the second compressed data comprises metadata storing a position of a non-zero weight and a value of the non-zero weight, of the second delta weights.
Ribalta teaches wherein the second compressed data comprises metadata storing a position of a non-zero weight and a value of the non-zero weight, of the second delta weights ([0169] Embodiments described herein allow for multiple neural networks to be performed simultaneously and/or sequentially, and for results to be combined together to enable Level 3-5 autonomous driving functionality. For example, in at least one embodiment, a CNN executing on a DLA or a discrete GPU (e.g., GPU(s) 1020) may include text and word recognition, allowing reading and understanding of traffic signs, including signs for which a neural network has not been specifically trained. In at least one embodiment, a DLA may further include a neural network that is able to identify, interpret, and provide semantic understanding of a sign, and to pass that semantic understanding to path planning modules running on a CPU Complex.; [0061] In at least one embodiment, neural network training logic 128 is used to train an untrained neural network 130 using the second compressed data comprises compressed training data 116 to generate a trained neural network 132.; [0084] In at least one embodiment, inference and/or training logic 715 may include, without limitation, a code and/or data storage 705 to metadata storing a position of a non-zero weight and a value of the non-zero weight, of the second delta weights store backward and/or output weight and/or input/output data corresponding to neurons or layers of a neural network trained and/or used for inferencing in aspects of one or more embodiments. In at least one embodiment, code and/or data storage 705 stores weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during backward propagation of input/output data and/or weight parameters during training and/or inferencing using aspects of one or more embodiments. In at least one embodiment, training logic 715 may include, or be coupled to code and/or data storage 705 to store graph code or other software to control timing and/or order, in which weight and/or other parameter information is to be loaded to configure, logic, including integer and/or floating point units (collectively, arithmetic logic units (ALUs).; [0085] In at least one embodiment, code, such as graph code, causes the loading of weight or other parameter information into processor ALUs based on an architecture of a neural network to which such code corresponds. In at least one embodiment, any portion of code and/or data storage 705 may be included with other on-chip or off-chip data storage, including a processor's L1, L2, or L3 cache or system memory. In at least one embodiment, any portion of code and/or data storage 705 may be internal or external to one or more processors or other hardware logic devices or circuits. In at least one embodiment, code and/or data storage 705 may be cache memory, DRAM, SRAM, non-volatile memory (e.g., flash memory), or other storage. In at least one embodiment, a choice of whether code and/or data storage 705 is internal or external to a processor, for example, or comprising DRAM, SRAM, flash memory or some other storage type may depend on available storage on-chip versus off-chip, latency requirements of training and/or inferencing functions being performed, batch size of data used in inferencing and/or training of a neural network, or some combination of these factors.).
Ravishankar, Fu, and Ribalta are combinable for the same rationale as set forth above with respect to claim 3.
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over Ravishankar, Fu, and further in view of Gonzalez et al. (U.S. Pre-Grant Publication No. 20220012563, hereinafter 'Gonzalez').
Regarding claim 6, Ravishankar, as modified by Fu, teaches The method of claim 1.
Fu teaches wherein the performing of the operation of the second neural network comprises: based on the sums of the weights of the second layer of the first neural network and the first delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on the sums of the weights of the second layer of the first neural network and the first delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.); and
Ravishankar, as modified by Fu, fails to teach restoring weights of the second layer of the second neural network, performing the operation of the second neural network on the first output data, based on the restored weights.
Gonzalez teaches restoring weights of the second layer of the second neural network, performing the operation of the second neural network on the first output data, based on the restored weights ([0043] Accordingly, the data lanes in the partition can be performing the operation of the second neural network on the first output data, based on the restored weights decompressed in response to the data lanes including their original respective data, which allows the neural network to utilize the weights to perform an inference. Additionally, different decompression engines can operate in parallel on different data lanes as the data lanes or portions thereof become decoded. As such, the weights can be decompressed in-line with the neural network performing an inference. For example, a first layer of the restoring weights of the second layer of the second neural network weights can be decompressed and utilized by the neural network as a second layer of the weights is decoded. As such, the neural network can operate with a high throughput (e.g., to the order of 64 Gigabytes per second) as weights are decompressed in parallel as the neural network performs the inference, which eliminates or otherwise reduces time that the neural network spends waiting for the weights to be decompressed.).
Ravishankar, Fu, and Gonzalez are considered to be analogous to the claimed invention because they are in the same field of machine learning. In view of the teachings of Ravishankar and Fu, it would have been obvious for a person of ordinary skill in the art to apply the teachings of Gonzalez to Ravishankar before the effective filing date of the claimed invention in order to decrease the amount of storage required for complex neural networks (cf. Gonzalez, [0033] To decrease the amount of storage required for complex neural networks, techniques have been developed to sparsify the neural network weights without significantly impacting the accuracy of the network. For instance, the weights may encounter pruning, which removes weights below a certain threshold, and the network can be retrained to determine the removed weights based on weights that were not removed. Additionally, the weights may encounter quantization to reduce the number of bits required to store each weight. For instance, weights can be divided into partitions to reduce a size thereof.).
Regarding claim 7, Ravishankar, as modified by Fu, teaches The method of claim 1.
Fu teaches wherein the performing of the operation of the second neural network comprises: based on the sums of the weights of the second layer of the first neural network and the second delta weights ([1. Introduction] (iii) We treat the obtained fine-tuning model weights as the based on the sums of the weights of the second layer of the first neural network and the second delta weights sum of pre-trained weights and weight difference (δ): Wf =Wp+Wδ, and propose a novel early-stage pruning method to obtain Wδ. A weight mask to represent pruning is generated for Wδ at the beginning of the fine-tuning. Rather than randomly initialized, Wδ is initialized with task specific gradient accumulation to get a robust weight mask.); and
Ravishankar, as modified by Fu, fails to teach restoring weights of the second layer of the third neural network, performing the operation of the third neural network on the second output data, based on the restored weights.
Gonzalez teaches restoring weights of the second layer of the third neural network, performing the operation of the third neural network on the second output data, based on the restored weights ([0043] Accordingly, the data lanes in the partition can be performing the operation of the third neural network on the second output data, based on the restored weights decompressed in response to the data lanes including their original respective data, which allows the neural network to utilize the weights to perform an inference. Additionally, different decompression engines can operate in parallel on different data lanes as the data lanes or portions thereof become decoded. As such, the weights can be decompressed in-line with the neural network performing an inference. For example, a first layer of the restoring weights of the second layer of the third neural network weights can be decompressed and utilized by the neural network as a second layer of the weights is decoded. As such, the neural network can operate with a high throughput (e.g., to the order of 64 Gigabytes per second) as weights are decompressed in parallel as the neural network performs the inference, which eliminates or otherwise reduces time that the neural network spends waiting for the weights to be decompressed.).
Ravishankar, Fu, and Gonzalez are combinable for the same rationale as set forth above with respect to claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAGGIE MAIDO whose telephone number is (703) 756-1953. The examiner can normally be reached M-Th: 6am - 4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached on (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MM/Examiner, Art Unit 2129
/MICHAEL J HUNTLEY/Supervisory Patent Examiner, Art Unit 2129
Prosecution Insights