DETAILED ACTION
This action is responsive to the application filed on 03/20/2026. Claims 1,4,8,11,15 and 18 are pending and have been examined. This action is Non-final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C.
120, 121, 365(c), or 386(c) is acknowledged.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 03/20/2026 has been entered.
Response to Arguments
Argument 1: The applicant argues that the 103 rejection should be withdrawn because claims 6, 13, and 19 have been canceled, making the rejection moot as to those claims, and because the amended claims are asserted to overcome the currently cited art. The applicant states that, during the examiner interview, it was generally agreed that the amendments overcame the current art, although further search and consideration would be required, and on that basis contends that any proposed combination of Song, Lin, McMahan, and Han fails to disclose or suggest at least the limitations of amended independent claims 1, 8, and 15. The applicant then argues that, because the cited combination allegedly does not support the rejection of amended independent claims 1, 8, and 15, it likewise does not support the rejection of dependent claims 4, 11, and 18, and therefore withdrawal of the rejection is requested.
Examiner Response to Argument 1: The examiner acknowledges that claims 6, 13, and 19 have been canceled, and the rejection is therefore moot as to those claims. However, applicant’s remarks are not persuasive because, although applicant generally asserts that the amendments overcome the cited art and references the examiner interview, the remarks do not identify a specific error in the rejection of amended claims 1, 8, and 15. Further, any discussion during the interview that additional search and consideration might be warranted did not constitute agreement that the amended claims were allowable. Upon further search and consideration, the newly added limitations remain taught or at least suggested by the applied combination. In particular, the added limitations directed to receipt of quantized model updates and decoding at the server are taught by Shlezinger, the added range based quantization and interval assignment limitations are taught by McMahan, the added Bayesian shape parameter and training cycle updating limitations are taught by Song, and the added limitation that the range is not derived from statistics of each client node is taught by Han. Accordingly, applicant has not shown error in the rejection, and the rejection of claims 1, 8, and 15 under 35 USC 103 is maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1,4,8,11,15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over NPL reference Lin et. al. “DEEP GRADIENT COMPRESSION: REDUCING THE COMMUNICATION BANDWIDTH FOR DISTRIBUTED TRAINING” (referred herein as Lin) in view of Shlezinger et. al. “UVeQFed: Universal Vector Quantization for Federated Learning” in view of US10657461B2 by McMahan et. al. (referred herein as McMahan) in view of NPL reference Song et. al., “COMMUNICATION EFFICIENTSGD VIA GRADIENT SAMPLING WITH BAYES PRIOR” (referred herein as Song) further in view of NPL reference Han et. al. “DEEP COMPRESSION:COMPRESSINGDEEP NEURAL NETWORKS WITH PRUNING, TRAINED QUANTIZATION AND HUFFMAN CODING” (referred herein as Han).
Regarding claim 1, Lin teaches:
A method for model updating in a federated learning environment, the method comprising: distributing, by a model coordinator, a current model to a plurality of client nodes, ([Lin, page 13] “The nodes push the gradients to the servers while the servers are waiting for the gradients from all nodes. Once all gradients are sent, the servers update the parameters, and then all nodes pull the latest parameters from the servers,” wherein the examiner interprets “all nodes pull the latest parameters from the servers” to be the same as “distributing, by a model coordinator, a current model to a plurality of client nodes” because they are both directed to a central server providing current model parameters to distributed training nodes).
wherein, upon receiving the current model, a client node of the plurality of client nodes trains the current model using local training data to generate a first set of gradient K-quant vectors, ([Lin, page 13] “The nodes push the gradients to the servers” and [Lin, page 3] “We reduce the communication bandwidth by sending only the important gradients (sparse update),” wherein the examiner interprets “nodes push the gradients to the servers” and “sending only the important gradients” to be the same as a client node training the current model using local data to generate a first set of gradient K-quant vectors because they are both directed to a client-side training process that produces compressed gradient update information for transmission to the server).
wherein the first set of gradient K-quant vectors is received instead of the local training data to minimize network bandwidth usage, ([Lin, page 13] “The nodes push the gradients to the servers” and [Lin, page 1] “Deep Gradient Compression ... greatly reduce the communication bandwidth,” wherein the examiner interprets “push the gradients to the servers” and “reduce the communication bandwidth” to be the same as receiving gradient K-quant vectors instead of local training data to minimize network bandwidth usage because they are both directed to sending compressed gradient update information rather than raw local data to reduce communication burden).
Lin teaches generating, by the model coordinator, a first updated model based on the first set of gradient K-quant vectors, ([Lin, page 13] “Once all gradients are sent, the servers update the parameters,” wherein the examiner interprets “the servers update the parameters” to be the same as “generating, by the model coordinator, a first updated model” because they are both directed to the server updating the model based on received gradient information).
generating, by the model coordinator, a first updated model based on the first set of gradient K-quant vectors, ([Lin, page 13] “The gradients from all nodes are summed up to optimize their models…Once all gradients are sent, the servers update the parameters,” wherein the examiner interprets “the servers update the parameters” to be the same as “generating, by the model coordinator, a first updated model” because they are both directed to the server updating the model based on received gradient information).
distributing the first updated model to the plurality of client nodes;…distributing the second updated model to the plurality of client nodes. ([Lin, page 13, Appendix A] “The gradients from all nodes are summed up to optimize their models. By this synchronization step, models on different nodes are always the same during the training.”, wherein the examiner interprets “models on different nodes are always the same during the training” to be the same as distributing the [first and/or second] updated model to client nodes, where the synchronization step (a.k.a. model coordinator) ensures that all nodes have the same updated model, similar to the claim's distribution of the first updated model to client nodes.)
Lin does not teach receiving, by the model coordinator and in response to distributing the current model, the first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from the client node…wherein generating the first updated model comprises: decoding, by the model coordinator, each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values, wherein the plurality of bin index values corresponds to a gradient vector position of each gradient K-quant vector of the first set of gradient K-quant vectors, wherein the model coordinator generates the first updated model using the plurality of bin index values, wherein each gradient K-quant vector is a vector of values that represents actual gradient values modified based on a K value, a user-defined minimum value of a range, and a user-defined maximum value of the range that the model coordinator sent to each client node, wherein each client node normalizes the actual gradient values to fall within the range and to obtain normalized gradient values, wherein each normalized gradient value is associated with a bin index value corresponding to a sub-portion of the range, storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, and making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters.
Shlezinger teaches:
receiving, by the model coordinator and in response to distributing the current model, the first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from the client node, ([Shlezinger, page 4] “time instance t, the server shares its current model, represented by the vector wt ∈ Rm, with the users” and “the kth user needs to communicate a finite-bit quantized representation of its model update to the server,” wherein the examiner interprets “finite-bit quantized representation of its model update” to be the same as “gradient K-quant vectors” because they are both directed to quantized model-update information sent from a client-side participant to the server after the server distributes the current model).
wherein generating the first updated model comprises: decoding, by the model coordinator, each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values, ([Shlezinger, page 4] “Quantization consists of encoding the model update into a set of bits, and decoding each bit combination into a recovered model update” and [Shlezinger, page 8] “The server first decodes each digital codeword,” wherein the examiner interprets “decoding each bit combination” and “decodes each digital codeword” to be the same as server-side decoding of received quantized update representations because they are both directed to the server decoding quantized transmitted update information before model updating).
wherein the model coordinator generates the first updated model using the plurality of bin index values, ([Shlezinger, page 4] “The server uses the received codewords ... to reconstruct h-hat” and “Finally, the global model wt+τ is updated via wt+τ = wt + h-hat,” wherein the examiner interprets “uses the received codewords ... to reconstruct” and “the global model ... is updated via” to be the same as generating the first updated model using received quantized update information because they are both directed to reconstructing a transmitted quantized update at the server and using it to update the model).
Lin and Shleizinger do not teach wherein each gradient K-quant vector is a vector of values that represents actual gradient values modified based on a K value, a user-defined minimum value of a range, and a user-defined maximum value of the range that the model coordinator sent to each client node, wherein each client node normalizes the actual gradient values to fall within the range and to obtain normalized gradient values, wherein each normalized gradient value is associated with a bin index value corresponding to a sub-portion of the range, storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters,.
McMahan teaches:
wherein each gradient K-quant vector is a vector of values that represents actual gradient values modified based on a K value, a user-defined minimum value of a range, and a user-defined maximum value of the range that the model coordinator sent to each client node, ([McMahan, page 4] “for b-bit quantization, [hmin, hmax] can be equally divided into 2b intervals” and “The client will typically transmit the min and max to the server regardless of whether dynamic spacing is used or not,” wherein the examiner interprets “2b intervals” to be the same as a K value and “hmin, hmax” to be the same as a user-defined minimum value and a user-defined maximum value of a range because they are both directed to quantizing update values by reference to a bounded range subdivided into a finite number of intervals).
wherein each client node normalizes the actual gradient values to fall within the range and to obtain normalized gradient values, wherein each normalized gradient value is associated with a bin index value corresponding to a sub-portion of the range, ([McMahan, page 4] “for b-bit quantization, [hmin, hmax] can be equally divided into 2b intervals. Suppose hi falls in the interval bounded by h′ and h″. The quantization can operate by replacing hmin and hmax ...with h′ and h,” wherein the examiner interprets “hi falls in the interval bounded by h′ and h″” to be the same as a normalized gradient value being associated with a bin index value corresponding to a sub-portion of the range because they are both directed to assigning each value to one of multiple bounded intervals within a quantization range).
Lin, Shleizinger, and McMahan do not teach storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters.
Song teaches:
storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, ([Song, page 4] “we adopt Bayes’ theorem to express the posterior sampling probability” and [Song, page 4] “In each training iteration, pt,i undergoes L1 normalization after it is updated by Eq. (12),” wherein the examiner interprets the posterior sampling probability and its iterative Bayesian updating to be the same as a plurality of shape parameters determining the shape of a probability distribution and being updated using a Bayesian update because they are both directed to using parameters of a probabilistic gradient-distribution model that are updated over time using Bayes-based updating).
generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, ([Song, page 1] “we sample important/large gradients based on the global gradient distribution, which is periodically updated across multiple workers” and [Song, page 4] “the posterior sampling probability” is updated, wherein the examiner interprets “global gradient distribution” and “posterior sampling probability” to be the same as using shape-parameter-based probabilistic information during model updating because they are both directed to updating the model using received gradient information together with updated distributional parameters).
making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters, ([Song page 1] “Specifically, we sample important/large gradients based on the global gradient distribution, which is periodically updated across multiple workers. Then we intro duce Bayes Prior into distribution model to further explore the gradients.” AND [Song, page 4] “So, we use the historical gradient distribution ▽f(xt0) as the sampling distribution for the subsequent tτ training steps: ▽f(xt0) can be obtained by one All-Reduce communication. In the implementation of Gradient Sampling, τ is set as large as 100,” wherein the examiner interprets use of a defined synchronization period τ and subsequent repeated training steps before gradient-distribution refresh to be the same as determining that a training session has ended, determining that an additional cycle remains, and updating the plurality of shape parameters for the next cycle because they are both directed to updating the probabilistic gradient-distribution parameters after a completed training interval and before a subsequent interval).
Lin, Shlezinger, McMahan, and Song do not teach wherein the range is not derived from any statistics of each client node of the plurality of client nodes, wherein the plurality of bin index values corresponds to a gradient vector position of each gradient K-quant vector of the first set of gradient K-quant vectors.
Han teaches:
wherein the range is not derived from any statistics of each client node of the plurality of client nodes; ([Han, page 4] “Linear initialization linearly spaces the centroids between the [min, max] of the original weights. This initialization method is invariant to the distribution of the weights,” wherein the examiner interprets “linearly spaces the centroids between the [min, max]” and “invariant to the distribution of the weights” to be the same as “the range is not derived from any statistics of each client node of the plurality of client nodes” because they are both directed to defining a quantization-related range without basing that range on underlying distribution statistics).
wherein the plurality of bin index values corresponds to a gradient vector position of each gradient K-quant vector of the first set of gradient K-quant vectors; ([Han, page 5] “An index into the shared weight table is stored for each connection. During back-propagation, the gradient for each shared weight is calculated and used to update the shared weight. This procedure is shown in Figure 3...Quantization and weight sharing are implemented by maintaining a codebook structure that stores the shared weight, and group-by-index after calculating the gradient of each layer. Each shared weight is updated with all the gradients that fall into that bucket.”, wherein the examiner interprets “An index into the shared weight table is stored for each connection...During back-propagation, the gradient for each shared weight is calculated and used to update the shared weight...Quantization and weight sharing are implemented by maintaining a codebook structure that stores the shared weight...Each shared weight is updated with all the gradients that fall into that bucket” to be the same as “the plurality of bin index values corresponds to a gradient vector position” because they are both directed to associating an index value with a particular element position in a compressed representation used for model updating.)
Lin, Shlezinger, McMahan, Song, Han, and the instant application are analogous art because they are all directed to communication-efficient distributed or federated model training in which a central server or coordinator receives compressed or quantized model-update information from multiple client-side participants and updates a shared model.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the distributed model-updating method disclosed by Lin to include the “the kth user needs to communicate a finite-bit quantized representation of its model update to the server” and “The server first decodes each digital codeword” disclosed by Shlezinger. One would be motivated to do so to efficiently communicate quantized model-update information from client-side participants to the server and decode the communicated update information for server-side reconstruction and updating, as suggested by Shlezinger (Shlezinger, page 4, “the kth user needs to communicate a finite-bit quantized representation of its model update to the server” and page 8, “The server first decodes each digital codeword.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the distributed model-updating method disclosed by Lin to include the “for b-bit quantization, [hmin, hmax] can be equally divided into 2b intervals” disclosed by McMahan. One would be motivated to do so to effectively quantize gradient-update values using a bounded range subdivided into intervals so that compressed update information can be communicated with reduced bandwidth while maintaining interval-based value representation, as suggested by McMahan (McMahan, page 4, “for b-bit quantization, [hmin, hmax] can be equally divided into 2b intervals.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the distributed model-updating method disclosed by Lin to include the “we adopt Bayes’ theorem to express the posterior sampling probability” disclosed by Song. One would be motivated to do so to effectively use probabilistic gradient-distribution parameters that are updated using Bayesian updating during model training so that later model updates can be generated using updated distributional information, as suggested by Song (Song, page 4, “we adopt Bayes’ theorem to express the posterior sampling probability.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the distributed model-updating method disclosed by Lin to include the “Linear initialization linearly spaces the centroids between the [min, max] of the original weights” disclosed by Han. One would be motivated to do so to effectively define a quantization-related range without relying on underlying distribution statistics, as suggested by Han (Han, page 4, “This initialization method is invariant to the distribution of the weights.”).Claims 8 and 15 are analogous to claim 1, and thus is similarly rejected as above.
Regarding claim 4, Lin, Shlezinger, McMahan, Song, and Han teaches The method of claim 1 (see rejection of claim 1).
Lin further teaches wherein, before receiving the first set of gradient K-quant vectors, the method further comprises distributing, by the model coordinator and to the plurality of client nodes ([Lin, page 3, Sec 3.1] “We reduce the communication bandwidth by sending only the important gradients (sparse update). We use the gradient magnitude as a simple heuristic for importance: only gradients larger than a threshold are transmitted. To avoid losing information, we accumulate the rest of the gradients locally. Eventually, these gradients become large enough to be transmitted. Thus, we send the large gradients immediately but eventually send all of the gradients overtime, as shown in Algorithm 1. The encode() function packs the 32-bit nonzero gradient values and 16-bit run lengths of zeros,” wherein the examiner interprets “only gradients larger than a threshold are transmitted” and “send the large gradients immediately but eventually send all of the gradients over time” to be the same as selectively distributing gradients to client nodes. The threshold value defined by Lin is, by definition, the minimum value of the range, and the maximum is the largest gradient value)
Han further teaches a K value and a range, wherein the plurality of client nodes use the Value and the range to generate the first set of gradient K-quant vectors. ([Han, page 3, sec 3] “The weights are quantized to 4 bins (denoted with 4 colors), all the weights in the same bin share the same value, thus for each weight, we then need to store only a small index into a table of shared weights. During update, all the gradients are grouped by the color,” wherein the examiner interprets “quantized to 4 bins” and “quantizing weights to bins, each bin sharing a common value “to be the same as K=4 and “using a K value and range to generate gradient K-quant vectors”. respectively).
Lin, Shlezinger, McMahan, Song, Han, and the instant application are analogous art because they are all directed to model updating in a federated learning environment.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the method of claim 1 disclosed by Lin, Shlezinger, McMahan, Song, and Han to include the selection of gradient values to be distributed to client nodes and to include the “quantizing weights to bins, each bin sharing a common value” disclosed by Lin and Han. One would be motivated to do so to effectively standardize gradient values for efficient model updating across client nodes, as suggested by Han (Han, [page 3, sec 3] “The weights are quantized to 4 bins...all the gradients are grouped by the color,”). Claims 11 and 18 are analogous to claim 4, and therefore will face the same rejection.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
Prosecution Insights