DETAILED ACTION
This action is responsive to the application filed on 07/29/2026. Claims 1, 4, 8, 11, 15, and 18 are pending and have been examined. This action is Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Argument 1: The applicant argues that the 35 U.S.C. 103 rejection of claims 1, 4, 8, 11, 15, and 18 over Lin in view of Shlezinger, McMahan, Song, and Han should be withdrawn because amended independent claims 1, 8, and 15 are not taught or suggested by the cited art, whether the references are considered separately or in combination. The applicant cites paragraphs [0027], [0036]-[0037], and [0042]-[0044] of the specification as support for the amendments, asserts that no new matter is added, and notes that the amendments were discussed at the July 21, 2026 examiner interview without agreement on allowability. The applicant makes three arguments. First, the applicant argues that Han, relied upon for "wherein the range is not derived from any statistics of each client node," teaches only an initialization method that is invariant to the distribution of the weights, and is silent as to the range being predefined by a model coordinator and being completely independent from any statistics of each client node, and that none of the other cited art discloses this limitation either. Second, the applicant argues that Shlezinger, relied upon for decoding each gradient K-quant vector to obtain bin index values, is silent as to the decoding further comprising replacing each bin index value with a mean value of the corresponding sub-portion of the range based on an upper value and a lower value of the range. In other words, Shlezinger does not contemplate reconstruction using mean bin values tied to a predefined range, and no other cited art discloses this decoding architecture. Third, the applicant argues that neither Song, relied upon for generating the second updated model from the second set of gradient K-quant vectors and the shape parameters, nor any other cited art discloses computing an expected gradient value for each gradient vector position of the second updated model using the shape parameters based on the Bayesian update, so that the cited art lacks the claimed pipeline of per-position Bayesian parameter updates and expected value computation. On that basis, the applicant contends that the examiner has not established a prima facie case of obviousness, so the applicant is under no obligation to submit evidence of non-obviousness (MPEP 2142). The applicant further argues that dependent claims 4, 11, and 18 are allowable for the same reasons because they depend from claims 1, 8, and 15. Finally, the applicant states that it does not concede any characterization of the art or any assertion that particular limitations are well known, and it reserves the right to present additional arguments later.
Response to Argument 1: The examiner has considered the argument set forth above, however it is not persuasive. The newly added limitations are now addressed by Bonawitz, Gray, and Yurochkin, so the applicant’s assertions that Han, Shlezinger, and Song are each silent on those limitations do not reach the rejection set forth above. Furthermore, nonobviousness cannot be shown by attacking references individually where the rejection is based on a combination of references (In re Keller, 642 F.2d 413, 426 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986)). First, as to the range being predefined by the model coordinator and not derived from any statistics of each client node, Bonawitz teaches that the clipping range must be selected “a priori,” that “it needs to be the same for each user,” and that “the server, or an engineer, chooses an appropriate t before the start of a training round” (Bonawitz, page 2, sec. II.A), while expressly declining to “gather additional signals from user devices to facilitate setting the clipping range” (Bonawitz, page 3, sec. II.A), a range fixed by the server before training, without any client-side signals, is a range predefined by the model coordinator and not derived from statistics of each client node. Second, as to replacing each bin index value with a mean value based on the upper and lower values, Bonawitz divides the range [-t, t] into κ equal-width bins (Bonawitz, page 2, sec. I.B), and Gray teaches that in uniform quantization the reproduction level selected by the “quantizer index” lies “in the center of each cell” (Gray, pages 2326, 2345), and the center of a bin is simply the mean of its upper and lower values, which is precisely how the instant specification defines the claimed mean (“the mean of the upper and lower values of the bin range,” [0036]). Third, as to computing an expected gradient value for each gradient position using shape parameters based on the Bayesian update, Song maintains a Bayesian posterior for each gradient coordinate (Song, pages 3-4), and Yurochkin teaches the server computing, through “Gaussian-Gaussian conjugacy,” the posterior “MAP estimate” of each global parameter “for i = 1, . . . , L” from the parameters of a prior distribution (Yurochkin, page 4, sec. 3.1), which under conjugacy is the posterior mean, i.e., an expected value computed for every position of the model. Accordingly, the combination of Shlezinger, Bonawitz, Gray, Song, and Yurochkin teaches or suggests each limitation of amended independent claims 1, 8, and 15, with a rationale to combine articulated for each reference, a prima facie case of obviousness is established, and claims 1, 8, and 15, as well as dependent claims 4, 11, and 18, which are argued only on the basis of their dependency, stand rejected.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this
Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not
identically disclosed as set forth in section 102, if the differences between the claimed invention and the
prior art are such that the claimed invention as a whole would have been obvious before the effective filing
date of the claimed invention to a person having ordinary skill in the art to which the claimed invention
pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are
summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 4, 8, 11, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over the NPL reference “UVeQFed: Universal Vector Quantization for Federated Learning” by Shlezinger et al. (referred herein as Shlezinger) in view of the NPL reference “Federated Learning with Autotuned Communication-Efficient Secure Aggregation” by Bonawitz et al. (referred herein as Bonawitz) in view of the NPL reference “Quantization” by Gray et al. (referred herein as Gray) in view of the NPL reference “Communication Efficient SGD via Gradient Sampling with Bayes Prior” by Song et al. (referred herein as Song) further in view of the NPL reference “Bayesian Nonparametric Federated Learning of Neural Networks” by Yurochkin et al. (referred herein as Yurochkin).
Regarding claim 1, Shlezinger teaches:
A method for model updating in a federated learning environment, the method comprising: distributing, by a model coordinator, a current model to a plurality of client nodes, ([Shlezinger, Abstract] “FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally” AND [Shlezinger, page 4, sec. II] “time instance t, the server shares its current model, represented by the vector wt ∈ Rm, with the users”, wherein the examiner interprets “FL” to be the same as “federated learning environment”, “server” to be the same as “model coordinator”, and “users” to be the same as “client nodes” because they are both directed to the central entity and the distributed participants of a federated learning system, respectively).
wherein, upon receiving the current model, a client node of the plurality of client nodes trains the current model using local training data to generate a first set of gradient K-quant vectors; ([Shlezinger, page 4, sec. II.A] “The kth user, k ∈ K, uses its set of nk labeled training samples to retrain the model wt” AND [Shlezinger, page 5, sec. II.A] “the kth user needs to communicate a finite-bit quantized representation of its model update”, wherein the examiner interprets “labeled training samples” of the kth user to be the same as “local training data” and “finite-bit quantized representation of its model update” to be the same as “gradient K-quant vectors” because they are both directed to data held locally by a client and to quantized model-update information that the client generates from local training, respectively).
receiving, by the model coordinator and in response to distributing the current model, the first set of gradient K-quant vectors, wherein each gradient K-quant vector of the first set of gradient K-quant vectors is received from the client node, ([Shlezinger, page 4, sec. II] “time instance t, the server shares its current model, represented by the vector wt ∈ Rm, with the users” AND [Shlezinger, page 5, sec II.A] “the kth user needs to communicate a finite-bit quantized representation of its model update”, wherein the examiner interprets “finite-bit quantized representation of its model update” to be the same as “gradient K-quant vectors” because they are both directed to quantized model-update information sent from a client-side participant to the server after the server distributes the current model).
wherein the first set of gradient K-quant vectors is received instead of the local training data to minimize network bandwidth usage; ([Shlezinger, page 2, sec. I] “Instead of requiring the users to share their possibly private labeled data, each user trains the network locally, and conveys its trained model updates to the server.” AND [Shlezinger, page 2, sec. I] “the transfer of a large number of updated model parameters over the uplink communication channel from the users to the server, whose throughput is typically constrained”, wherein the examiner interprets conveying “trained model updates” rather than sharing “private labeled data”, over an uplink “whose throughput is typically constrained”, to be the same as receiving the gradient K-quant vectors instead of the local training data to minimize network bandwidth usage because they are both directed to transmitting compact model-update information in place of raw local data to reduce the communication load).
generating, by the model coordinator, a first updated model based on the first set of gradient K-quant vectors, wherein generating the first updated model comprises: decoding, by the model coordinator, each gradient K-quant vector of the first set of gradient K-quant vectors to obtain a plurality of bin index values, ([Shlezinger, page 5, sec II.A] “Quantization consists of encoding the model update into a set of bits, and decoding each bit combination into a recovered model update” AND [Shlezinger, page 8, sec III.A] “The server first decodes each digital codeword u(k)t+τ into the discrete value {QL(h̄(k)i + z(k)i)}.”, wherein the examiner interprets “decodes each digital codeword” to be the same as “decoding … each gradient K-quant vector” and “discrete value” to be the same as “bin index values” because they are both directed to the server decoding received quantized update information into the quantization values that it represents).
wherein the model coordinator generates the first updated model using the plurality of bin index values, ([Shlezinger, page 5, sec II.A] “The server uses the received codewords [Eq] to reconstruct ĥt+τ ∈ Rm” AND [Shlezinger, page 5, sec. II.A] “Finally, the global model wt+τ is updated via [Eq. 4]”, wherein the examiner interprets “uses the received codewords” and “the global model wt+τ is updated” to be the same as generating the first updated model using the plurality of bin index values because they are both directed to reconstructing a transmitted quantized update at the server and using it to update the model).
distributing the first updated model to the plurality of client nodes; ([Shlezinger, Abstract] “FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally” AND [Shlezinger, page 4, sec II] “time instance t, the server shares its current model, represented by the vector wt ∈ Rm, with the users”, wherein the examiner interprets the server sharing its current model with the users in “each iteration” to be the same as distributing the first updated model to the plurality of client nodes because they are both directed to the server providing the most recently updated global model to the clients for the next round of training).
receiving, by the model coordinator and in response to distributing the first updated model, a second set of gradient K-quant vectors, wherein each gradient K-quant vector of the second set of gradient K-quant vectors is received from a second client node of the plurality of client nodes; ([Shlezinger, page 5, sec. II.A] “the kth user needs to communicate a finite-bit quantized representation of its model update” AND [Shlezinger, Abstract] “FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally”, wherein the examiner interprets each “kth user” communicating a quantized model update in a subsequent “iteration” to be the same as a second client node sending a second set of gradient K-quant vectors because they are both directed to receiving quantized model updates from a plurality of clients over successive training rounds).
distributing the second updated model to the plurality of client nodes; ([Shlezinger, Abstract] “FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally” AND [Shlezinger, page 4, sec. II] “time instance t, the server shares its current model, represented by the vector wt ∈ Rm, with the users”, wherein the examiner interprets the server sharing its current model with the users in “each iteration” to be the same as distributing the second updated model to the plurality of client nodes because they are both directed to the server providing the most recently updated global model to the clients for each subsequent round of training).
Shlezinger does not teach wherein each gradient K-quant vector is a vector of values that represents actual gradient values modified based on a K value, a user-defined minimum value of a range, and a user-defined maximum value of the range that the model coordinator sent to each client node, wherein the range is predefined by the model coordinator and not derived from any statistics of each client node of the plurality of client nodes, wherein each client node normalizes the actual gradient values to fall within the range and to obtain normalized gradient values, wherein each normalized gradient value is associated with a bin index value corresponding to a sub-portion of the range…wherein the plurality of bin index values corresponds to a gradient vector position of each gradient K-quant vector of the first set of gradient K-quant vectors…wherein the decoding further comprises replacing each bin index value of the plurality of bin index values with a mean value of a corresponding sub-portion of the range, wherein the mean value is based on an upper value of the range and a lower value of the range;…storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update,, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, or making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters… wherein the generating further comprises computing an expected gradient value for each gradient position of a plurality of gradient vector positions of the second updated model, wherein the computing comprises using the plurality of shape parameters based on the Bayesian update;.
Bonawitz teaches:
wherein each gradient K-quant vector is a vector of values that represents actual gradient values modified based on a K value, a user-defined minimum value of a range, and a user-defined maximum value of the range that the model coordinator sent to each client node, ([Bonawitz, page 1, sec. I.A] “Each device in the cohort downloads a copy of the current model parameters from the server, then uses the devices local training examples to form a model update, i.e. by taking some number of steps of stochastic gradient descent” AND [Bonawitz, page 2, sec. I.B] “Note that model updates are generally real-valued vectors in federated learning, but SECAGG (and similar cryptographic protocols) require input vector elements to be integers mod k. In practice, this is typically solved by choosing a fixed range of the real numbers, say [-t, t], clipping each user update x(u) onto this range, then uniformly quantizing the remaining values using κ bins” AND [Bonawitz, page 2, sec. II.A] “We also observe that quantizing to a fixed point representation requires selecting the clipping range [- t, t] a priori - it needs to be the same for each user and thus the server, or an engineer, chooses an appropriate t before the start of a training round.”, wherein the examiner interprets a model update formed by “steps of stochastic gradient descent” to be the same as “actual gradient values…κ bins” to be the same as a “K value”, and -t ,t chosen by “an engineer” to be the same as a “user-defined minimum value of a range” and a “user-defined maximum value of the range”, and a range that must be “the same for each user” and is chosen by “the server” to be the same as a range “that the model coordinator sent to each client node” because they are both directed to a common quantization range and bin count, set centrally, that each client applies to its gradient-based model update.)
wherein the range is predefined by the model coordinator and not derived from any statistics of each client node of the plurality of client nodes, ([Bonawitz, page 2, sec. II.A] “We also observe that quantizing to a fixed point representation requires selecting the clipping range [-t, t] a priori - it needs to be the same for each user and thus the server, or an engineer, chooses an appropriate t before the start of a training round” AND [Bonawitz, page 3, sec. II.A] “While we could gather additional signals from user devices to facilitate setting the clipping range appropriate, we would prefer not to do so.”, wherein the examiner interprets the server selecting the clipping range “a priori…before the start of a training round” to be the same as the range being “predefined by the model coordinator”, and declining to “gather additional signals from user devices” to set the clipping range to be the same as the range being “not derived from any statistics of each client node” because they are both directed to a quantization range that is fixed centrally in advance without relying on per-client data statistics).
wherein each client node normalizes the actual gradient values to fall within the range and to obtain normalized gradient values, wherein each normalized gradient value is associated with a bin index value corresponding to a sub-portion of the range, ([Bonawitz, page 2, sec. I.B] “clipping each user update x^(u) onto this range, then uniformly quantizing the remaining values using κ bins, each of width 2t/(κ-1), such that a real value of -t maps to a quantized value of 0 and a real value of +t maps to a quantized value of κ - 1”, wherein the examiner interprets “clipping each user update” onto the range to be the same as normalizing the actual gradient values to fall within the range, and each value being mapped to one of the “κ bins” of width 2t/(κ-1) with an integer “quantized value” from 0 to κ - 1 to be the same as each normalized gradient value being associated with a bin index value corresponding to a sub-portion of the range because they are both directed to bounding each update value to the range and assigning it the index of the equal-width sub-interval in which it falls.)
wherein the plurality of bin index values corresponds to a gradient vector position of each gradient K-quant vector of the first set of gradient K-quant vectors, ([Bonawitz, page 1, sec. I.A] “when the rotated update vector x^(u) is quantized to a single bit per dimension, a mean squared error (MSE)…is achieved” AND [Bonawitz, page 2, sec. I.B] “Note that model updates are generally real-valued vectors in federated learning, but SecAgg (and similar cryptographic protocols) require input vector elements to be integers mod k.”, wherein the examiner interprets quantizing the update vector “per dimension” so that the “input vector elements” are integers to be the same as the plurality of bin index values corresponding to a gradient vector position because they are both directed to each quantized index being tied to a particular element position of the transmitted update vector).
Shlezinger and Bonawitz do not teach wherein the decoding further comprises replacing each bin index value of the plurality of bin index values with a mean value of a corresponding sub-portion of the range, wherein the mean value is based on an upper value of the range and a lower value of the range;…storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, or making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters…wherein the generating further comprises computing an expected gradient value for each gradient position of a plurality of gradient vector positions of the second updated model, wherein the computing comprises using the plurality of shape parameters based on the Bayesian update;.
Gray teaches wherein the decoding further comprises replacing each bin index value of the plurality of bin index values with a mean value of a corresponding sub-portion of the range, wherein the mean value is based on an upper value of the range and a lower value of the range; ([Gray, page 2326] “the quantizer “codes” an input into a binary representation or channel codeword of the quantizer index specifying which reproduction level should be used in the reconstruction.” AND [Gray, page 2345] “In uniform quantization, all cells have the same size and shape and the levels are in the center of each cell (except for the outermost cells which are ignored).” AND [Gray, page 2329] “the reproduction level corresponding to a given cell is the conditional expectation or centroid of the source value given that it lies in the specified cell”, wherein the examiner interprets reconstruction using the “reproduction level” specified by the “quantizer index”, where the levels are “in the center of each cell”, to be the same as replacing each bin index value with a mean value of a corresponding sub-portion of the range, and the “center” of a cell, which lies midway between the upper and lower bounds of that cell, to be the same as a mean value based on an upper value and a lower value because they are both directed to dequantizing an index to the midpoint of the interval it identifies.)
Shlezinger, Bonawitz, and Gray do not teach storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update,, generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, or making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters.
Song teaches:
storing, by the model coordinator, a plurality of shape parameters, wherein each shape parameter of the plurality of shape parameters determines a shape of a curve representing a probability distribution used to estimate expected gradient values during model updating, wherein each shape parameter of the plurality of shape parameters are updated using a Bayesian update, ([Song, page 12068] “In each training iteration, pt,i undergoes L1 normalization after it is updated by Eq. (12). Based on Eq. (12), we adopt Bayes’ theorem to express the posterior sampling probability as: [Eq 14]”, wherein the examiner interprets the posterior sampling probability and its iterative Bayesian updating to be the same as a plurality of shape parameters determining the shape of a probability distribution and being updated using a Bayesian update because they are both directed to using parameters of a probabilistic gradient-distribution model that are updated over time using Bayes-based updating).
generating, by the model coordinator, a second updated model based on the second set of gradient K-quant vectors and the plurality of shape parameters, ([Song, Abstract] “Specifically, we sample important/large gradients based on the global gradient distribution, which is periodically updated across multiple workers.” AND [Song, page 12068] “the posterior sampling probability” is updated, wherein the examiner interprets “global gradient distribution” and “posterior sampling probability” to be the same as using shape-parameter-based probabilistic information during model updating because they are both directed to updating the model using received gradient information together with updated distributional parameters).
making, after the distributing the second model update, a first determination that a training session has ended; making a second determination, based on the first determination, that a cycle threshold has not been reached; and updating, based on the second determination, the plurality of shape parameters, ([Song, Abstract] “Specifically, we sample important/large gradients based on the global gradient distribution, which is periodically updated across multiple workers. Then we introduce Bayes Prior into distribution model to further explore the gradients.” AND [Song, page 12068] “So, we use the historical gradient distribution ▽f(xt0) as the sampling distribution for the subsequent τ training steps…f(xt0) can be obtained by one All-Reduce communication. In the implementation of Gradient Sampling, τ is set as large as 100, so the communication overhead of getting ▽f(xt0) can be limited.”, wherein the examiner interprets use of a defined synchronization period τ and subsequent repeated training steps before gradient-distribution refresh to be the same as determining that a training session has ended, determining that an additional cycle remains, and updating the plurality of shape parameters for the next cycle because they are both directed to updating the probabilistic gradient-distribution parameters after a completed training interval and before a subsequent interval).
Shlezinger, Bonawitz, Gray, and Song do not teach wherein the generating further comprises computing an expected gradient value for each gradient position of a plurality of gradient vector positions of the second updated model, wherein the computing comprises using the plurality of shape parameters based on the Bayesian update;.
Yurochkin teaches wherein the generating further comprises computing an expected gradient value for each gradient position of a plurality of gradient vector positions of the second updated model, wherein the computing comprises using the plurality of shape parameters based on the Bayesian update; ([Yurochkin, page 3, sec. 3.1] “First, draw a collection of global atoms (hidden layer neurons) from a Beta process prior with a base measure H and mass parameter γ0” AND [Yurochkin, page 3, sec. 3.1] “In our experiments we choose H = N(µ0, Σ0) as the base measure” AND [Yurochkin, page 4, sec. 3.1] “Note that the next proposition easily follows from Gaussian-Gaussian conjugacy: Proposition 1. Given {B^j}, the MAP estimate of {θi} is given by [Eq. 4]…for i = 1, . . . , L”, wherein the examiner interprets computing the “MAP estimate” of each global parameter θi “for i = 1, . . . , L” from the prior parameters µ0 and Σ0 by “Gaussian-Gaussian conjugacy” to be the same as computing an expected gradient value for each gradient position of the second updated model using the plurality of shape parameters based on the Bayesian update because they are both directed to computing, for every position of the global model, a Bayesian posterior estimate from the parameters of a probability distribution, where under Gaussian-Gaussian conjugacy the MAP estimate is the posterior mean, i.e., the expected value).
Shlezinger, Bonawitz, Gray, Song, Yurochkin, and the instant application are analogous art because they are all directed to quantizing, communicating, and reconstructing or estimating numerical model values exchanged between distributed participants and a central coordinator during distributed or federated model training.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the federated learning method disclosed by Shlezinger to include the “choosing a fixed range of the real numbers, say [-t, t], clipping each user update x(u) onto this range, then uniformly quantizing the remaining values using κ bins” disclosed by Bonawitz. One would be motivated to do so to efficiently quantize every client update against a common, centrally selected range so that the updates can be communicated with smaller fixed point representations, as suggested by Bonawitz ([Bonawitz, page 3, sec. II.B] “With this precisely tuned quantization, secure aggregation can then operate with significantly smaller fixed point integer representations and achieve improved communication efficiency.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “levels are in the center of each cell” disclosed by Gray. One would be motivated to do so to effectively reconstruct each quantized value with the least expected reconstruction error, as suggested by Gray ([Gray, page 2329] “it is the minimum mean-squared error estimate of the source sample.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention include the “we adopt Bayes’ theorem to express the posterior sampling probability” disclosed by Song. One would be motivated to do so to effectively use probabilistic gradient-distribution parameters that are updated using Bayesian updating during model training so that later model updates can be generated using updated distributional information, as suggested by Song ([Song, page 12068] “we adopt Bayes’ theorem to express the posterior sampling probability.”).
It would have also been obvious to a person of ordinary skill in the art before the effective filing date of the invention to include the “MAP estimate of {θi}” obtained through “Gaussian-Gaussian conjugacy” disclosed by Yurochkin. One would be motivated to do so to efficiently synthesize an accurate global model from the participants’ local models with few communication rounds, as suggested by Yurochkin ([Yurochkin, page 1, Abstract] “synthesize a more expressive global network without additional supervision, data pooling and with as few as a single communication round.”).
Claims 8 and 15 are analogous to claim 1, aside from claim type and minute differences, and thus the same rejection applies as above.
Regarding claim 4, Shlezinger, Bonawitz, Gray, Song, and Yurochkin teaches The method of claim 1 (see rejection of claim 1).
Bonawitz further teaches wherein, before receiving the first set of gradient K-quant vectors, the method further comprises distributing, by the model coordinator and to the plurality of client nodes, the K value and the range, wherein the plurality of client nodes uses the K value and the range to generate the first set of gradient K-quant vectors. ([Bonawitz, page 2, sec. II.A] “it needs to be the same for each user and thus the server, or an engineer, chooses an appropriate t before the start of a training round.” AND [Bonawitz, page 4, sec. II.F] “The server compute the new quantization bin size b∗ = 2t/(k-1), such that the range [-t, t] maps to the full set of quantized output values [0, k - 1]…The next iteration of federated learning repeats this recipe using the new bin size b∗.”, wherein the examiner interprets the server choosing t, and therefore the range [-t, t] and the bin size applied by every user, “before the start of a training round”, with the range required to be “the same for each user” to be the same as distributing, by the model coordinator and to the plurality of client nodes, the K value and the range before receiving the first set of gradient K-quant vectors because they are both directed to the server fixing the quantization parameters that all clients then apply when quantizing their updates.)
Shlezinger, Bonawitz, Gray, Song, Yurochkin, and the instant application are analogous art because they are all directed to quantizing, communicating, and reconstructing or estimating numerical model values exchanged between distributed participants and a central coordinator during distributed or federated model training.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the federated learning method of claim 1 disclosed by Shlezinger, Bonawitz, Gray, Song, and Yurochkin to include the “the server, or an engineer, chooses an appropriate t before the start of a training round” disclosed by Bonawitz. One would be motivated to do so to efficiently ensure that every client quantizes its update with the same range and bin size so that the updates can be communicated with smaller fixed-point representations, as suggested by Bonawitz ([Bonawitz, page 3, sec. II.B] “With this precisely tuned quantization, secure aggregation can then operate with significantly smaller fixed point integer representations and achieve improved communication efficiency.”).
Claims 11 and 18 are analogous to claim 4, aside from claim type and minute differences, and thus the same rejection applies as above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DEVAN KAPOOR whose telephone number is (703)756-1434. The examiner can normally be reached Monday - Friday: 9:00AM - 5:00 PM EST (times may vary).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Yi can be reached at (571) 270-7519. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEVAN KAPOOR/Examiner, Art Unit 2126
/DAVID YI/Supervisory Patent Examiner, Art Unit 2126
