DETAILED ACTION
This action is in response to the amendments and remarks filed 06/02/2026. Claims 1-21 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-4, 8-11, 15-17, and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Samek et al. (CONCEPTS FOR DISTRIBUTED LEARNING OF NEURAL NETWORKS AND/OR TRANSMISSION OF PARAMETERIZATION UPDATES THEREFOR, published 3/4/2021, US 2021/0065002 A1), hereafter referred to as Samek, in view of Spratling (A review of predictive coding algorithms, 2017, Brain and Cognition 112 92–97).
Regarding claim 1, Samek teaches [a]n apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to [perform operations]: “A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein” (Samek, [0179]); “Still another embodiment may have a non-transitory digital storage medium (memory) having stored thereon a computer program (instructions) for performing a method of for federated learning of a neural network” (Samek, [0015]).
Samek further discloses a method, comprising:
receiv[ing] a compressed residual local weight update comprising a difference between an intended local weight update for an institute and an institute-based predicted local weight update for the institute from each institute of a plurality of institutes, such that a plurality of compressed residual local weight updates are received from the plurality of institutes: “Clients (the plurality of institutes) that want to contribute to the global training first synchronize with the current global model, by downloading it from a server. They then compute a local weight-update using their own local data and upload it to the server (receive[d] by the server)” (Samek, [0067]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089]). The clients disclosed by Samek are analogous to the institutes disclosed by the claimed invention, as made evident by paragraph [0067] of the instant Specification.
generat[ing] an intended global weight update by aggregating the plurality of local weight updates: “Clients that want to contribute to the global training first synchronize with the current global model, by downloading it from a server (intended global weight update). They then compute a local weight-update using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (intended global weight update)” (Samek, [0067]); “Instead of downloading the full model W at every communication round or cycle, we can instead just download the global weight-update
∆
W
and then apply this weight update locally” (Samek, [0090]). While local updates are aggregated, their cumulative effect is the intended global weight update which is subsequently used to update the model on the server and the clients.
updat[ing] a model on a server based at least on the [intended] global weight update: “At the server all weight-updates are aggregated (intended global weight update) to form a new global model” (Samek, [0067]). The aggregated update of Samek is being mapped to the intended global weight update, not the global weight update calculated by the sum of the predicted and residual weight updates.
wherein the model is configured to be applied to perform at least one task: “The goal in supervised learning is to find parameters W, a setting (configur[ation]) for the parameterization, for which the DNN (model) most closely matches the desired output on the training data
PNG
media_image1.png
62
476
media_image1.png
Greyscale
, i.e. to solve the optimization problem (perform at least one task)” (Samek, [0063]).
transfer[ing] the compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update, with the at least one second parameter to the plurality of institutes: “Clients (the plurality of institutes) that want to contribute to the global training first synchronize with the current global model, by downloading it from a server (intended global weight update)” (Samek, [0090]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089]). Update downloaded from a server in Samek is being mapped to the intended global weight update, not the global weight update comprised of a sum.
wherein the compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update that is transferred is computed in response to receiving the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute, the compressed residual local weight update being the compressed residual local weight update that is received from each institute of the plurality of institutes: “Clients that want to contribute to the global training first synchronize with the current global model (previous global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (current global weight update)” (Samek, [0067])
wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute received from each institute of the plurality of institutes is computed based on a model state updated using another respective compressed residual global weight update, the another respective compressed residual global weight update comprising a difference between another intended global weight update and another predicted global weight update, wherein the another compressed residual global weight update is transferred prior to transferring the compressed residual global weight update: “Clients that want to contribute to the global training first synchronize with the current global model (another global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (global weight update)” (Samek, [0067])
While Samek fails to disclose the further limitations of the claim, Spratling, in combination with Samek, teaches a method, comprising:
receiv[ing] a compressed residual local weight update comprising a difference between an intended local weight update for an institute and an institute-based predicted local weight update for the institute from each institute of a plurality of institutes, such that a plurality of compressed residual local weight updates are received from the plurality of institutes:
(Spratling) “Digital signal processing concerns the manipulation and analysis of a continuous signal, x, sampled at discrete time points (indexed by i) so that the signal is represented as a sequence of numbers, x(i), called a ‘time series’ (plurality of time-series data points)” (Spratling, page 93, left column, paragraph 3);
(Spratling) “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20)
(Spratling) “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2); “it should be noted that if only the residual error is transmitted, then the receiver … cannot recover the components of the signal that have been removed” (Spratling, page 93, right column, paragraph 3).
Examiner’s Note: Local and global weight updates in a supervised learning model are a form of the time-series signal data used by Spratling’s method (each weight update has a different value for each iteration i of training). Thus, this method can be used to calculate a compressed residual local weight update from intended and predicted update values corresponding to an institute.
receiv[ing] a first parameter from each institute such that a plurality of first parameters are received from the plurality of institutes: (Spratling) “The basic idea of linear predictive coding is that each sample of a time series (plurality of [time-series data points]) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (plurality of parameters), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3); “This can be used for signal compression, where only the coefficients (plurality of parameters) and the first n samples need to be stored or transmitted (receiv[ed]) and then the remaining signal is approximated (or synthesised) from these values” (Spratling, page 93, right column, paragraph 1). The coefficients are respective to the corresponding time series data points. This method enables the reception of a plurality of first parameter(s) calculated from a local weight update time series. While Samek does not disclose transmitting multiple parameters for each update, the combination of the predictor coefficients and residuals from Spratling’s method are substituting each direct update transmission.
determin[ing] a predicted local weight update for each institute using the respective first parameter received from a respective institute, to determine a plurality of predicted local weight updates for the plurality of institutes: (Spratling) “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) (a plurality of predicted time-series data points) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (the respective parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3). This method can be used to determine a plurality of predicted local weight updates with a linear combination of the first parameter(s) and previous value(s) in the local weight time series.
determin[ing] a local weight update for each institute-based on a sum of the respective predicted local weight update for an institute and the respective compressed residual local weight update received from the institute, such that a plurality of local weight updates are determined for the plurality of institutes: (Spratling)
PNG
media_image4.png
89
293
media_image4.png
Greyscale
(Spratling, page 93, left column, paragraph 3);
PNG
media_image2.png
77
287
media_image2.png
Greyscale
(Spratling, page 93, right column, paragraph 2). Spratling discloses through the above equations that
x
i
=
e
i
+
r
(
i
)
, or in plain English, the original values x(i) (plurality of time-series data points) are equivalent to the residuals e(i) (plurality of respective compressed residual time-series data points) added to r(i) (plurality of respective predicted time-series data points), the estimate of x(i). Thus, this method can be used to determine a local weight update for each institute-based on the sum of the respective predicted local weight update for an institute and the respective compressed residual local weight update received from the institute.
calculat[ing] a predicted global weight update and at least one second parameter: (Spratling) “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) (a predicted time-series data point) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (at least one parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3). This method can be used to calculate a predicted global weight update with a linear combination of the second parameter(s) and previous value(s) in the global weight time series.
compress[ing] a difference between the intended global weight update and the predicted global weight update to generate a compressed residual global weight update: (Spratling) “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of time-series data point difference[s]), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). As this method is applicable to time-series data, it can be used to calculate a compressed residual global weight update from predicted and intended values.
transfer[ring] the compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update … to the plurality of institutes: (Spratling) “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal, as calculated by Eq. (1), is subtracted from the true value, x(i), to determine the residual error, e(i) (residual time-series data point), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted (transfer[red]) with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). As noted previously, Spratling discloses a method that can be used to calculate a compressed residual global weight update.
transfer[ring] … the at least one second parameter to the plurality of institutes: (Spratling) “The basic idea of linear predictive coding is that each sample of a time series (plurality of [time-series data points]) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (at least one … parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3); “This can be used for signal compression, where only the coefficients (at least one … parameter) and the first n samples need to be stored or transmitted (transfer[red]) and then the remaining signal is approximated (or synthesised) from these values” (Spratling, page 93, right column, paragraph 1). This method enables the transfer of second parameter(s) calculated from a global weight update time series.
wherein the compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update that is transferred is computed in response to receiving the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute, the compressed residual local weight update being the compressed residual local weight update that is received from each institute of the plurality of institutes; wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute received from each institute of the plurality of institutes is computed based on a model state updated using another respective compressed residual global weight update, the another respective compressed residual global weight update comprising a difference between another intended global weight update and another predicted global weight update, wherein the another compressed residual global weight update is transferred prior to transferring the compressed residual global weight update: (Spratling) “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). Samek discloses transmitting global updates to local models, local models producing local updates based on the global updates, sending local updates to the global model, and producing a new global update, all in a cyclic fashion, as discussed above. Spratling discloses that instead of sending a time-series signal (update) directly, residuals can be transmitted and used to reconstruct the original signal (update) by adding a predicted data point to it. Samek and Spratling in combination fully disclose these limitations.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to transfer weight update residuals and associated parameters in lieu of whole updates, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. Additionally, the associated parameters can be compressed by only storing the first n values, of which relatively few can characterize the original signal. See Spratling, page 93, right column, paragraphs 1-2.
Regarding claim 2, the rejection of claim 1 in view of Samek and Spratling is incorporated. Spratling, in combination with Samek, further discloses a method wherein:
the predicted local weight update for an institute for a current iteration is calculated by applying a prediction function having arguments comprising at least one actual local weight update for the institute from a respective at least one prior iteration and the first parameter received from the institute, and the predicted global weight update for a current iteration is calculated by applying a prediction function having arguments comprising at least one actual global weight update from a respective at least one prior iteration and the at least one second parameter: (Spratling) “The basic idea of linear predictive coding (Makhoul, 1975; O’Shaughnessy, 1988; Vaseghi, 2000) is that each sample of a time series can be approximated as a linear combination (prediction function) of preceding samples (at least one actual (local / global) weight update for the institute from a respective at least one prior iteration), such that:
PNG
media_image5.png
34
599
media_image5.png
Greyscale
Or more compactly:
PNG
media_image6.png
81
282
media_image6.png
Greyscale
where r(i) (predicted (local / global) weight update for a current iteration) is the estimate of x(i) and n is a parameter, called the order of the model, that determines how many previous samples are used in the estimation. For the predictor coefficients, y1 . . . yn (the (first / second) parameter received from the institute), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i” (Spratling, page 93, left column, paragraph 1).
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to reconstruct weight update values from transmitted residual updates and associated parameters, instead of sending weight update values directly, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. See Spratling, page 93, right column, paragraph 2.
Regarding claim 3, the rejection of claim 1 in view of Samek and Spratling is incorporated. Spratling, in combination with Samek, further teaches a method, wherein a compressed residual local weight update received from an institute is a compressed difference between an intended weight update at the institute and a predicted local weight update determined at the institute: (Spratling): “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘‘residual”)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data points), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data points), to determine the residual error, e(i) (residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). As discussed regarding claim 1, Samek teaches a method of receiving local weight updates of the plurality of institutes, and Spratling’s method is applicable to time-series data points, including local and global weight updates. Together, they disclose a method of calculating residual local weight updates by subtracting predicted local weight updates from local weight updates.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to calculate residual weight update values from a difference of intended and predicted weight update values, as disclosed by Spratling. The residual, as calculated by Spratling’s method, has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. See Spratling, page 93, right column, paragraph 2.
Regarding claim 4, the rejection of claim 1 in view of Samek and Spratling is incorporated. Samek further teaches a method of distribut[ing] an initial model to each institute of the plurality of institutes: “In each communication round, every client (each institute) performs the following operations: First, it downloads the latest model from the server” (Samek, [0073]). The model downloaded in the first round of communication is an initial model distributed to all clients.
Regarding claim 8, Samek teaches [a]n apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to [perform operations]: “A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein” (Samek, [0179]); “Still another embodiment may have a non-transitory digital storage medium (memory) having stored thereon a computer program (instructions) for performing a method of for federated learning of a neural network” (Samek, [0015]).
Samek further discloses a method, comprising:
transfer the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute for the institute institute to a server or other institute with a first parameter: “Clients (institute[s]) that want to contribute to the global training first synchronize with the current global model, by downloading it from a server. They then compute a local weight-update using their own local data and upload (transfer) it to the server” (Samek, [0067]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089]).
receiv[ing] a compressed residual global weight update comprising a difference between an intended global weight update and a server based predicted global weight update from the server or the other institute with a second parameter: “Clients that want to contribute to the global training first synchronize with the current global model, by downloading (receiv[ing]) it from a server (global weight update)” (Samek, [0067]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089]).
updat[ing] a local model for the institute-based at least on the global weight update: “Clients (institutes with local model[s]) that want to contribute to the global training first synchronize with the current global model, by downloading it from a server (global weight update). They then compute a local weight-update using their own local data and upload it to the server” (Samek, [0067]). While Samek fails to disclose updating a model based on a residual weight update, Spratling teaches that all processes based off a global weight update are also based off the corresponding residual global weight update (See below)
wherein the local model is applied to perform at least one task; and perform[ing] the at least one task by applying the local model: “They (the clients) then compute a local weight-update using their own local data and upload it to the server” (Samek, [0067]). Computing a local weight-update using local data is a task.
wherein the compressed residual global weight update comprising the difference between the intended global weight update and the server-based predicted global weight update that is received from the server or the other institute is computed in response to transferring the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute, the compressed residual local weight update being the compressed residual local weight update that is transferred to the server or other institute: “Clients that want to contribute to the global training first synchronize with the current global model (previous global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (current global weight update)” (Samek, [0067])
wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute transferred to the server or other institute is computed based on a model state updated using another compressed residual global weight update, the another compressed residual global weight update comprising a difference between another intended global weight update and another server-based predicted global weight update received from the server or the other institute, wherein the another compressed residual global weight update is received prior to receiving the compressed residual global weight update: “Clients that want to contribute to the global training first synchronize with the current global model (another global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (global weight update)” (Samek, [0067])
While Samek fails to disclose the further limitations of the claim, Spratling discloses a method, comprising:
compress[ing] a difference between an intended local weight update for the institute and the predicted local weight update for the institute to generate a compressed residual local weight update for the institute: “Digital signal processing concerns the manipulation and analysis of a continuous signal, x, sampled at discrete time points (indexed by i) so that the signal is represented as a sequence of numbers, x(i), called a ‘time series’ (plurality of time-series data points)” (Spratling, page 93, left column, paragraph 3); “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘‘residual”)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data points), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data points), to determine the residual error, e(i), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). This method can be used to generate a compressed residual local weight update by subtracting a predicted local weight update from an intended local weight update.
transfer[ring] the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute for the institute institute to a server or other institute: “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal, as calculated by Eq. (1), is subtracted from the true value, x(i), to determine the residual error, e(i) (residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted (transfer[red]) with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2).
transfer[ring] … a first parameter: “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (a … parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3); “This can be used for signal compression, where only the coefficients (a … parameter) and the first n samples need to be stored or transmitted (transfer[red]) and then the remaining signal is approximated (or synthesised) from these values” (Spratling, page 93, right column, paragraph 1). This method enables the calculation of the first parameter(s) from a local weight update time series.
receiv[ing] a compressed residual global weight update comprising a difference between an intended global weight update and a server based predicted global weight update from the server or the other institute: “Digital signal processing concerns the manipulation and analysis of a continuous signal, x, sampled at discrete time points (indexed by i) so that the signal is represented as a sequence of numbers, x(i), called a ‘time series’ (plurality of time-series data points)” (Spratling, page 93, left column, paragraph 3); “e is used to denote the error (difference) between the reconstruction (predicted data point) and the actual sensory input (intended data point) (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal, as calculated by Eq. (1), is subtracted from the true value, x(i), to determine the residual error, e(i) (residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2); “it should be noted that if only the residual error is transmitted, then the receiver … cannot recover the components of the signal that have been removed” (Spratling, page 93, right column, paragraph 3).
receiv[ing] … a second parameter: “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (a … parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3); “This can be used for signal compression, where only the coefficients (a … parameter) and the first n samples need to be stored or transmitted and then the remaining signal is approximated (or synthesised) from these values” (Spratling, page 93, right column, paragraph 1). By being transmitted, the coefficients are being receiv[ed] by some entity. The coefficients are respective to the corresponding time series data points. This method enables the reception of a plurality of second parameters from a global weight update time series.
wherein the compressed residual global weight update comprising the difference between the intended global weight update and the server-based predicted global weight update that is received from the server or the other institute is computed in response to transferring the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute, the compressed residual local weight update being the compressed residual local weight update that is transferred to the server or other institute; wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the predicted local weight update for the institute transferred to the server or other institute is computed based on a model state updated using another compressed residual global weight update, the another compressed residual global weight update comprising a difference between another intended global weight update and another server-based predicted global weight update received from the server or the other institute, wherein the another compressed residual global weight update is received prior to receiving the compressed residual global weight update: (Spratling) “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). Samek discloses transmitting global updates to local models, local models producing local updates based on the global updates, sending local updates to the global model, and producing a new global update, all in a cyclic fashion, as discussed above. Spratling discloses that instead of sending a time-series signal (update) directly, residuals can be transmitted and used to reconstruct the original signal (update) by adding a predicted data point to it. Samek and Spratling in combination fully disclose these limitations.
determine a predicted global weight update for the institute using the second parameter received from the server or other institute: global weight update: “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) (plurality of respective predicted time-series data points) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (at least one … parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3). This method can be used to determine a predicted global weight update with a linear combination of the second parameter(s) and weight value(s) in the time series.
determin[ing] a global weight update for the institute as a sum of the predicted global weight update and the compressed residual global weight update received from the server or the other institute:
PNG
media_image4.png
89
293
media_image4.png
Greyscale
(Spratling, page 93, left column, paragraph 3);
PNG
media_image2.png
77
287
media_image2.png
Greyscale
(Spratling, page 93, right column, paragraph 2). Spratling discloses through the above equations that
x
i
=
e
i
+
r
(
i
)
, or in plain English, the original values x(i) (plurality of time-series data points) are equivalent to the residuals e(i) (compressed residual time-series data points) added to r(i) (predicted time-series data points), the estimate of x(i). Thus, this method can be used to determine a global weight update for each institute-based on the sum of the respective predicted global weight update for an institute and the respective compressed residual global weight update received from the institute.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to transfer weight update residuals and associated parameters instead of weight updates directly, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. Additionally, the associated parameters can be compressed by only storing the first n values, of which relatively few can characterize the original signal. See Spratling, page 93, right column, paragraphs 1-2.
Regarding claim 9, the rejection of claim 8 in view of Samek and Spratling is incorporated. Samek, in combination with Spratling, further teaches a method, comprising:
train[ing] the local model using local data during a previous iteration to generate the intended local weight update for the institute for a current iteration (Samek):
“usually, in order to accurately train a neural network via the federated learning method, many communication rounds 30 (iteration[s]) (that is, many download and upload steps) are used” (Samek, [0053])
“In each communication round, every client (institute) performs the following operations: First, it downloads the latest model (initial local model) from the server. Second, it computes a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]).
Examiner’s note: During a first iteration, each client trains the latest global model received from the server with its local data. The updates from all local models are then used to update the server’s global model. For the next iteration, this updated global model is downloaded by all clients and used to update the local models. Thus, the local model from a previous iteration is used to generate the intended local weight updates in a subsequent, current iteration.
train the local model following the update to the local model during the current iteration to generate a next intended local weight update for the institute for a next iteration: “In each communication round, every client (institute) performs the following operations: First, it downloads the latest model (update to the local model) from the server. Second, it computes a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]). As noted above, the local model from a previous iteration is used to generate the intended local weight updates in a subsequent, current iteration.
Regarding claim 10, the rejection of claim 8 in view of Samek and Spratling is incorporated. Samek further discloses a method of calculating a global weight update received from the server or the other institute … calculated at the server or the other institute: “In each communication round 30, every client 14 performs the following operations: First, it downloads (receive[s]) the latest model (global weight update) from the server. Second, it computes 34 a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging (calculate[ion] at the server), applies 38' them to the global model to obtain the new parametrization setting (global weight update) and then broadcasts the new global model or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0073])
While Samek alone fails to disclose the further limitations of the claim, Spratling, in combination with Samek, discloses a method, wherein the compressed residual global weight update received from the server or the other institute is a compressed difference between an intended global weight update and a predicted global weight update calculated at the server or other institute: (Spratling): “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘‘residual”)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data points), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data points), to determine the residual error, e(i), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). This method can be used to generate the compressed residual global weight update by subtracting a predicted global weight update from an intended global weight update.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to generate a compressed residual global weight update from the difference between an intended and a predicted update, as disclosed by Spratling. The residual, as calculated by this method, has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. See Spratling, page 93, right column, paragraph 2.
Regarding claim 11, the rejection of claim 8 in view of Samek and Spratling is incorporated. Samek further teaches a method of receiv[ing] an initial model from the server or the other institute as the local model: “In each communication round, every client performs the following operations: First, it downloads (receive[s]) the latest model (initial model) from the server” (Samek, [0072]). The model downloaded in the first round of communication is an initial model distributed to all clients.
Regarding claim 15, Samek teaches [a]n apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: [perform operations]: “A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein” (Samek, [0179]); “Still another embodiment may have a non-transitory digital storage medium (memory) having stored thereon a computer program (instructions) for performing a method of for federated learning of a neural network” (Samek, [0015]).
Samek further discloses a method, comprising:
receiv[ing] a compressed residual local weight update comprising a difference between an intended local weight update for an institute and an institute-based predicted local weight update for the institute from each institute of a plurality of institutes, such that a plurality of compressed residual local weight updates are received from the plurality of institutes: “Clients (the plurality of institutes) that want to contribute to the global training first synchronize with the current global model, by downloading it from a server. They then compute a local weight-update using their own local data and upload it to the server (receive[d] by the server)” (Samek, [0067]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089]). The clients disclosed by Samek are analogous to the institutes disclosed by the claimed invention, as made evident by paragraph [0067] of the instant Specification.
determin[ing] a local weight update for each institute-based on a sum of the respective predicted local weight update received for an institute and the respective compressed residual local weight update received from the institute, such that a plurality of local weight updates are determined for the plurality of institute: “In each communication round 30, every client 14 (institute) performs the following operations: First, it downloads the latest model from the server. Second, it computes 34 a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W” (Samek, [0073]).
determine a model estimate for each institute-based on a previous model from a previous iteration for each institute using the respective local weight update for each institute, such that a plurality of model estimates are determined for the plurality of institutes: “usually, in order to accurately train a neural network via the federated learning method, many communication rounds 30 (iteration[s]) (that is, many download and upload steps) are used” (Samek, [0053]); “In each communication round, every client (institute) performs the following operations: First, it downloads the latest model (previous model) from the server. Second, it computes a local weight-update (model estimate) based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update (model estimate) to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model (current model) or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]). Each local weight update is an estimated global update generated by the local client’s model. Thus, each is a model estimate.
determine an adjusted local weight update for each institute-based on the respective model estimate for an institute and a previous global model from a previous iteration, such that a plurality of adjusted local weight updates are determined for the plurality of institutes: “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates (model estimate[s]) exactly. Instead it is possible to compress (adjust) the weight-updates lossy” (Samek, [0089]). As noted above, the local weight updates are based on a previous global model from a previous iteration. These updates can be compressed to get adjusted local weight updates.
generat[ing] an intended global weight update at a server by aggregating the plurality of adjusted local weight updates: “Clients that want to contribute to the global training first synchronize with the current global model, by downloading it from a server (intended global weight update). They then compute a local weight-update using their own local data and upload it to the server. At the server all weight-updates (local weight update[s]) are aggregated to form a new global model (intended global weight update)” (Samek, [0067]); “Instead of downloading the full model W at every communication round or cycle, we can instead just download the global weight-update
∆
W
and then apply this weight update locally” (Samek, [0090]). While local updates are aggregated, their cumulative effect is the intended global weight update which is then used to update the model on the server and the clients. As noted above, these updates can be compressed to get adjusted local weight updates.
determin[ing] a global model for a current iteration based on the previous global model from the previous iteration and the intended global weight update: “In each communication round, every client performs the following operations: First, it downloads the latest model (global model from the previous iteration / intended global weight update) from the server. Second, it computes a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model (global model for a current iteration) or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]).
determin[ing] an intended global weight update for each institute-based on the global model for the current iteration and a respective previous model from the previous iteration for each institute, such that a plurality of intended global weight updates are determined for the plurality of institutes: “In each communication round, every client (each institute) performs the following operations: First, it downloads the latest model (previous model from the previous iteration) from the server. Second, it computes a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model (global model for the current iteration / intended global weight update) or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]).
updat[ing] a model for each institute-based on the respective previous model from the previous iteration for each institute and the respective global weight update for each institute, such that a plurality of models are updated for the plurality of institutes:
“usually, in order to accurately train a neural network via the federated learning method, many communication rounds 30 (iteration[s]) (that is, many download and upload steps) are used” (Samek, [0053])
“In each communication round, every client (institute) performs the following operations: First, it downloads the latest model (previous model from the previous iteration / global weight update) from the server. Second, it computes a local weight-update based on it's local training data using a fixed amount of iteration of SGD, starting at the global model W. Third, it uploads 36 the local weight-update to the server 12. The server 12 then accumulates 38 the weight updates from all participating clients, usually by weighted averaging, applies 38' them to the global model to obtain the new parametrization setting and then broadcasts the new global model or setting back to all clients at the beginning of the cycle 30 at 32 to ensure that everything remains synchronized” (Samek, [0072]).
Examiner’s note: During a first iteration, each client trains the latest global model received from the server with its local data. The updates from all local models are then used to update the server’s global model. For the next iteration, this updated global model is downloaded by all clients and used to update the local models. Thus, the local and global models from a previous iteration are used to update each institute’s model in a subsequent iteration.
wherein the plurality of models are configured to be applied to perform a respective at least one task: “The goal in supervised learning is to find parameters W, a setting for the parameterization, for which the DNN (model) most closely matches the desired output on the training data
PNG
media_image1.png
62
476
media_image1.png
Greyscale
, i.e. to solve the optimization problem (perform at least one task)” (Samek, [0063]).
transfer[ring] a respective compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update and the respective second parameter to each respective institute such that the plurality of compressed residual global weight updates are transferred to the plurality of institutes: “In each communication round 30, every client 14 (plurality of institutes) performs the following operations: First, it downloads the latest model (global weight update) from the server”(Samek, [0073]); “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy” (Samek, [0089]).
wherein the respective compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update that is transferred to each respective institute is computed in response to receiving the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the respective institute and the respective institute-based predicted local weight update for the respective institute, the compressed residual local weight update being the compressed residual local weight update that is received from each institute of the plurality of institutes: “Clients that want to contribute to the global training first synchronize with the current global model (previous global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (current global weight update)” (Samek, [0067])
wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute received from each institute of the plurality of institutes is computed based on a model state updated using another respective compressed residual global weight update, the another respective compressed residual global weight update comprising a difference between another intended global weight update and another predicted global weight update, wherein the another compressed residual global weight update was transferred prior to transferring the compressed residual global weight update: “Clients that want to contribute to the global training first synchronize with the current global model (another global update), by downloading it from a server. They then compute a local weight-update (local weight update) using their own local data and upload it to the server. At the server all weight-updates (local weight updates) are aggregated to form a new global model (global weight update)” (Samek, [0067])
While Samek fails to disclose the further limitations of the claim, Spratling teaches a method, comprising:
receiv[ing] a compressed residual local weight update comprising a difference between an intended local weight update for an institute and an institute-based predicted local weight update for the institute from each at least one institute of a plurality of institutes, such that a plurality of compressed residual local weight updates are received from the plurality of institutes: “Digital signal processing concerns the manipulation and analysis of a continuous signal, x, sampled at discrete time points (indexed by i) so that the signal is represented as a sequence of numbers, x(i), called a ‘time series’ (plurality of time-series data points)” (Spratling, page 93, left column, paragraph 3); “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2); “it should be noted that if only the residual error is transmitted, then the receiver … cannot recover the components of the signal that have been removed” (Spratling, page 93, right column, paragraph 3).
receiv[ing] a first parameter from each institute such that a plurality of first parameters are received from the plurality of institutes: “The basic idea of linear predictive coding is that each sample of a time series (plurality of [time-series data points]) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (plurality of parameters), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3); “This can be used for signal compression, where only the coefficients (plurality of parameters) and the first n samples need to be stored or transmitted and then the remaining signal is approximated (or synthesised) from these values” (Spratling, page 93, right column, paragraph 1). By being transmitted, the coefficients are being receiv[ed] by some entity. The coefficients are respective to the corresponding time series data points. This method enables the reception of a plurality of first parameter(s) calculated from a local weight update time series.
determin[ing] a predicted local weight update for each institute using the respective first parameter received from a respective institute, to determine a plurality of predicted local weight updates for the plurality of institutes: “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) (plurality of predicted time-series data points) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (respective parameter), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3). This method can be used to determine a plurality of predicted local weight update with a linear combination of the first parameter(s) and previous value(s) of the local weight update.
determin[ing] a local weight update for each institute-based on a sum of the respective predicted local weight update received for an institute and the respective compressed residual local weight update received from the institute, such that a plurality of local weight updates are determined for the plurality of institutes:
PNG
media_image4.png
89
293
media_image4.png
Greyscale
(Spratling, page 93, left column, paragraph 3);
PNG
media_image2.png
77
287
media_image2.png
Greyscale
(Spratling, page 93, right column, paragraph 2). Spratling discloses through the above equations that
x
i
=
e
i
+
r
(
i
)
, or in plain English, the original value x(i) (plurality of time-series data points) are equivalent to the residual e(i) (respective compressed residual time-series data points) added to r(i) (respective predicted time-series data points), the estimate of x(i). This method can be used to determine a local weight update by adding the residual local weight updates to the predicted local weight updates.
calculate a predicted global weight update and a second parameter for each institute, such that a plurality of predicted global weight updates are calculated for the plurality of institutes and a plurality of second parameters calculated for the plurality of institutes: “The basic idea of linear predictive coding is that each sample of a time series (plurality of time-series data points) can be approximated as a linear combination of preceding samples, such that:
PNG
media_image3.png
190
618
media_image3.png
Greyscale
where r(i) (plurality of predicted time-series data points) is the estimate of x(i) … For the predictor coefficients, y1 . . . yn (plurality of parameters), to be appropriate for estimating every sample, Eq. (1) needs to be true for all values of i. The coefficients are therefore determined by minimising the error (the squared difference) between the actual value of the signal and the linearly predicted one, summed over every sample in the time series” (Spratling, page 93, left column, paragraph 3). This method can be used to calculate a predicted global weight update with a linear combination of the second parameter(s) and previous value(s) in the global weight time series.
compress[ing] a difference between the intended global weight update and the predicted global weight update for each institute to generate a compressed residual global weight update for each institute, such that a plurality of compressed residual global weight updates are determined for the plurality of institutes: “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal, as calculated by Eq. (1), is subtracted from the true value, x(i), to determine the residual error, e(i) (at least one … residual time-series data point), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2).
transfer[ring] at least one compressed residual global weight update: “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal, as calculated by Eq. (1), is subtracted from the true value, x(i), to determine the residual error, e(i) (at least one … residual time-series data point), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted (transfer[red]) with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2).
generat[ing] a global weight update for each institute as a sum of the respective predicted global weight update for each institute and the respective compressed residual global weight update for each institute, such that a plurality of global weight updates are generated for the plurality of institutes:
PNG
media_image4.png
89
293
media_image4.png
Greyscale
(Spratling, page 93, left column, paragraph 3);
PNG
media_image2.png
77
287
media_image2.png
Greyscale
(Spratling, page 93, right column, paragraph 2). Spratling discloses through the above equations that
x
i
=
e
i
+
r
(
i
)
, or in plain English, the original values x(i) (time-series data points) are equivalent to the residuals e(i) (compressed residual time-series data points) added to r(i) (predicted time-series data points), the estimate of x(i). This method can be used to generate a global weight update as a sum of the respective predicted global weight update and the respective compressed residual global weight update.
wherein the respective compressed residual global weight update comprising the difference between the intended global weight update and the predicted global weight update that is transferred to each respective institute is computed in response to receiving the compressed residual local weight update, the compressed residual local weight update comprising the difference between the intended local weight update for the respective institute and the respective institute-based predicted local weight update for the respective institute, the compressed residual local weight update being the compressed residual local weight update that is received from each institute of the plurality of institutes; wherein the compressed residual local weight update comprising the difference between the intended local weight update for the institute and the institute-based predicted local weight update for the institute received from each institute of the plurality of institutes is computed based on a model state updated using another respective compressed residual global weight update, the another respective compressed residual global weight update comprising a difference between another intended global weight update and another predicted global weight update, wherein the another compressed residual global weight update was transferred prior to transferring the compressed residual global weight update: (Spratling) “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data point), to determine the residual error, e(i) (plurality of residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). Samek discloses transmitting global updates to local models, local models producing local updates based on the global updates, sending local updates to the global model, and producing a new global update, all in a cyclic fashion, as discussed above. Spratling discloses that instead of sending a time-series signal (update) directly, residuals can be transmitted and used to reconstruct the original signal (update) by adding a predicted data point to it. Samek and Spratling in combination fully disclose these limitations.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek use weight update residuals to calculate weight updates, instead of transferring the weight updates themselves, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. Additionally, the associated parameters can be compressed by only storing the first n values, of which relatively few can characterize the original signal. See Spratling, page 93, right column, paragraphs 1-2.
Regarding claim 16, the rejection of claim 15 in view of Samek and Spratling is incorporated. Samek, in combination with Spratling, discloses an apparatus, wherein the difference between the intended global weight update and the predicted global weight update is compressed using quantization or sparsification:
(Samek) “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089])
(Spratling) “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value (intended time-series data point), x(i), to determine the residual error, e(i) (at least one … residual time-series data point), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2).
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to compress weight update values from transmitted residual updates, instead of compressing weight update values directly, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. See Spratling, page 93, right column, paragraph 2.
Regarding claim 17, the rejection of claim 15 in view of Samek and Spratling is incorporated. Spratling, in combination with Samek, further teaches a method, wherein a compressed residual local weight update received from an institute is a compressed difference between an intended weight update at the institute and a predicted local weight update determined at the institute: (Spratling): “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘‘residual”)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data points), as calculated by Eq. (1), is subtracted from the true value, x(i) (intended time-series data points), to determine the residual error, e(i) (residual time-series data points), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2). As discussed regarding claim 1, Samek teaches a method of receiving local weight updates of the plurality of institutes, and Spratling’s method is applicable to time-series data points, including local and global weight updates. Together, they disclose a method of calculating residual local weight updates by subtracting predicted local weight updates from local weight updates.
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to calculate residual weight update values from a difference of intended and predicted weight update values, as disclosed by Spratling. The residual, as calculated by Spratling’s method, has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used to reconstruct the original time-series signal. See Spratling, page 93, right column, paragraph 2.
Regarding claim 21, the rejection of claim 1 in view of Samek and Spratling is incorporated. Samek, in combination with Spratling, discloses an apparatus, wherein the difference between the intended global weight update and the predicted global weight update is compressed using quantization or sparsification:
(Samek) “Since stochastic gradients are noisy anyway, it is not necessary to transfer the weight-updates exactly. Instead it is possible to compress the weight-updates lossy, without causing significant harm to the convergence speed. Compression, such as quantization or sparsification can interpreted as a special form of noise” (Samek, [0089])
(Spratling) “e is used to denote the error between the reconstruction and the actual sensory input (or the ‘residual’)” (Spratling, page 93, left column, paragraph 20); “the estimated value of the signal (predicted time-series data point), as calculated by Eq. (1), is subtracted from the true value (intended time-series data point), x(i), to determine the residual error, e(i) (at least one … residual time-series data point), for transmission:
PNG
media_image2.png
77
287
media_image2.png
Greyscale
This residual has a smaller dynamic range than the original signal, and hence, can be transmitted with greater accuracy using the same bandwidth (compressed)” (Spratling, page 93, right column, paragraph 2).
Samek and Spratling relate to data compression and are analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Samek to compress weight update values from transmitted residual updates, instead of compressing weight update values directly, as disclosed by Spratling. The residual has a smaller dynamic range than its original counterpart, allowing more accurate transfer with the same bandwidth required, and can be used
Claim(s) 5, 12, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Samek et al. (CONCEPTS FOR DISTRIBUTED LEARNING OF NEURAL NETWORKS AND/OR TRANSMISSION OF PARAMETERIZATION UPDATES THEREFOR, published 3/4/2021, US 2021/0065002 A1), hereafter referred to as Samek, in view of Spratling (A review of predictive coding algorithms, 2017, Brain and Cognition 112 92–97), and further in view of Ericson et al. (MODULO-PCM: A NEW SOURCE CODING SCHEME, 1979, The University of Linköping, Linkoping, Sweden), hereafter referred to as Ericson.
Regarding claim 5, the rejection of claim 1 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Ericson, in combination with Samek and Spratling, teaches an apparatus, wherein:
the compressed residual global weight update is determined with a modulo operation that returns a remainder of a term divided with a quantization level, the term is the predicted global weight update subtracted from the intended global weight update added to the quantization level, the predicted global weight update is a first discrete value determined with the quantization level, and the intended global weight update is a second discrete value determined with the quantization level: (Ericson)
“let A be the finite alphabet A = {0, 1, … , M-1}. A quantizer is a mapping q: R [Wingdings font/0xE0] A; x [Wingdings font/0xE0] q(x). M is called the number of quantization levels” (Ericson, page 1, left column, paragraph 4). In quantization, a real value is mapped to a new value space, which may be entirely discrete (as shown above). Quantization can be considered a form of compression when the alphabet is smaller than the input space.
“the quantizer q corresponding to {Bi} can be implemented as a uniform quantizer q' , where
PNG
media_image7.png
56
395
media_image7.png
Greyscale
followed by a reduction modulo M, i.e.
PNG
media_image8.png
36
271
media_image8.png
Greyscale
” (Ericson, page 2, left column, paragraph 5). Ericson discloses a method of further quantizing a quantized variable q’(x) by taking the modulo (remainder of division) of it with respect to the quantization level of q’. This method is applicable to discrete quantized values, including a difference of a discrete quantized intended global weight update and a discrete quantized predicted global weight update.
While Ericson does not disclose adding the quantization level to the quantized variable q’(x) before taking the modulo, the two formulas are equivalent:
q
'
x
m
o
d
M
≡
q
'
x
+
M
m
o
d
M
. Any integer multiple of M added to q’(x) will result in the same remainder as q’(x) alone when divided by M, since the multiples of M will be divided out. Thus, Ericson’s formula is applicable to a difference of a discrete quantized intended global weight update and a discrete quantized predicted global weight update, added to the quantization level.
Ericson relates to predictive compression and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to further compress quantized residuals using the modulo operation (Modulo-PCM), as disclosed by Ericson. This operation can reduce the number of quantization regions in the quantized variable, reducing its bitrate, while still maintaining nearly the same performance. See Ericson, page 2, right column, paragraph 1.
Regarding claim 12, the rejection of claim 8 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Ericson, in combination with Samek and Spratling, teaches an apparatus, wherein:
the compressed residual local weight update is determined with a modulo operation that returns a remainder of a term divided with a quantization level; wherein the term is the predicted local weight update subtracted from the intended local weight update added to the quantization level; wherein the predicted local weight update is a first discrete value determined with the quantization level, and the intended local weight update is a second discrete value determined with the quantization level:
“let A be the finite alphabet A = {0, 1, … , M-1}. A quantizer is a mapping q: R [Wingdings font/0xE0] A; x [Wingdings font/0xE0] q(x). M is called the number of quantization levels” (Ericson, page 1, left column, paragraph 4). In quantization, a real value is mapped to a new value space, which may be entirely discrete (as shown above). Quantization can be considered a form of compression when the alphabet is smaller than the input space.
“the quantizer q corresponding to {Bi} can be implemented as a uniform quantizer q' , where
PNG
media_image7.png
56
395
media_image7.png
Greyscale
followed by a reduction modulo M, i.e.
PNG
media_image8.png
36
271
media_image8.png
Greyscale
” (Ericson, page 2, left column, paragraph 5). Ericson discloses a method of further quantizing a quantized variable q’(x) by taking the modulo (remainder of division) of it with respect to the quantization level of q’. This method is applicable to discrete quantized values, including a difference of a discrete quantized intended local weight update and a discrete quantized predicted local weight update.
While Ericson does not disclose adding the quantization level to the quantized variable q’(x) before taking the modulo, the two formulas are equivalent:
q
'
x
m
o
d
M
≡
q
'
x
+
M
m
o
d
M
. Any integer multiple of M added to q’(x) will result in the same remainder as q’(x) alone when divided by M, since the multiples of M will be divided out. Thus, Ericson’s formula is applicable to a difference of a discrete quantized intended local weight update and a discrete quantized predicted local weight update, added to the quantization level.
Ericson relates to predictive compression and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to further compress quantized residuals using the modulo operation (Modulo-PCM), as disclosed by Ericson. This operation can reduce the number of quantization regions in the quantized variable, reducing its bitrate, while still maintaining nearly the same performance. See Ericson, page 2, right column, paragraph 1.
Regarding claim 18, the rejection of claim 15 in vie of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Ericson, in combination with Samek and Spratling, teaches an apparatus, wherein:
the compressed residual global weight update for an institute is determined with a modulo operation that returns a remainder of a term divided with a quantization level, the term is the predicted global weight update for the institute subtracted from the intended global weight update for the institute added to the quantization level, and the predicted global weight update for the institute is a first discrete value determined with the quantization level, and the intended global weight update for the institute is a second discrete value determined with the quantization level: (Ericson)
“let A be the finite alphabet A = {0, 1, … , M-1}. A quantizer is a mapping q: R [Wingdings font/0xE0] A; x [Wingdings font/0xE0] q(x). M is called the number of quantization levels” (Ericson, page 1, left column, paragraph 4). In quantization, a real value is mapped to a new value space, which may be entirely discrete (as shown above). Quantization can be considered a form of compression when the alphabet is smaller than the input space.
“the quantizer q corresponding to {Bi} can be implemented as a uniform quantizer q' , where
PNG
media_image7.png
56
395
media_image7.png
Greyscale
followed by a reduction modulo M, i.e.
PNG
media_image8.png
36
271
media_image8.png
Greyscale
” (Ericson, page 2, left column, paragraph 5). Ericson discloses a method of further quantizing a quantized variable q’(x) by taking the modulo (remainder of division) of it with respect to the quantization level of q’. This method is applicable to discrete quantized values, including a difference of a discrete quantized intended global weight update and a discrete quantized predicted global weight update.
While Ericson does not disclose adding the quantization level to the quantized variable q’(x) before taking the modulo, the two formulas are equivalent:
q
'
x
m
o
d
M
≡
q
'
x
+
M
m
o
d
M
. Any integer multiple of M added to q’(x) will result in the same remainder as q’(x) alone when divided by M, since the multiples of M will be divided out. Thus, Ericson’s formula is applicable to a difference of a discrete quantized intended global weight update and a discrete quantized predicted global weight update, added to the quantization level.
Ericson relates to predictive compression and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to further compress quantized residuals using the modulo operation (Modulo-PCM), as disclosed by Ericson. This operation can reduce the number of quantization regions in the quantized variable, reducing its bitrate, while still maintaining nearly the same performance. See Ericson, page 2, right column, paragraph 1.
Claim(s) 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Samek et al. (CONCEPTS FOR DISTRIBUTED LEARNING OF NEURAL NETWORKS AND/OR TRANSMISSION OF PARAMETERIZATION UPDATES THEREFOR, published 3/4/2021, US 2021/0065002 A1), hereafter referred to as Samek, in view of Spratling (A review of predictive coding algorithms, 2017, Brain and Cognition 112 92–97), and further in view of Atallah (Algorithms and Theory of Computation Handbook, Volume 2: Special Topics and Techniques, 2nd edition, 2009, Chapter 14, Milton: Chapman and Hall/CRC).
Regarding claim 6, the rejection of claim 1 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Atallah, in combination with Samek and Spratling, discloses a method, comprising:
partition[ing] predicted global weight update into two or more parts: (Atallah) “The following are three techniques for splitting a value x (the at least one input variable) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to partition each of the at least one predicted global weight update.
partition[ing] the at least one second parameter into two or more parts respectively corresponding to the two or more parts of the at least one predicted global weight update: (Atallah) “The following are three techniques for splitting a value x (the at least one … parameter) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to the at least one second parameter, which, as discussed regarding claim 1, corresponds to the predicted global weight update.
generat[ing] a first part of a global weight update based on a first part of the predicted global weight update; wherein the model on the server is updated based on the first part of the global weight update: (Atallah) “The input servers split their inputs (using the techniques described in the previous section) (parts of an input) among the computation servers so that no small group of computation servers can recover the inputs. The computation servers then engage in a secure protocol to compute the results (parts of an output) in a split fashion. And finally, the split results are sent to the output servers who then learn the results. This representative-based architecture is depicted in Figure 14.2” (Atallah, page 14-6, paragraph 6);
PNG
media_image9.png
800
785
media_image9.png
Greyscale
(Atallah, page 7, figure 14.2). As discussed regarding claim 1, the combination of Samek and Spratling teaches a method of generating a global weight update based on a predicted global weight update, and using the global weight update to update the model on the server. Atallah teaches how this operation can be split up and distributed across multiple parts, including a first part.
Atallah relates to encoding data in distributed systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to distribute global weight update calculation across multiple parts, as disclosed by Atallah. Doing so would allow multiple parties to contribute to the same calculation, without the data being fully revealed to any given one, helping to ensure data security. See Atallah, page 5, paragraph 5.
Regarding claim 13, the rejection of claim 8 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Atallah, in combination with Samek and Spratling, discloses a method, comprising:
partition[ing] the predicted global weight update into two or more parts: (Atallah) “The following are three techniques for splitting a value x (the at least one input variable) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to partition each of the at least one predicted global weight update.
partition[ing] the second parameter into two or more parts respectively corresponding to the two or more parts of the predicted global weight update: (Atallah) “The following are three techniques for splitting a value x (the … parameter) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to the at least one second parameter, which, as discussed regarding claim 1, corresponds to the predicted global weight update.
generat[ing] a first part of the global weight update based on a first part of the predicted global weight update; wherein the local model for the institute is updated based on the first part of the predicted global weight update: (Atallah) “The input servers split their inputs (using the techniques described in the previous section) (parts of an inputs) among the computation servers so that no small group of computation servers can recover the inputs. The computation servers then engage in a secure protocol to compute the results (parts of an output) in a split fashion. And finally, the split results are sent to the output servers who then learn the results. This representative-based architecture is depicted in Figure 14.2” (Atallah, page 14-6, paragraph 6);
PNG
media_image9.png
800
785
media_image9.png
Greyscale
(Atallah, page 7, figure 14.2). As discussed regarding claim 8, the combination of Samek and Spratling teaches a method of generating a local weight update from a global weight update, based on a predicted global weight update, and using the global weight update to update the model on the server. Atallah teaches how this operation can be split up and distributed across multiple parts, including a first part..
Atallah relates to encoding data in distributed systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to distribute global weight update calculation across multiple parts, as disclosed by Atallah. Doing so would allow multiple parties to contribute to the same calculation, without the data being fully revealed to any given one, helping to ensure data security. See Atallah, page 5, paragraph 5.
Regarding claim 19, the rejection of claim 15 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Atallah, in combination with Samek and Spratling discloses a method, comprising:
partition[ing] the predicted global weight update for an institute into two or more parts: (Atallah) “The following are three techniques for splitting a value x (the at least one input variable) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to partition each of the at least one predicted global weight update.
partition[ing] the second parameter for the institute into two or more parts respectively corresponding to the two or more parts of the predicted global weight update for the institute: (Atallah) “The following are three techniques for splitting a value x (the at least one … parameter) among n participants. Sharing: In sharing approaches all n participants are required to recover the value. One example, of this approach is that party i has a value xi such that x = x1 ⊕ x2 ⊕ · · · ⊕ xn (partition[ing] … into two or more parts) where ⊕ is XOR. To split x in such a manner, n − 1 random values are chosen for x1, . . . , xn−1 and xn is set to x ⊕ x1 ⊕· · ·⊕xn−1” (Atallah, page 14-6, paragraph 1). This technique can be applied to the at least one second parameter, which, as discussed regarding claim 1, corresponds to the predicted global weight update.
generat[ing] a first part of the global weight update for the institute-based on a first part of the global weight update for the institute; wherein the model on the server is updated based on the first part of the global weight update for the institute: (Atallah) “The input servers split their inputs (using the techniques described in the previous section) (parts of an input) among the computation servers so that no small group of computation servers can recover the inputs. The computation servers then engage in a secure protocol to compute the results (parts of an output) in a split fashion. And finally, the split results are sent to the output servers who then learn the results. This representative-based architecture is depicted in Figure 14.2” (Atallah, page 14-6, paragraph 6);
PNG
media_image9.png
800
785
media_image9.png
Greyscale
(Atallah, page 7, figure 14.2). As discussed regarding claim 15, the combination of Samek and Spratling teaches a method of generating a global weight update based on a predicted global weight update, and using the global weight update to update the model on the server. Atallah teaches how this operation can be split up and distributed across multiple parts, including a first part.
Atallah relates to encoding data in distributed systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to distribute global weight update calculation across multiple parts, as disclosed by Atallah. Doing so would allow multiple parties to contribute to the same calculation, without the data being fully revealed to any given one, helping to ensure data security. See Atallah, page 5, paragraph 5.
Claim(s) 7, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Samek et al. (CONCEPTS FOR DISTRIBUTED LEARNING OF NEURAL NETWORKS AND/OR TRANSMISSION OF PARAMETERIZATION UPDATES THEREFOR, published 3/4/2021, US 2021/0065002 A1), hereafter referred to as Samek, in view of Spratling (A review of predictive coding algorithms, 2017, Brain and Cognition 112 92–97), and further in view of Zhang et al. (SAFELearning: Enable Backdoor Detectability In Federated Learning With Secure Aggregation, February 2021, arXiv:2102.02402v1), hereafter referred to as Zhang.
Regarding claim 7, the rejection of claim 1 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Zhang teaches a method, comprising:
receiv[ing] a random seed from an institute of the plurality of institutes: “federated learning allows participants (i.e., users) (plurality of institutes) to locally train models with their private data sets and only transmit the trained model parameters (or gradients) to the remote server” (Zhang, page 1, left column, paragraph 1); “each user u ([member of] the plurality of institutes) generates another random seed
b
u
… Random shares of
b
u
are also generated and sent to other users (received by the other institutes)” (Zhang, page 4, left column, paragraph 2). The users disclosed by Zhang are analogous to the institutes disclosed by the claimed invention, as made evident by paragraph [0067] of the instant Specification.
determin[ing] the predicted local weight update for the institute using the random seed or determin[ing] the predicted global weight update using the random seed: “The server holds a global model
X
i
of size m and each user
u
∈
U
possesses a private training data set. Users train the global model shared by the server with their private training data at each iteration and upload the local model parameters (local weight update[s]) to the server. The server aggregates local parameters and compute
∑
u
∈
U
x
u
, where
x
u
(also of size m) is the local model parameter trained by u using
X
i
and his local data. The server returns the latest global model to each user (global weight update) at the end of each iteration” (Zhang, page 2, right column, paragraph 3); “Each user u obfuscates the parameter vector
x
u
(local weight update) using a mask
P
R
G
(
b
u
)
(takes random seed as input) in addition to the pairwise mask vector:
PNG
media_image10.png
123
394
media_image10.png
Greyscale
” (Zhang, page 4, left column, paragraph 2). To execute this process on a predicted update instead of the regular update disclosed by Zhang, one need only perform the same operations with predicted values instead.
Zhang relates to data transmission in federated learning systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to generate a seed value that’s dispersed to other institutes and added to the update, as disclosed by Zhang. Doing so would ensure that the generating institute’s secret won’t be revealed if its output is delayed, helping to protect the confidentiality of parameters in the system while accounting for dropouts. See Zhang, page 4, left column, paragraphs 2-4.
Regarding claim 14, the rejection of claim 1 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Zhang teaches a method, comprising:
receiv[ing] a random seed from the server or other institute: “federated learning allows participants (i.e., users) (institute[s]) to locally train models with their private data sets and only transmit the trained model parameters (or gradients) to the remote server” (Zhang, page 1, left column, paragraph 1); “each user u (institute) generates another random seed
b
u
… Random shares of
b
u
are also generated and sent to other users (received by the other institute[s])” (Zhang, page 4, left column, paragraph 2). The users disclosed by Zhang are analogous to the institutes disclosed by the claimed invention, as made evident by paragraph [0067] of the instant Specification.
determin[ing] the predicted global weight update using the random seed, or determin[ing] the predicted local weight update using the random seed: “The server holds a global model
X
i
of size m and each user
u
∈
U
possesses a private training data set. Users train the global model shared by the server with their private training data at each iteration and upload the local model parameters (local weight update[s]) to the server. The server aggregates local parameters and compute
∑
u
∈
U
x
u
, where
x
u
(also of size m) is the local model parameter trained by u using
X
i
and his local data. The server returns the latest global model to each user (global weight update) at the end of each iteration” (Zhang, page 2, right column, paragraph 3); “Each user u obfuscates the parameter vector
x
u
(local weight update) using a mask
P
R
G
(
b
u
)
(takes random seed as input) in addition to the pairwise mask vector:
PNG
media_image10.png
123
394
media_image10.png
Greyscale
” (Zhang, page 4, left column, paragraph 2). To execute this process on a predicted update instead of the regular update disclosed by Zhang, one need only perform the same operations with predicted values instead.
Zhang relates to data transmission in federated learning systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to generate a seed value that’s dispersed to other institutes and added to the update, as disclosed by Zhang. Doing so would ensure that the generating institute’s secret won’t be revealed if its output is delayed, helping to protect the confidentiality of parameters in the system while accounting for dropouts. See Zhang, page 4, left column, paragraphs 2-4.
Regarding claim 20, the rejection of claim 15 in view of Samek and Spratling is incorporated. While Samek and Spratling fail to disclose the further limitations of the claim, Zhang teaches a method, comprising:
receiv[ing] a random seed from an institute of the plurality of institutes: “federated learning allows participants (i.e., users) (plurality of institutes) to locally train models with their private data sets and only transmit the trained model parameters (or gradients) to the remote server” (Zhang, page 1, left column, paragraph 1); “each user u ([member of] the plurality of institutes) generates another random seed
b
u
… Random shares of
b
u
are also generated and sent to other users (received by the other institutes)” (Zhang, page 4, left column, paragraph 2). The users disclosed by Zhang are analogous to the institutes disclosed by the claimed invention, as made evident by paragraph [0067] of the instant Specification.
determin[ing] the predicted local weight update for the institute using the random seed, or determin[ing] the predicted global weight update for the institute using the random seed: “The server holds a global model
X
i
of size m and each user
u
∈
U
possesses a private training data set. Users train the global model shared by the server with their private training data at each iteration and upload the local model parameters (local weight update[s]) to the server. The server aggregates local parameters and compute
∑
u
∈
U
x
u
, where
x
u
(also of size m) is the local model parameter trained by u using
X
i
and his local data. The server returns the latest global model to each user (global weight update) at the end of each iteration” (Zhang, page 2, right column, paragraph 3); “Each user u obfuscates the parameter vector
x
u
(local weight update) using a mask
P
R
G
(
b
u
)
(takes random seed as input) in addition to the pairwise mask vector:
PNG
media_image10.png
123
394
media_image10.png
Greyscale
” (Zhang, page 4, left column, paragraph 2). To execute this process on a predicted update instead of the regular update disclosed by Zhang, one need only perform the same operations with predicted values instead.
Zhang relates to data transmission in federated learning systems and is analogous to the claimed invention. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Samek and Spratling to generate a seed value that’s dispersed to other institutes and added to the update, as disclosed by Zhang. Doing so would ensure that the generating institute’s secret won’t be revealed if its output is delayed, helping to protect the confidentiality of parameters in the system while accounting for dropouts. See Zhang, page 4, left column, paragraphs 2-4.
Response to Arguments
The following responses address arguments and remarks made in the instant remarks dated 06/02/2026
112 Rejections
Previous rejections under 35 U.S.C. 112(a) have been withdrawn in light of the instant amendments.
103 Rejections / Allowable Subject Matter
In light of the instant amendments, claims 1-21 are found to be obvious over the prior art, and are rejected under 35 U.S.C. 103.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Liu et al. (A Double Residual Compression Algorithm for Efficient Distributed Learning, published 2019, arXiv:1910.07561v1) discloses a method of transmitting local and global model updates in a distributed learning system via residuals
Thapa et al. (SplitFed: When Federated Learning Meets Split Learning, 2020, arXiv:2004.12088v2) teaches a federated learning system where local weight updates are calculated across multiple machines
Chen et al. (A new lossy compression algorithm for wireless sensor networks using Bayesian predictive coding, 2020, Wireless Netw 26, 5981–5995 (2020). https://doi.org/10.1007/s11276-020-02425-w) teaches a method of using predictive coding to encode data being transmitted over a network
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Aaron P Gormley whose telephone number is (571)272-1372. The examiner can normally be reached Monday - Friday 12:00 PM - 8:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T Bechtold can be reached at (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AG/Examiner, Art Unit 2148 /MICHELLE T BECHTOLD/Supervisory Patent Examiner, Art Unit 2148
Prosecution Insights