Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 8, 10, 15, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US 20230196205 A1), in view of Wu (US 20250086474 A1) and Subramanian (Practical Quantization in PyTorch).
Regarding claim 1, Lee discloses “A method performed in a distributed (DL) or federated learning (FL) system including a plurality of participating clients and a parameter server, the method comprising:” (See [0024]; Lee discloses a distributed artificial intelligent learning system that uses a plurality of local wireless devices (client) and a parameter server)
“computing, by each participating client during a current training round, a gradient with respect to a local copy of an artificial neural network (ANN);” (See [0100], [0102]; the gradient is computed by each local wireless device (client), and this is done for each iteration of distributed learning)
“transmitting, by each participating client, the compressed gradient to the parameter server;” (See [0010]; the local wireless device (client) transmits the compressed gradient to a parameter server)
“computing, by the parameter server, a compressed global gradient by aggregating the compressed gradients while the compressed gradients remain compressed;” (See [0010], [0068]; the parameter server reconstructs gradients from the received compressed gradient information while the compressed gradients remain compressed)
“transmitting, by the parameter server, the compressed global gradient and the adjusted quantization range to the participating clients (See [0024]; Lee discloses that the parameter server transmits compressed global model parameter information to the local wireless devices (clients), which could include the compressed global model and adjusted quantization range);
“decompressing, by each participating client, the compressed global gradient using the linear quantization technique; and” (See [0024], [0135]; the gradients are reconstructed (decompressing gradients is also referred to as reconstructing) from the model received from the server using a linear quantization process (a non-linear quantization process transformed into a linear quantization process))
“updating, by each participating client, one or more model weights of the local copy of the ANN based on the decompressed global gradient.” (See [0028]; the local model parameters are updated based on the global model from the server)
Lee fails to explicitly disclose, “compressing, by each participating client, the gradient using: (i) a linear quantization technique that is common across the participating clients in the current training round, and (ii) a quantization range that is common across the participating clients in the current training round”.
Wu teaches “compressing, by each participating client, the gradient using: (i) a linear quantization technique that is common across the participating clients in the current training round…” (See [0026], [0065]; a linear quantization is used to compress gradients with a computation device (client)).
Wu fails to explicitly disclose, “compressing… (ii) a quantization range that is common across the participating clients in the current training round”.
Subramanian teaches “compressing… (ii) a quantization range that is common across the participating clients in the current training round” (See [Per-Tensor and Per-Channel Quantization Schemes, Page 7]; Subramanian discloses that the same clipping range (quantization range) is applied to all channels in a layer using a per-tensor quantization scheme).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee, Wu, and Subramanian before them to modify Lee to compress gradients using linear quantization by each client, as well as using the same quantization range across all participating clients. One would be motivated to use the same linear quantization technique across all participating clients to ensure that participating clients will compress a gradient in the same way. One would be motivated to specify a common quantization range to use for all clients to ensure that all clients compress gradients according to a specific range of permissible inputs, see [Quantization Parameters, Page 3], where Subramanian teaches how a clipping range (quantization range) declares the boundaries of permissible inputs for quantization, and [Per-Tensor and Per-Channel Quantization Schemes, Page 7], where Subramanian teaches that per-tensor quantization uses the same compression scheme for each channel in a tensor.
Lee fails to explicitly disclose, “determining, by the parameter server, an adjusted quantization range for a subsequent training round;”.
Subramanian teaches “determining, by the parameter server, an adjusted quantization range for a subsequent training round;” (See [Post-Training Dynamic/Weight-only Quantization, Page 10]; Subramanian discloses that each time there is a new input, a clipping range is calibrated for the input)
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Subramanian before them to modify Lee to adjust the quantization range for a subsequent training round. One would be motivated to adjust the range for a training round to calibrate the range for the new or modified gradients that may be in the new training round, see [Post-Training Dynamic/Weight-only Quantization, Page 10], where Subramanian teaches that a clipping range is calibrated for each time a new input is added for higher accuracy during quantization. One would be inspired to calibrate a clipping range for each new round instead of each time a new input is added to decrease the frequency of adjusting the range while making sure the range is updated periodically.
Regarding claim 3, Lee discloses “prior to compressing the gradient, each participating client pre-processes the gradient by applying a transform,” (See [0057]; Lee discloses that generating compressed gradients comprises transforming the gradients to prepare them for compression)
“subsequent to decompressing the compressed global gradient, each participating client applies an inverse transform to the decompressed global gradient that corresponds to the transform.” (See [0068]; Lee discloses reconstructing (decompressing) gradients using an inverse normalization transformation)
Regarding claims 8 and 15, these claims are similar in scope to claim 1.
Regarding claims 10 and 17, these claims are similar in scope to claim 3.
Claim Rejections - 35 USC § 103
Claims 2, 9, 16 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US 20230196205 A1), in view of Wu (US 20250086474 A1) and Subramanian (Practical Quantization in PyTorch), and further in view of Marzban (US 20240171991 A1).
Regarding claim 2, Lee fails to explicitly disclose, “the parameter server computes the compressed global gradient without decompressing the compressed gradients received from the participating clients.”
Mazban teaches “the parameter server computes the compressed global gradient without decompressing the compressed gradients received from the participating clients” (See [0086]; Marzban discloses receiving gradients or compressed gradients to compute a global gradient by aggregating the gradients and makes no mention of any decompression in the entire publication).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Mazban before them to modify Lee to aggregate the compressed gradients without decompressing them. One would be motivated to do so in order to skip the amount of computational work and runtime needed to decompress the gradients before aggregating them, as aggregation does not require the gradients to be decompressed if it is possible to aggregate them without decompression.
Regarding claims 9 and 16, these claims are similar in scope to claim 2.
Claim Rejections - 35 USC § 103
Claims 4, 11, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US 20230196205 A1), in view of Wu (US 20250086474 A1) and Subramanian (Practical Quantization in PyTorch), and further in view of Theodoridis (Pattern Recognition).
Regarding claim 4, Lee fails to explicitly disclose, “the transform and the inverse transform are super-linear in time complexity.”
Theodoridis teaches “the transform and the inverse transform are super- linear in time complexity” (See [Section 6.10, The Hadamard Transform, page 369]; Theodoridis discloses that the Hadamard transform, one of the transformations listed as an example in the application's specification, has a time complexity of O(N log N), which is a super-linear time complexity (faster than exponential time, but slower than linear time)).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Theodoridis before them to modify Lee to specify what the time complexity of the algorithms used for the transform and inverse transform are. One would be motivated to do so in order to define what kinds of algorithms could be used for the transformations, as only a specific class of algorithms are able to run at a specific time complexity, and competing algorithms would have to either run at a slower or faster time complexity. Other algorithms that run at a similar time complexity could potentially be determined to be too similar to the algorithm used by the applicant.
Regarding claims 11 and 18, these claims are similar in scope to claim 4.
Claim Rejections - 35 USC § 103
Claims 5-7, 12-14, 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US 20230196205 A1), in view of Wu (US 20250086474 A1) and Subramanian (Practical Quantization in PyTorch), and further in view of Alistarh (QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding).
Regarding claim 5, Lee fails to explicitly disclose, “the compressing comprises: compressing the gradient using stochastic quantization.”
Alistarh teaches “the compressing comprises: compressing the gradient using stochastic quantization” (See [Section 3.1]; Alistarh discloses compressing a gradient using a stochastic quantization function and also shows that the number of quantization levels to use as the range is common to each value (client)).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Alistarh before them to modify Lee to compress gradients using stochastic quantization while using a quantization range that all clients would use for the current round. One would be motivated to compress using stochastic quantization to reduce the size of gradients for the purpose of reducing communication bandwidth between the server and clients. Additionally, using a common quantization range ensures that all gradients are compressed with stochastic quantization in a consistent manner.
Regarding claim 6, Lee fails to explicitly disclose, “the quantization range is statically set at the start of the DL or FL procedure”.
Subramanian teaches “the quantization range is statically set at the start of the DL or FL procedure” (See [Post-Training Static Quantization (PTQ), page 12]; Subramanian discloses setting the range by pre-calibrating the range before the operations).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Subramanian before them to modify Lee to set the quantization range to a static range at the start of a DL or FL procedure before running the operations. One would be motivated to do so in order to pre-determine a range so the procedure has a set quantization range to adhere to during runtime.
Regarding claim 7, Lee fails to explicitly disclose, “the adjusted quantization range is dynamically determined for successive training rounds of the DL or FL procedure”.
Subramanian teaches “the adjusted quantization range is dynamically determined for successive training rounds of the DL or FL procedure” (See [Post-Training Dynamic/Weight-only Quantization, page 10]; Subramanian discloses setting the range during inference for each input).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention having Lee and Subramanian before them to modify Lee to change the quantization range during runtime for each round of the procedure. One would be motivated to do so in order to adjust the range during runtime to adapt to the changes in data distribution during the procedure, as this ensures that the model remains accurate as the data evolves.
Regarding claims 12 and 19, these claims are similar in scope to claim 5.
Regarding claims 13 and 20, these claims are similar in scope to claim 6.
Regarding claims 14 and 21, these claims are similar in scope to claim 7.
Response to Arguments
The 35 USC 101 rejections for claims 1-20 have been withdrawn in view of applicant’s amendments.
Applicant's arguments regarding the 35 USC 103 rejection are moot in view of the new grounds of rejection necessitated by applicant's amendments.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID KIM whose telephone number is (571)272-4331. The examiner can normally be reached 7:30 AM - 4:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Ell can be reached at (571) 270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.K./Examiner, Art Unit 2141
/MATTHEW ELL/Supervisory Patent Examiner, Art Unit 2141
Prosecution Insights