Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claims 1-24 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Allowable Subject Matter
Claims 2-8, 10-16 and 18-24 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art of record does not teach or make obvious the dependent claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Applicant claims, “wherein one or more network bandwidth constraints or determine a training batch size for an amount of data…” It is unclear what constraints or determine means. The other independent claims recite that the constraints determine a training batch size – which makes more sense.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over US20200226424A1 to Willcock et al, Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descent by Li et al, and 3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning by Lim et al.
Willcock teaches claims 1, 9 and 17. (Currently Amended) A method comprising:
sampling, by a first computer system, a batch of data instances from a training dataset local to the first computer system; (Willcock para 37 “Each host computer divides the respective batch of training examples received by the host computer into N minibatches…”)
divides the respective batch of training examples received by the host computer into N minibatches (Willcock para 38 “each host computer process the N minibatches using the embedding layer of the neural network (250). Each host computer processes only one minibatch at a time. This process will be described in more detail in reference to FIG. 3.” Willcock para 40 “receive instructions to execute a batch of training examples (310).” Willcock para 41 “ each host computer divides its respective batches of training examples into N minibatches (320).” Willcock para 45 “using the backpropagation data for the batch of training examples, each host computer can compute the gradient of the loss with respect to the current value of each dimension of each embedding that the host computer stores.” Backpropagating error is training.)
wherein one or more or determine a training batch size for an amount of data (Willcock para 8 “because the host computers have to exchange embeddings with each other during each iteration of training…. the number of embeddings that a host computer can obtain at one time is limited, which in turn limits the maximum minibatch size …”) the training batch size being adjustable during iterations and (Willcock para 9 “The system described in this specification dynamically adjusts the number of minibatches before each iteration of training.” Adjusting the number of minibatches for a batch necessarily adjusts the size of the minibatch. The minibatch size is associated with the compression level, because the size of the scratchpad memory is the constraint, and the batch is minibatch’d/compressed to a level that can fit in the smallest scratchpad among the host computers.)
Willcock doesn’t teach compression.
However, Li teaches compressing, by the first computer system, one or more data instances in the batch using a (Li p. 2 “TOC has both good compression ratios on mini-batches and no decompression overheads for matrix operations, which are the main operations executed by MGD on compressed data.” Li p. 6 “The goal of TOC is to (1) compress a mini-batch as much as possible…”)
(Li p. 4 “Figure 2: Optimization efficiencies of BGD, SGD, and MGD for training a neural network… on Mnist. MGD (250 rows) has 250 rows in a mini-batch.” Li p. 3 “We design a suite of compressed execution techniques that operate directly over the compressed data rep resentation without decompression overheads for three classes of matrix operations. These matrix operations are used by MGD to train popular ML models…”)
the training batch size being adjustable during iterations and (Li p. 3 “Note that MGD can cover the spectrum of gradient descent methods by setting the mini-batch size |Bt|…. we focus on MGD with mini-batch size ranging from tens to hundreds of tuples.” Tuples are associated with compression level so the batch size is associated with the tuple/compression level.)
wherein the compressing results in a data size for the compressed batch that is less than or equal to the training batch size. (Li p. 2 “TOC has both good compression ratios on mini-batches…” A compressed mini-batch is a smaller minibatch. Compressed minibatches are a compressed batch, a compressed batch is a smaller batch.)
Li, Willcock and the claims are directed to minibatch training. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to compress the minibatches in Willcock to take advantage of the space-limited faster scratchpad memory in Willcock.
Willcock doesn’t teach transmitting lossy compressed data.
However, Lim teaches lossy compression (Lim sec. 2.2 “Quantization and sparsification techniques make state change transmission generate less network traffic by ap plying lossy compression to the state change data.”) and transmitting, by the first computer system, the compressed the machine learning (ML) model, wherein the second computer system is configured to… (Lim sec. 2 “The workers retrieve from the servers model deltas that record the model changes, and apply the deltas to the local model.”)
wherein one or more network bandwidth constraints … amount of data that may be transmitted from the first computer system to the second computer system… (Lim sec. 2.2 “Gradient pushes and model pulls are sensitive to the available network bandwidth, as these steps need to transmit large data quickly, and state change transmission can take longer as the model size grows and/or the network bandwidth is more constrained…”)
Lim, Willcock and the claims update machine learning models. It would have been obvious to a person having ordinary skill in the art, at the time of filing, to distribute the learning system in Willcock because “[d]istributed ML reduces the total training time by parallelization…” Lim sec. 2.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Austin Hicks whose telephone number is (571)270-3377. The examiner can normally be reached Monday - Thursday 8-4 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AUSTIN HICKS/Primary Examiner, Art Unit 2142
Prosecution Insights