DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 21-24, 29-34, and 36-39 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Lobacheva US Patent Pub. No.:20190347551 A1, hereinafter, ‘Lobacheva’.
Consider claims 21, 31 and 40 (Please see computing Apparatus architecture in at least 0139 to address the hardware features with respect to claims 31 and 41) Lobacheva teaches a computer-implemented method for sparsifying machine-learned models to process data using less memory or fewer processor operations ( Lobacheva describes “ The electronic apparatus and the method thereof include a sparsification technique for the recurrent neural network, obtaining first to third multiplicative variables to learn the recurrent neural network, and performing sparsification for the recurrent neural network to compress the recurrent neural network”) , the computer-implemented method comprising: obtaining, by one or more computing devices, data descriptive of a neural network comprising a plurality of network units corresponding to a plurality of gating parameters that gate outputs of the plurality of network units(e.g., see at least 0015 –“The recurrent neural network being included a gated structure, obtaining third multiplicative variables for pre-activation of gates to make gates and information flow elements of a recurrent layer of the recurrent neural network constant, wherein the obtaining of the mean and the variance may include obtaining a mean and a variance for the weights of the recurrent neural network, the first multiplicative variables, the second multiplicative variables, and the third multiplicative variables” – gated structure – 0016, 0024-0025, 0155-0156, and 0182), wherein one or more network parameters of the one or more network units were trained together with the one or more gating parameters(e.g., see at least 0092 –“In the rest of the specification x=[x0, . . . , xT] is an input sequence, y is a true output and is an output predicted by an RNN (y and may be vectors, sequences of vectors, etc.). X, Y denote a training set {(x1, y1)” ) ;sparsifying, by the one or more computing devices, the neural network based at least in part on the one or more gating parameters to generate a sparsified neural network; and outputting, by the one or more computing devices, the sparsified neural network (e.g., see at least 0156, figure 5 and 0182 – “If the recurrent neural network includes the gated structure, the processor 120 may further include the third multiplicative variables to learn the recurrent neural network and perform the sparsification for the recurrent neural network artificial intelligence model, in a case in which the processor 120 performs the optimization and the sparsification. That is, the processor 120 obtains the first multiplicative variables to the third multiplicative variables, and may then learn the recurrent neural network by using the weights of the recurrent neural network, the first multiplicative variables, the second multiplicative variables, and the third multiplicative variables”).
Consider claims 22 Lobacheva teaches wherein the neural network is configured to generate a network output based on the outputs of the plurality of network units(e.g., see at least 0092 “In the rest of the specification x=[x0, . . . , xT] is an input sequence, y is a true output and is an output predicted by an RNN (y and may be vectors, sequences of vectors, etc.). X, Y denote a training set {(x1, y1), . . . , (xN, yN)}.” ).
Consider claims 23 and 33, Lobacheva teaches wherein the neural network is configured to generate the network output based on a combination of the outputs of the plurality of network units(e.g., see at least 0092 “In the rest of the specification x=[x0, . . . , xT] is an input sequence, y is a true output and is an output predicted by an RNN (y and may be vectors, sequences of vectors, etc.). X, Y denote a training set {(x1, y1), . . . , (xN, yN)}.” ).
Consider claims 24 and 34, Lobacheva teaches wherein the plurality of gating parameters parameterize a plurality of scaling functions that scale the outputs of the plurality of network units (e.g., this is met based on 0127 –“it is proposed to use networks with one LSTM layer of 1000/256 hidden units and fully-connected layer with softmax activation to predict next character or word. The SparseVD and SparseVD-Voc models have been trained for 250/150 epochs on character-level/word-level tasks.”- the softamax activation function provides scaling).
Consider claims 29 Lobacheva teaches wherein outputting the sparsified neural network comprises: deploying the sparsified neural network for inference (e.g., see at least 0092 In the rest of the specification x=[x0, . . . , xT] is an input sequence, y is a true output and is an output predicted by an RNN (y and may be vectors, sequences of vectors, etc.). X, Y denote a training set {(x1, y1). Applicant is reminded/cautioned: a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim.
Consider claims 30 Lobacheva teaches wherein, to generate an output based on an input, the sparsified neural network as compared to the neural network: performs fewer operations; or occupies less memory (e.g., this is anticipated based on the suggestion in 0030- that According to the diverse embodiments of the disclosure as described above, it is possible to accelerate a language modeling work by compressing the recurrent neural network artificial intelligence model using the sparsification technique, and it is possible to perform the language modeling work using the recurrent neural network artificial intelligence model even in the portable apparatus having the limited memory, or the like.)
Consider claims 36 Lobacheva teaches wherein the operations comprise: discarding, from a multiplication matrix used in the forward propagation, one or more multiplicands associated with the network unit (e.g., see at least 0077 - The technique provided herein works as follows: the input-to-hidden and hidden-to-hidden weight matrices are sampled (one per mini-batch), the variational lower bound (10) is optimized w.r.t. {Θ, log σ}, and the posterior is obtained for many weights in the form of the zero-centered δ-function, because the KL-divergence encourages sparsity. These weights can then be safely removed from the model. ).
Consider claims 37 Lobacheva teaches wherein the operations comprise: omitting, from the sparsified neural network, the network unit; and storing, in the one or more non-transitory computer-readable media for inference, the sparsified neural network (e.g., see at least 0084 – “As in (4), if some component of z.sup.x or z.sup.h approaches 0, the corresponding neuron may be removed from the model. But a similar property also exists for the gates: if some component of z.sup.i, z.sup.f, z.sup.o, z.sup.g approaches 0, the corresponding gate or information flow component becomes constant. This means that it is not needed to compute this gate, and the forward pass through the LSTM is accelerated.” ).
Consider claims 38 Lobacheva teaches wherein, to generate an output based on an input, the sparsified neural network as compared to the neural network: performs fewer operations using the one or more processors; or occupies less memory of the computing system (e.g., see at 0008- A conventional recurrent neural network model requires a lot of learning time and large storage space because it uses a large number of parameters. Therefore, a learning of the conventional recurrent neural network model is often performed in an external server capable of having the large storage space and performing high computation, and there is a need to discuss a method for efficiently using a recurrent neural network artificial intelligence model in a portable apparatus having a limited memory such as a smart phone.). 0030 teaches According to the diverse embodiments of the disclosure as described above, it is possible to accelerate a language modeling work by compressing the recurrent neural network artificial intelligence model using the sparsification technique, and it is possible to perform the language modeling work using the recurrent neural network artificial intelligence model even in the portable apparatus having the limited memory, or the like.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 32 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lobacheva US Patent Pub. No.:20190347551 A1, hereinafter, ‘Lobacheva’ in view of Sather et al. US Patent No.: 12,136,039, hereinafter, ‘Sather’.
Consider Claim 32, Lobacheva teaches the claimed invention except wherein the sparsified neural network is sparsified based on a desired amount of computing resources to be used at the inference time.
In analogous art, Sather teaches wherein the sparsified neural network is sparsified based on a desired amount of computing resources to be used at the inference time(i.e., memory reads on computing resource) (e.g., Claim 1 states in part “…the MT network fulfills the sparsity constraint to be executable on a particular type of limited-memory neural network inference circuit having sparsity requirements based on the neural network inference circuit using less memory for parameters set to the value zero than for parameters set to non-zero values.).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date to include wherein the sparsified neural network is sparsified based on a desired amount of computing resources to be used at the inference time for the purpose of optimization.
Allowable Subject Matter
Claims 25-28, 35 and 40 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The prior art of record does not teach wherein sparsifying the neural network comprises: removing, from a forward propagation of the neural network, processing of a network unit based on comparing, to a clipping threshold, a gating value obtained based on a corresponding gating parameter value.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20190138887 teaches a gated recurrent neural networks (RNNs) with reduced parameter gating signals and/or memory cell units are disclosed. In some embodiments, methods for analyzing sequential data are provided, the methods comprising: providing training data to an RNN including a first gate and gating signal; calculating an array of first parameters in a first equation used to calculate values of the first gating signal, including two or fewer parameters corresponding to arrays of values; receiving input data including first data and second data, the second data comes after the first data in a sequence; providing first data to the RNN; calculating a first gating signal; generating a first output; providing second data as input to the RNN; generating a second output; and providing a third output identifying one or more characteristics of the input data based on the first output and the second output.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES TERRELL SHEDRICK whose telephone number is (571)272-8621. The examiner can normally be reached 8A-5P.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew D Anderson can be reached at 571 272 4177. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES T SHEDRICK/Primary Examiner, Art Unit 2646
