DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to under 37 CFR 1.83(a). The drawings must show every feature of the invention specified in the claims.
Therefore, the:
“weight multiplexer array” configured to output one or more weight values stored in the weight buffer as in claim 1, must be shown;
“weight multiplexer array” comprises a first weight multiplexer configured to select a weight register” as in claim 2, and similarly recited in claim 8 must be shown;
“second weight multiplexer’ configured to select a first weight register as in claim 14, must be shown;
“third weight multiplexer” configured to select a second weight register as in claim 15, must be shown;
“fourth weight multiplexer” configured to select a second weight register as in claim 15 must be shown;
or the features canceled from the claims.
No new matter should be entered.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-22 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites a “weight multiplexer array configured to output one or more weight values stored in the weight buffer as first operand values based on the selected fine-grained structured sparsity weight arrangement”. Claim 2 recites the “weight multiplexer comprises a first weight multiplexer configured to select a weight register based on the selected fine-grain structured sparsity weight arrangement and tot output the weight value stored in the selected weight register as a first operand value”. Claim 8 recites a “first weight multiplexer configured to select a first weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected first weight register as a first operand value”. Claim 14 recites a “second weight multiplexer configured to select a first weight register based on the selected fine-train structure sparsity weight arrangement and output the weight value stored in the selected first weight register as a third operand value”. Claim 15 recites a “third weight multiplexer configured to select a second weight register based on the selected fine-grain structure sparsity weight arrangement and output the weight values stored in the selected second weight register as a fifth operand value” and a “fourth weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected second weight register as a seventh operand value”. However, the specification provides no written description, and no drawings as to a weight multiplexer array, or the various weight multiplexers claimed with description of the functioning of the weight multiplexer beyond merely repeating the claimed limitations. Each dependent claim inherits the same deficiency as the claim upon which it depends.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 14-22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 14 line 2 recites “a second weight multiplexer configured to select a first weight register”. It is unclear if the “first weight register” is the same first weight register recited in claim 8 or a different first weight register. For purposes of examination, Examiner interprets as a different first weight register. Claims 15-22 inherit the same deficiency as claim 14 based on dependence.
Claim 15 line 19 recites “a fourth weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement”. It is unclear whether the second weight register is the same second weight register as recited in line 4 or different. For purposes of examination, Examiner interprets as a different second weight register. Claims 16-20 inherit the same deficiency as claim 15 based on dependence.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over J.H. Shin et al., Design Space Exploration of Sparse Accelerators for Deep Neural Networks, arXiv:2107.12922v1 [cs.AR] 27 Jul 2021 (hereinafter “Shin”) in view of US 20230062503 et al., (hereinafter “Wu”).
Regarding claim 1, Shin teaches the following:
a weight buffer configured to store weight values in a fine-grain structured sparsity weight arrangement (Section II.A. first paragraph B matrix represents weights, section III, Load Balancing section fine-grain approach, section IV.A., fig 3 BBUF for weight buffer, section IV.A.2)-3) for structured sparsity weight arrangement);
a weight multiplexer array configured to output one or more weight values stored in the weight buffer as first operand values based on the selected fine-grain structured sparsity weight arrangement (fig 3. BMUX);
an activation buffer configured to store activation values (Section II.A, first and second paragraph, A matrix represents input activation, fig 3. ABUF);
an activation multiplexer array comprising inputs to the activation multiplexer array coupled to the activation buffer, the activation multiplexer array configured to output one or more activation values stored in the activation buffer as second operand values, each respective second operand value and a corresponding first operand value forming an operand value pair (Fig 3. AMUX, with input select from A operand mask); and
a multiplier array configured to output a product value for each operand value pair (fig 3 PE with multiplier array).
Shin discloses a fine-grain structured sparsity weight arrangement including various numbers of ones in the zero mask (fig 3 control unit), but does not explicitly disclose wherein the fine-grain structure sparsity weight arrangement is selected from a group of fine-grain structured sparsity weight arrangements comprising at least two arrangements of a 1:4 fine-grain structured sparsity weight arrangement, a 2:4 fine-grain structured sparsity weight arrangement, a 4:8 fine-grain structured sparsity weight arrangement, and a 2:8 fine-grain structured sparsity weight arrangement.
However, in the same field of endeavor, Wu discloses an apparatus similar to Shin for hierarchical structured sparse parameter pruning to select activations and weights for a multiplication and accumulation operation (abstract, fig 1A). Wu further discloses wherein the structured sparsity pattern includes 1:4, 2:4, 4:8, and 2:8 ([0049-0050], [0052]. fig 1C, fig 1D). It would have been obvious to one of ordinary skill in the art before the effective filing date, to use the structure of Shin, including the controller of Shin to generate masks according to the structured sparsity pattern on Wu. It would have been obvious to achieve the benefit of pruning according to the particular hardware attribute (request size, storage unit, bus width, processing parallelism etc (Wu [0050]).
Regarding claim 8, Shin teaches the following:
A first weight buffer comprising an array of first weight registers, each first weight register being configured to store weight value in a fine-grain structured sparsity weight arrangement (Section II.A. first paragraph B matrix represents weights, section III, Load Balancing section fine-grain approach, section IV.A., fig 3 BBUF for weight buffer, section IV.A.2)-3) for structured sparsity weight arrangement, section VI.E. first paragraph for the BBUF comprising registers);
a first weight multiplexer configured to select a first weight register based on the fine-grain structure sparsity weight arrangement and output the weight value stored in the selected first weight register as a first operand (fig 3. BMUX, selection to BMUX based on the sparsity arrangement);
a first activation buffer comprising a first predetermined number of first activation registers, each first activation register being configured to store an activation value (Section II.A, first and second paragraph, A matrix represents input activation, fig 3. ABUF, section VI.E. first paragraph for the ABUF comprising registers);
a first activation multiplexer comprising a second predetermined number of first activation multiplexer inputs, each respective input of the first activation multiplexer being connected to a corresponding first activation register within a first group of first activation registers, the first activation multiplexer being configured to select a first activation register based on the fine-grain structure sparsity weight arrangement and output the activation value stored in the selected first activation register as a second operand value, the activation output as the second operand value corresponding to the weight value output as the first operand value (Fig 3. AMUX, with input select from A operand mask); and
a first multiplier unit configured to output a first product value of the first operand value and the second operand value (fig 3 PE with multiplier array, section based on sparsity mask).
Shin discloses a fine-grain structured sparsity weight arrangement including various numbers of ones in the zero mask (fig 3 control unit), but does not explicitly disclose wherein the fine-grain structure sparsity weight arrangement is selected from a group of fine-grain structured sparsity weight arrangements comprising at least two arrangements of a 1:4 fine-grain structured sparsity weight arrangement, a 2:4 fine-grain structured sparsity weight arrangement, a 4:8 fine-grain structured sparsity weight arrangement, and a 2:8 fine-grain structured sparsity weight arrangement.
However, in the same field of endeavor, Wu discloses an apparatus similar to Shin for hierarchical structured sparse parameter pruning to select activations and weights for a multiplication and accumulation operation (abstract, fig 1A). Wu further discloses wherein the structured sparsity pattern includes 1:4, 2:4, 4:8, and 2:8 ([0049-0050], [0052]. fig 1C, fig 1D). It would have been obvious to one of ordinary skill in the art before the effective filing date, to use the structure of Shin, including the controller of Shin to generate masks according to the structured sparsity pattern on Wu. It would have been obvious to achieve the benefit of pruning according to the particular hardware attribute (request size, storage unit, bus width, processing parallelism etc (Wu [0050]).
Regarding claim 14, in addition to the teachings addressed in the claim 8 analysis, Shin teaches:
a second weight multiplexer configured to select a first weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected first weight register as a third operand value (fig 3. BMUX, selection to BMUX based on the sparsity arrangement as arranged in a different PE as in figure 1) ;
a second activation multiplexer comprising a second predetermined number of second activation multiplexer inputs, each respective input of the second activation multiplexer being connected to a corresponding first activation register within a second group of activation registers that is different from the first group of first activation registers, the second activation multiplexer being configured to select a first activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected first activation register as a fourth operand value, the activation value output as the fourth operand value corresponding to the weight value output as the third operand value (Fig 3. AMUX, with input select from A operand mask, as arranged in the different PE as in figure 1) ; and
a second multiplier unit configured to output a second product value of the third operand value and the fourth operand value (a multiplier in the different PE as in figure 1).
Regarding claim 15, in addition to the teachings addressed in the claim 14 analysis, Shin teaches:
a second weight buffer configured to store weight values of fine-grain structured sparsity weights based on the selected fine-grain structured sparsity weight arrangement (fig 3.BBUF, as arranged in a different PE as in figure 1, which shows ellipses for an array of buffers);
a third weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected second weight register as a fifth operand value; a second activation buffer comprising a first predetermined number of second activation registers, each second activation register being configured to store an activation value (fig 3. BMUX, selection to BMUX based on the sparsity arrangement as arranged in a different PE as in figure 1, which shows ellipses for an array) ;
a third activation multiplexer comprising a second predetermined number of third activation multiplexer inputs, each respective input of the third activation multiplexer being connected to a corresponding second activation register within a first group of second activation registers, the third activation multiplexer being configured to select a second activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected second activation register as a sixth operand value, the activation value output as the sixth operand value corresponding to the weight value output as the fifth operand value (Fig 3. AMUX, with input select from A operand mask, as arranged in the different PE as in figure 1, which shows ellipses for an array);
a third multiplier unit configured to output a third product value of the fifth operand value and the sixth operand value (a multiplier in the different PE as in figure 1, which shows ellipses for an array);
a fourth weight multiplexer configured to select a second weight register based on the selected fine-grain structured sparsity weight arrangement and output the weight value stored in the selected second weight register as a seventh operand value (fig 3. BMUX, selection to BMUX based on the sparsity arrangement as arranged in a different PE as in figure 1, which shows ellipses for an array) ;
a fourth activation multiplexer comprising a second predetermined number of fourth activation multiplexer inputs, each respective input of the fourth activation multiplexer being connected to a corresponding second activation register within a fourth group of second activation registers that is different from the third group of activation registers, the fourth activation multiplexer being configured to select a second activation register based on the selected fine-grain structured sparsity weight arrangement and output the activation value stored in the selected second activation register as an eighth operand value (Fig 3. AMUX, with input select from A operand mask, as arranged in the different PE as in figure 1, which shows ellipses for an array) ; and
a fourth multiplier unit configured to output a fourth product value of the seventh operand value and the eighth operand value (a multiplier in the different PE as in figure 1, which shows ellipses for an array) .
Allowable Subject Matter
Claims 2-7, 9-13, and 16-22 would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, and rewritten to overcome the respective rejections under 35 USC 112(a), and 35 USC 112(b).
The following is a statement of reasons for the indication of allowable subject matter. Applicant claims apparatus related to a neural processing unit including a weight buffer, a weight multiplexer array, an activation buffer, an activation multiplexer array and a multiplier array as in claim 1 and a first weight buffer comprising an array of first weight registers, a first weight multiplexer, a first activation buffer comprising a first predetermined number of first activation registers, a first activation multiplexer, and a first multiplier unit as in claim 8.
Reasons for indication of allowable subject matter include in combination with the remaining limitations:
wherein the wherein the activation multiplexer array comprises a first activation multiplexer comprising seven inputs, each respective input of the first activation multiplexer being connected to a corresponding activation register within a first group of activation registers, the first activation multiplexer being configured to select an activation register in the first group of activation registers based on the selected fine-grain structured sparsity weight arrangement and to output the activation value stored in the selected activation register as a second operand value, the second operand value corresponding to the first operand value and forming a first operand value pair as in claim 2;
wherein the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row as in claim 6, and similarly as in claim 21;
wherein the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row as in claim 7, and similarly as in claim 22;
wherein the first predetermined number of first activation registers comprises 8, and the second predetermined number of activation inputs comprises 7 as in claim 9;
wherein the first predetermined number of first activation registers comprises 8 first activation registers, wherein the second predetermined number of first activation multiplexer inputs comprises 7 first activation multiplexer inputs, wherein the second predetermined number of second activation multiplexer inputs comprises 7 second activation multiplexer inputs, wherein the first predetermined number of second activation registers comprises 8 second activation registers, wherein the second predetermined number of third activation multiplexer inputs comprises 7 third activation multiplexer inputs, and wherein the second predetermined number of fourth activation multiplexer inputs comprises 7 fourth activation multiplexer inputs as in claim 16.
Shin is the closest prior art found. Shin discloses the claimed invention according to the above claim mappings. Shin discloses activation buffer registers (Section VI.E, fig 3), but does not teach or suggest in combination with remaining limitations wherein sizes of these registers including 8 activation registers to store 8 activation values, a first activation multiplexer comprising seven inputs, a predetermined number of activation inputs comprises 7, the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row, the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.
Wu discloses the claimed invention according to the above claim mappings. Wu further discloses an activation register and an activation selector (fig 1B), wherein the activation register stores 8 activation values (fig 1A). Wu does not, however, teach or suggest in combination with the remaining limitations wherein a first activation multiplexer comprising seven inputs, a predetermined number of activation inputs comprises 7, the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row, the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.
US 20180129935 A1 Kim et al., (hereinafter “Kim”) discloses a convolutional neural network system including a data selector configured to output an input value corresponding to a position of a sparse weight from among input values of input data on a basis of a sparse index (abstract, fig 7). Kim further discloses eight inputs to each data selector (fig 9). Kim does not, however, teach or suggest in combination with the remaining limitations wherein a first activation multiplexer comprising seven inputs, a predetermined number of activation inputs comprises 7, the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row, the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.
A.D. Lascorz et al., Bit-Tactical: A Software/Hardware Approach to Exploiting Value and Bit Sparsity in Neural Networks, ASPLOS’19, Session: Machine Learning I, 2019 (hereinafter “Lascorz”) discloses a software scheduling algorithm that approximates an optimal weight schedule within hardware constraints based on a window-based search of weights (Section 1 p. 750 right column, first full paragraph, fig 1). Lascorz further discloses an activation selection unit including activation block registers that hold N input activations (fig 5). Lascorz does not, however, teach or suggest in combination with the remaining limitations wherein sizes of these registers including 8 activation registers to store 8 activation values, a first activation multiplexer comprising seven inputs, a predetermined number of activation inputs comprises 7, the activation registers comprise two rows of four activation registers in which two output multiplexers are configured to select one activation register from each row, the activation registers comprise two rows of two activation registers in which two output multiplexers are configured to select one activation register from each row.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMILY E LAROCQUE whose telephone number is (469)295-9289. The examiner can normally be reached on 10:00am - 1200pm, 2:00pm - 8pm ET M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Andrew Caldwell can be reached on 571-272-3701. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMILY E LAROCQUE/Examiner, Art Unit 2182
Prosecution Insights