DETAILED ACTION
This action is responsive to the claims filed on 06/08/2026. Claims 1-4, 7-12, 15-18, and 20-25 are pending for examination.
This action is Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/05/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Amendments
Applicant’s arguments with respect to the 35 U.S.C. 103 rejection of the claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s arguments regarding the rejection under 35 U.S.C. § 101 have been considered but are not persuasive. Amended independent claims 1, 11, and 17 continue to recite abstract ideas including mathematical concepts and mental processes. Specifically, “generating, through one or more AND operations, a bitmap from an activation sparsity vector and a weight sparsity vector” recites a Boolean mathematical calculation applied to corresponding binary-vector elements. Consistent with this interpretation, Specification paragraph [0069] explains that each bit in the combined sparsity bitmap corresponds to a product of a bit in the activation bitmap and a bit in the weight bitmap, thereby expressly describing the claimed bitmap generation as an element-by-element mathematical product. Further, determining whether the number of nonzero elements in the bitmap is greater than the number of nonzero activations or weights recites counting and comparing numerical values, which constitutes a mathematical calculation and an evaluation that may practically be performed mentally using presented values. The remaining limitations do not integrate these abstract ideas into a practical application: storing the compressed operands constitutes insignificant extra-solution activity; “mitigating the fault” merely recites a desired result without specifying how the fault is technologically mitigated; identifying a nonzero value constitutes data identification; and executing the neural-network model using the identified values merely applies the abstract calculations in the field of neural-network processing. Accordingly, the claims, considered individually and as an ordered combination, do not integrate the judicial exceptions into a practical application or provide significantly more than the judicial exceptions, and the rejection under 35 U.S.C. § 101 is maintained.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-4, 7-12, and 15-25 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Statutory Categories
Claims 1-3 and 6-10 are directed to a method.
Claims 11-12 and 15-16 are directed to a computer-readable medium.
Claims 17-25 are directed to an apparatus.
Independent Claims – Claims 1, 11, and 17
Step 2A Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes. Independent claims 1, 11, and 17 recites limitations that are abstract ideas in the form of mental processes:
Claim 1 recites:
generating, through one or more AND operations, a bitmap from an activation sparsity vector and a weight sparsity vector, the activation sparsity vector indicating one or more positions of the one or more nonzero valued activations in the activation operand, the weight sparsity vector indicating one or more positions of the one or more nonzero valued weights in the weight operand; (generation of a bitmap through operations is being considered as a mathematical concept in the form of mathematical calculation, see paragraph [0069] for the related mathematical disclosure wherein each output bit in the bitmap is the product of the corresponding input bits)
determining whether there is any fault in the sparsity computation based on a number of one or more nonzero elements in the bitmap wherein determining whether there is any fault in the sparsity computation comprises determining whether the number of one or more nonzero elements in the bitmap is greater than a number of the one or more nonzero valued activations in the activation operand or is greater than a number of the one or more nonzero valued weights in the weight operand; (this limitation merely amounts to determining a value at a high level of generality being interpreted as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and in response to determining that there is a fault in the sparsity computation: mitigating the fault, (this limitation merely amounts to determining a value at a high level of generality being interpreted as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
after mitigating the fault, identifying a nonzero valued activation in the compressed activation operand or a nonzero valued weight in the compressed weight operand, (this limitation merely amounts to determining a value at a high level of generality being interpreted as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
Claim 1 also recites the following additional elements for the purposes of Step 2A Prong Two
analysis:
A method for accelerating a neural network model through sparsity computation, the method, comprising: storing a compressed activation operand and a compressed weight operand, the compressed activation operand comprising one or more nonzero valued activations in an activation operand of a neural network model, the compressed weight operand comprising one or more nonzero valued weights in a weight operand of the neural network model; (storing information compressed values is merely data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g))
and executing the neural network model using the nonzero valued activation and nonzero valued weight. (this limitation is merely applying execution of a neural network using non zero activation and weights and is being considered mere instructions to apply an exception, see MPEP 2106.05(h))
The additional limitations fail step 2A Prong 2 of the 101 analysis because they do not transform the claim into a practical application. These limitations are too abstract or lack technical improvement that would make the concept practically useful. Without clear utility or integration into a specific field, the claim does not relate to any particular application. It does not meet the requirements of Step 2A Prong 2, as it fails to make the concept meaningfully applicable in practice.
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is “directed” to an abstract idea. This claim recites the following additional elements for the purposes of Step 2B analysis:
A method for accelerating a neural network model through sparsity computation, the method, comprising: storing a compressed activation operand and a compressed weight operand, the compressed activation operand comprising one or more nonzero valued activations in an activation operand of a neural network model, the compressed weight operand comprising one or more nonzero valued weights in a weight operand of the neural network model; (storing information compressed values is merely data gathering and is considered insignificant extra-solution activity under MPEP 2106.05(g), furthermore it should noted that the courts have recognized receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information) as well-understood, routine, and conventional activity.)
and executing the neural network model using the nonzero valued activation and nonzero valued weight. (this limitation is merely applying execution of a neural network using non zero activation and weights and is being considered mere instructions to apply an exception, see MPEP 2106.05(h))
The claim also fails Step 2B of the analysis because the additional limitations do not amount to significantly more than the abstract idea itself. The additional limitations do not enhance the claim in a way that would move it beyond its abstract ideas as they minimally elaborate on the core concept without adding any inventive or technical substance. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claims 11 and 17 recite limitations substantially similar to claim 1, as such a similar analysis applies.
Claim 11 recites an additional limitation for consideration:
One or more non-transitory computer-readable media storing instructions executable to perform operations for in-network computing (Under step 2A prong II and step 2B, this limitation is invoking computers or other machinery merely as a tool to perform an existing process, see MPEP 2106.05(f))
Claim 17 recites an additional limitation for consideration:
An apparatus, comprising: a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising (Under step 2A prong II and step 2B, this limitation is invoking computers or other machinery merely as a tool to perform an existing process, see MPEP 2106.05(f))
Dependents of Claims 1, 11, and 17
The remaining dependent claims corresponding to independent claims 1, 11, and 17 do not recite additional elements, whether considered individually or in combination, that are sufficient to integrate the judicial exception into a practical application or amount to significantly more than the judicial exception. The analysis of which is shown below:
The claims below recite additional limitations which fail step 2A Prong 2 of the 101 analysis because they do not transform the claim into a practical application. These limitations are too abstract or lack technical improvement that would make the concept practically useful. Without clear utility or integration into a specific field, the claim does not relate to any particular application. It does not meet the requirements of Step 2A Prong 2, as it fails to make the concept meaningfully applicable in practice.
The claims also fails Step 2B of the analysis because the additional limitations do not amount to significantly more than the abstract idea itself. The additional limitations do not enhance the claim in a way that would move it beyond its abstract ideas as they minimally elaborate on the core concept without adding any inventive or technical substance. The claims are unpatentable.
Claim 2 recites the additional limitation of:
The method of claim 1, wherein the bitmap is generated based on a previous bitmap, another nonzero valued activation in the compressed activation operand or another nonzero valued weight in the compressed weight operand was identified based on the previous bitmap, (generation of the bitmap based on a previous bitmap is still being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and determining whether there is any fault in the sparsity computation comprises: determining a number of one or more nonzero elements in the previous bitmap; (this limitation merely amounts to determining a value at a high level of generality being interpreted as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and determining whether the number of one or more nonzero elements in the bitmap is not equal to a sum of one plus the number of one or more nonzero elements in the previous bitmap. (this limitation merely amounts to determining a value at a high level of generality being interpreted as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 3 recites the additional limitation of:
The method of claim 2, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap; (generation of the bitmap based on a previous bitmap is still being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand based on the new bitmap. (a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 4 recites the additional limitation of:
The method of claim 3, wherein generating the new bitmap based on the previous bitmap comprises: replacing a nonzero element in the previous bitmap with zero. (replacing a value with 0 is being considered a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 7 recites the additional limitation of:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (generation of the bitmap based on a previous bitmap is still being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
determining a position of the nonzero valued activation in the compressed activation operand based on the bitmap; (determining a position of a value in a bitmap is being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
determining a first position of the nonzero valued activation in the compressed activation operand based on the first bitmap; (determining a position of a value in a bitmap is being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
determining a second position of the nonzero valued activation in the compressed activation operand based on the second bitmap; (determining a position of a value in a bitmap is being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and identifying the nonzero valued activation in the compressed activation operand based on the position, the first position, and the second position. (a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 8 recites the additional limitation of:
The method of claim 1, wherein identify the nonzero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (generation of the bitmap based on a previous bitmap is still being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
determining a position of the nonzero valued weight in the compressed weight operand based on the bitmap; (determining a position of a value in a bitmap is being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
determining a first position of the nonzero valued weight in the compressed weight operand based on the first bitmap; (determining a position of a value in a bitmap is being considered as a mental process of evaluation which can reasonably be permed in human mind or with aid of pen and paper)
and identifying the nonzero valued weight in the compressed weight operand based on the position, the first position, and the second position. (a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 9 recites the additional limitation of:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand comprises: identifying another nonzero valued activation in the compressed activation operand, wherein the another nonzero valued activation is subsequently next to a previously identified nonzero valued activation in the compressed activation operand. (a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 10 recites the additional limitation of:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand comprises: determining a position of the nonzero valued activation in the compressed activation operand; (determining a position of non-zero value is a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
and determining a position of the nonzero valued weight in the compressed weight operand, wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (a mental process of position identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claims 12 and 18 are substantially similar to claim 2, as such a similar analysis applies.
Claims 15 and 20 are substantially similar to claim 7, as such a similar analysis applies.
Claim 16 is substantially similar to claim 9, as such a similar analysis applies.
Claim 19 is substantially similar to claim 5, as such a similar analysis applies.
Claim 21 recites the additional limitation of:
The one or more non-transitory computer-readable media of claim 12, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap; (generation of a bitmap through operations is being considered as a mathematical concept in the form of mathematical calculation, see paragraph [0069] for the related mathematical disclosure wherein each output bit in the bitmap is the product of the corresponding input bits)
and identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand based on the new bitmap. (determining a position of non-zero value is a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
and determining a position of the nonzero valued weight in the compressed weight operand, (a mental process of position identification which can reasonably be permed in human mind or with aid of pen and paper)
wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (this limitation directly recites mathematical processes in the form of mathematical calculation (multiplication))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 22 recites the additional limitation of:
The one or more non-transitory computer-readable media of claim 21, wherein generating the new bitmap based on the previous bitmap comprises: replacing a nonzero element in the previous bitmap with zero. (a mental process of value replacement which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 23 recites the additional limitation of:
The one or more non-transitory computer-readable media of claim 11, wherein identify the nonzero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (generation of a bitmap through operations is being considered as a mathematical concept in the form of mathematical calculation, see paragraph [0069] for the related mathematical disclosure wherein each output bit in the bitmap is the product of the corresponding input bits)
determining a position of the nonzero valued weight in the compressed weight operand based on the bitmap; (determining an arbitrary position for a weight operand in a bitmap is being considered a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
determining a first position of the nonzero valued weight in the compressed weight operand based on the first bitmap; (determining an arbitrary position for a weight operand in a bitmap is being considered a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
determining a second position of the nonzero valued weight in the compressed weight operand based on the second bitmap; (determining an arbitrary position for a weight operand in a bitmap is being considered a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
and identifying the nonzero valued weight in the compressed weight operand based on the position, the first position, and the second position. (a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 24 recites the additional limitation of:
The one or more non-transitory computer-readable media of claim 11, wherein identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand comprises: determining a position of the nonzero valued activation in the compressed activation operand; (generation of a bitmap through operations is being considered as a mathematical concept in the form of mathematical calculation, see paragraph [0069] for the related mathematical disclosure wherein each output bit in the bitmap is the product of the corresponding input bits)
And determining a position of the nonzero valued weight in the compressed weight operand, (determining an arbitrary position for a weight operand in a bitmap is being considered a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (this limitation directly recites mathematical processes in the form of mathematical calculation (multiplication))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim 25 recites the additional limitation of:
The apparatus of claim 17, wherein identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand comprises: determining a position of the nonzero valued activation in the compressed activation operand; (generation of a bitmap through operations is being considered as a mathematical concept in the form of mathematical calculation, see paragraph [0069] for the related mathematical disclosure wherein each output bit in the bitmap is the product of the corresponding input bits)
and determining a position of the nonzero valued weight in the compressed weight operand, (determining an arbitrary position for a weight operand in a bitmap is being considered a mental process of identification which can reasonably be permed in human mind or with aid of pen and paper)
wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (this limitation directly recites mathematical processes in the form of mathematical calculation (multiplication))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, nor provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 7-8, 10-11, 17, and 23-25 are rejected under 35 U.S.C. 103 as being unpatentable over Raha et al., (US 12141683 B2), hereafter referred to as Raha in view of Moyer et al. (US 2009/0016480 A1), hereafter referred to as Moyer and further in view of Mody et al. (US 2020/0074287 A1), hereafter referred to as Mody.
Claim 1: Raha teaches:
A method for accelerating a neural network model through sparsity computation, the method comprising: storing a compressed activation operand and a compressed weight operand, the compressed activation operand comprising one or more nonzero valued activations in an activation operand of a neural network model, the compressed weight operand comprising one or more nonzero valued weights in a weight operand of the neural network model; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”
Raha, col. 33, lines 5-14, “As used herein, the terms "sparse vector", "sparse matrix", and "sparse array" refer to an input vector, matrix, or array including both non-zero elements and zero elements. As used herein, the terms "ZVC data vector" "ZVC matrix", and "ZVC array" refer to a vector, matrix, or array that includes all non-zero elements of a vector, matrix, or array in the same order as a sparse vector, matrix, or array, but excludes all zero elements.”, Under Broadest Reasonable Interpretation (BRI), the claimed “activation operand” and “weight operand” read on Raha’s IF (input activation/input feature map) and FL (filter data/weights), respectively. Raha teaches these operands are stored in compressed format and explicitly ties compression to accompanying bitmaps (“IF 681 and FL 682 … stored in compressed format … with the accompanying bitmaps”). Raha’s ZVC definition further supports that compressed storage can be nonzero-only (“includes all non-zero elements … excludes all zero elements”), matching the claim’s “compressed … comprising one or more nonzero valued [activations/weights].”)
Generating, through one or more AND operations, a bitmap from an activation sparsity vector and a weight sparsity vector, the activation sparsity vector indicating one or more positions of the one or more nonzero valued activations in the activation operand, the weight sparsity vector indicating one or more positions of the one or more nonzero valued weights in the weight operand; (Raha, col. 12, line 37, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output.”
col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wren signals coming from the schedule aware sparse decoders. Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.”, The claim’s “activation sparsity vector” and “weight sparsity vector” read on Raha’s sparsity bitmaps of IF and sparsity bitmaps of FL. Under BRI, a “bitmap” is a bit-vector and therefore a type of “vector” that indicates positions of interest (here, sparsity/nonzero structure). Raha expressly teaches these sparsity bitmaps exist (including that the “sparsity bitmap is automatically recreated”) and that IF/FL sparsity bitmaps are used as the sparsity metadata for controlling sparse compute. Raha’s IF bitmap is the claimed activation sparsity vector because it represents the locations of nonzero activation data, AND Raha’s FL bitmap is the claimed weight sparsity vector because it represents corresponding sparse filter/weight data,)
identifying a nonzero valued activation in the compressed activation operand or a nonzero valued weight in the compressed weight operand; (Raha, col. 12, lines 37-42, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output.”, The claimed “bitmap” used for identification reads on Raha’s Combined Bitmap produced from the Activation Bitmap (IF sparsity bitmap) and Weight Bitmap (FL sparsity bitmap), furthermore, Raha explicitly uses the ANDed combination (“combination of these two bitmaps (AND)”) as the control signal to drive operand selection via “find-first logic.” Because Raha then states “only those IF … and FL … operands are read” (i.e., selected/identified for compute) such that they “will result in non-zero partial sums,” it teaches identifying the nonzero(-relevant) activation/weight operands based on the bitmap. This aligns with claim 1’s “identifying a nonzero valued activation … or … weight … based on the bitmap.”)
and executing the neural network model using the nonzero valued activation and nonzero valued weight. (Raha, col. 12, lines 37-42, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output.”, Raha teaches that its IF activation and FL weight operands are supplied to MAC circuitry and that the bitmap-driven find-first logic selects those activation/weight operand pairs that produce nonzero partial sums during neural-network computation. Raha therefore teaches executing the neural-network model by supplying the selected nonzero activation and corresponding nonzero weight to the MAC circuitry for multiplication and accumulation as part of neural-network inference.)
Moyer, in the same field of , teaches the following limitations which Settles fails to teach:
determining whether there is any fault in the sparsity computation, wherein determining whether there is any fault in the sparsity computation comprises determining whether the number of one or more nonzero elements in the bitmap is greater than a number of the one or more nonzero valued activations in the activation operand or is greater than a number of the one or more nonzero valued weights in the weight operand; (Moyer, paragraph 5, “Population count (“popcount”) circuits function to output the number of logic “1” bits in data such as an input word or a vector value. For example, a population count is performed on a vector to determine the amount of nonzero elements within the vector.”; Paragraph 26, “In a step 86 the first population count is compared with the second population count. In other words the count value associated with the correlated data inputs is compared with the count value associated with the uncorrelated data inputs. In response to the comparison, a determination is made of a confidence level of the correlation. The confidence level is a determination of the level of certainty that the count value indicates multiple errors or the presence of multiple bits of the predetermined bit value in the value being counted. For example, if the correlated set count value is much higher than the uncorrelated set count value, an assumption may be made that the higher count value in the correlated set is attributable to a single bit value from which a number of other counted bits are correlated to. Based upon this confidence determination, a subsequent action such as modification of an integrated circuit operating parameter may or may not be taken.” Moyer therefore teaches determining numbers of asserted/nonzero elements in binary vectors, comparing respective population counts, and using the relationship between such counts to identify erroneous operation. When Moyer’s population-count fault-checking technique is applied to Raha’s activation sparsity bitmap, weight sparsity bitmap, and AND-generated combined bitmap, the respective population counts provide an integrity check on Raha’s sparsity computation. Because an AND-generated bitmap represents positions asserted in both source sparsity vectors, the number of asserted elements in the resulting bitmap cannot properly exceed the number represented by either source sparsity vector. A resulting bitmap count exceeding the number of nonzero activations represented by the activation sparsity vector or the number of nonzero weights represented by the weight sparsity vector therefore indicates an erroneous sparsity-computation condition. Accordingly, the combined teachings of Raha and Moyer teach the claimed fault determination based on the recited population-count comparison.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the bitmap-driven sparsity computation of Raha to incorporate the population-count fault-checking technique of Moyer. Raha teaches the use of correlated activation-side, weight-side, and AND-combined sparsity bitmaps to control selection of operands for neural-network computation, while Moyer teaches determining population counts of binary information, comparing such counts, and using population-count information to test or check erroneous integrated-circuit operation. A person of ordinary skill in the art would have been motivated to employ Moyer’s population-count checking technique with Raha’s sparsity bitmaps in order to detect inconsistent or erroneous sparsity-control information before such information results in incorrect operand selection and computation, thereby improving the reliability of Raha’s bitmap-driven sparse-processing operation.
Mody, in the same field of neural network implementation, teaches the following which Raha and Moyer fails to teach:
in response to determining that there is a fault in the sparsity computation: mitigating the fault, (Mody, paragraph 49, “To facilitate error correction, in one example an A panel memory 712 is used to hold the entire A matrix before proceeding to the multiplication operation. A dual B panel memory 714 is used to store two copies of the B matrix for ping-pong operation, where the two copies are used for alternating operations. Matrices A and B are first read or retrieved from the memory 702 and loaded into the A and B panel memories 712, 714. The reference checksums are also read from the memory 702 and loaded into an A and B checksum buffer 716. During the matrix read or retrieval operation, the checksums of the A and B matrices are computed in A and B checksum blocks 718, 720. At the end of reading from memory 702, the reference checksums are compared against the computed checksums and any errors in the A or B matrices are detected and the necessary correction computed.”, Mody therefore teaches responding to detection of a computational fault in neural-network processing by calculating and applying a correction to the erroneous data or computation. Such correction constitutes mitigation of the detected fault before the affected data is subsequently used by the neural-network engine.)
after mitigating the fault, (Mody, paragraph 56, “Neural network scaler and nonlinear computational blocks or logic may be triplicated to provide error detection and error correction capability. The inputs to the nonlinear matrix block 772 are the output accumulated matrix of the C panel memory 768 and the error output of the C checksum block 774. Any error is corrected and the corrected received accumulated matrix is operated on in the triplicate computational blocks of the nonlinear matrix block 772.”, Mody therefore teaches an ordered fault-handling operation in which a detected computational error is corrected before subsequent neural-network computation proceeds. Accordingly, Mody teaches performing the subsequent operand-processing operations after mitigation of the detected fault.)
It would further have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined teachings of Raha and Moyer with the fault-mitigation teachings of Mody. Mody is directed to fault-tolerant neural-network hardware and teaches detecting computational errors, correcting the detected errors, and thereafter continuing neural-network computation using corrected data. A person of ordinary skill in the art would have been motivated to apply Mody’s fault-mitigation technique when the population-count check incorporated from Moyer detects erroneous sparsity information in Raha’s bitmap-driven operand-selection logic, thereby preventing propagation of erroneous sparsity-control information into the neural-network computation while permitting operation of the neural-network accelerator to continue.
Claim 7: Raha, Moyer, and Mody teaches the limitations of claim 1. Raha further teaches:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”
Raha, col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wren signals coming from the schedule aware sparse decoders.”, Raha teaches two bitmaps for sparsity when IF (input activation / feature map) and FL (weights/filters) are stored compressed with “accompanying bitmaps.” Under the claim’s BRI: the “activation sparsity vector” maps to the IF-side sparsity bitmap (recreated using IF wren signals), and the “weight sparsity vector” maps to the FL-side sparsity bitmap (recreated using FL wren signals). Thus, Raha teaches generating a first bitmap (IF) and second bitmap (FL) from sparsity-indicating information associated with each operand.)
determining a position of the nonzero valued activation in the compressed activation operand based on the bitmap; (Raha, col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wren signals coming from the schedule aware sparse decoders. Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.”,
Raha, col. 12, line 65, “As stated previously, the MACs can source the operands either from the M number of IF, FL, and OF RFs 608 based on the optimal schedule. Due to data dependent sparsity, the number of read ports in each IF subbank may need to be increased from 1 to 4 for the M×M mode of operation as each of the subbanks will be independently accessed by 4 rd pointers based on the combined sparsity of data in each IF subbank with 4 different FL subbanks.”, Raha’s “bitmap” maps to the combined (AND) sparsity bitmap (IF-bitmap AND FL-bitmap) that is explicitly fed to find-first logic. Under BRI, “determining a position … based on the bitmap” corresponds to using the combined bitmap (and find-first) to derive which element index/location to access, which Raha further characterizes operationally via “rd pointers based on the combined sparsity.” That pointer/index is the claimed “position” used to identify the activation to be processed.)
determining a first position of the nonzero valued activation in the compressed activation operand based on the first bitmap; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, in Raha, each compressed operand (IF and FL) is associated with its own bitmap (BM) used “for decoding into dense data.” Under BRI, a bitmap that supports decoding inherently provides positional information (i.e., which dense indices correspond to nonzero entries). Thus, the first bitmap (mapped above to the IF/activation bitmap) is used to obtain the first position of a nonzero activation in the compressed IF operand during decode/selection.)
determining a second position of the nonzero valued activation in the compressed activation operand based on the second bitmap; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, in Raha, each compressed operand (IF and FL) is associated with its own bitmap (BM) used “for decoding into dense data.” Under BRI, a bitmap that supports decoding inherently provides positional information (i.e., which dense indices correspond to nonzero entries). Thus, the second bitmap (mapped above to the FL/weight bitmap) is used to obtain the second position of a nonzero activation in the compressed FL operand during decode/selection.)
and identifying the nonzero valued activation in the compressed activation operand based on the position, the first position, and the second position. (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, Raha teaches identification/selection of which IF activation operand is to be read by using (i) combined bitmap (AND) via find-first (claimed “position”) and (ii) operand-side bitmap decode (the claimed “first position” for IF, “second position” for FL). The result is that the selected IF operand actually read is determined using the combined selection logic plus bitmap positional information; Raha expresses this outcome as reading only those IF/FL operands indicated by the bitmap logic.)
Claim 8: Raha, Moyer, and Mody teaches the limitations of claim 1. Raha further teaches:
The method of claim 1, wherein identifying the nonzero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”
Raha, col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wren signals coming from the schedule aware sparse decoders.”, Raha teaches two bitmaps for sparsity when IF (input activation / feature map) and FL (weights/filters) are stored compressed with “accompanying bitmaps.” Under the claim’s BRI: the “activation sparsity vector” maps to the IF-side sparsity bitmap (recreated using IF wren signals), and the “weight sparsity vector” maps to the FL-side sparsity bitmap (recreated using FL wren signals). Thus, Raha teaches generating a first bitmap (IF) and second bitmap (FL) from sparsity-indicating information associated with each operand.)
determining a position of the nonzero valued activation in the compressed weight operand based on the bitmap; (Raha, col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wren signals coming from the schedule aware sparse decoders. Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.”,
Raha, col. 12, line 65, “As stated previously, the MACs can source the operands either from the M number of IF, FL, and OF RFs 608 based on the optimal schedule. Due to data dependent sparsity, the number of read ports in each IF subbank may need to be increased from 1 to 4 for the M×M mode of operation as each of the subbanks will be independently accessed by 4 rd pointers based on the combined sparsity of data in each IF subbank with 4 different FL subbanks.”, Raha’s “bitmap” maps to the combined (AND) sparsity bitmap (IF-bitmap AND FL-bitmap) that is explicitly fed to find-first logic. Under BRI, “determining a position … based on the bitmap” corresponds to using the combined bitmap (and find-first) to derive which element index/location to access, which Raha further characterizes operationally via “rd pointers based on the combined sparsity.” That pointer/index is the claimed “position” used to identify the activation to be processed.)
determining a first position of the nonzero valued weight in the compressed weight operand based on the first bitmap; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, in Raha, each compressed operand (IF and FL) is associated with its own bitmap (BM) used “for decoding into dense data.” Under BRI, a bitmap that supports decoding inherently provides positional information (i.e., which dense indices correspond to nonzero entries). Thus, the first bitmap (mapped above to the IF/activation bitmap) is used to obtain the first position of a nonzero weight in the compressed IF/FL bitmap during decode/selection.)
determining a second position of the nonzero valued weight in the compressed weiht operand based on the second bitmap; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, in Raha, each compressed operand (IF and FL) is associated with its own bitmap (BM) used “for decoding into dense data.” Under BRI, a bitmap that supports decoding inherently provides positional information (i.e., which dense indices correspond to nonzero entries). Thus, the second bitmap (mapped above to the FL/weight bitmap) is used to obtain the second position of a nonzero weight in the compressed FL/IF bitmap during decode/selection.)
and identifying the nonzero valued weight in the compressed weight operand based on the position, the first position, and the second position. (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, Raha teaches identification/selection of which FL weight operand is to be read by using (i) combined bitmap (AND) via find-first (claimed “position”) and (ii) operand-side bitmap decode (the claimed “first position” for IF, “second position” for FL). The result is that the selected FL (weight) operand actually read is determined using the combined selection logic plus bitmap positional information; Raha expresses this outcome as reading only those IF/FL operands indicated by the bitmap logic.)
Claim 10: Raha, and Narad teaches the limitations of claim 1, Raha further teaches:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand comprises: determining a position of the nonzero valued activation in the compressed activation operand; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, Under BRI, the “compressed activation operand” maps to the compressed IF data in the IF RF, with an accompanying IF bitmap. Raha then teaches using the combined bitmaps as input to find-first logic, and implementing operand selection via read pointers based on combined sparsity. Those pointers/indexes are the claimed “position” used to locate/select the nonzero activation from the compressed IF operand.)
and determining a position of the nonzero valued weight in the compressed weight operand, (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”, Under BRI, the “compressed weight operand” maps to the compressed FL (weights/filters) in the FL RF, with an accompanying FL bitmap. Raha teaches that the ANDed bitmap drives find-first selection such that only those FL operands are read (selected by bitmap/pointers). That selection inherently determines the position/index of the nonzero weight within the compressed FL operand.)
wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (Raha, col. 10, lines 8-12, “Here, an input activation/input feature map (IF) and weights/filters (FL) are fed into the MAC 606, and the MAC 606 generates an output activations/output feature map (OF).”, Raha explicitly links IF (activations) and FL (weights) to a MAC datapath, i.e., IF and FL are “fed into the MAC,” satisfying the claim’s requirement that activation is multiplied with weight (MAC multiplication component).)
Claim 11 is substantially similar to claim 1, as such a similar analysis applies.
Claim 11 has the following additional limitations for consideration which Raha further teaches:
One or more non-transitory computer-readable media storing instructions executable to perform operations for in-network computing, (Raha, claim 12, “One or more non-transitory computer-readable media (NTCRM) storing instructions, ”)
Claim 17 is substantially similar to claim 1, as such a similar analysis applies.
Claim 17 has the following additional limitations for consideration which Raha further teaches:
a computer processor for executing computer program instructions; and a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising: (Raha, claim 12, “One or more non-transitory computer-readable media (NTCRM) storing instructions, when executed by one or more processors, cause the one or more processors to:”)
Claim 23: Raha, and Narad teaches the limitations of claim 1, Raha further teaches:
The one or more non-transitory computer-readable media of claim 11, wherein identify the nonzero valued weight in the compressed weight operand comprises: generating a first bitmap and a second bitmap based on the activation sparsity vector and the weight sparsity vector; (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”; Col. 12, lines 56-63, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wr_en signals coming from the schedule aware sparse decoders. Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.”, Raha teaches respective sparsity bitmaps associated with its IF activation data and FL weight data, with the IF bitmap identifying the sparsity of the activation operand and the FL bitmap identifying the sparsity of the weight operand. Raha expressly teaches recreating the respective IF and FL sparsity bitmaps from the activation and weight sparsity information and subsequently combining those bitmaps during sparse computation. Thus, Raha teaches generating the claimed first and second bitmaps based on the activation sparsity vector and weight sparsity vector.)
determining a position of the nonzero valued weight in the compressed weight operand based on the bitmap; (Raha, col. 12, lines 36-43, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output.”; Col. 13, line 1, “Due to data dependent sparsity, the number of read ports in each IF subbank may need to be increased from 1 to 4 for the M×M mode of operation as each of the subbanks will be independently accessed by 4 rd pointers based on the combined sparsity of data in each IF subbank with 4 different FL subbanks.”, Raha’s combined sparsity bitmap is supplied to its find-first logic to determine which compressed IF and FL operands are to be accessed, and the corresponding compressed operand locations are accessed through read pointers determined from the combined sparsity information. The read pointer associated with the selected FL operand therefore identifies the position of the nonzero-valued weight in the compressed weight operand based on the combined bitmap.)
determining a first position of the nonzero valued weight in the compressed weight operand based on the first bitmap; (Raha, col. 12, lines 33-40, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums…”; Col. 12, lines 56-59, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wr_en signals coming from the schedule aware sparse decoders”, Raha associates its compressed IF and FL operands with respective sparsity bitmaps that preserve the sparse positional relationship of the compressed operands. The bitmap associated with the activation-side sparse information therefore provides a first positional indication for the corresponding activation/weight pair selected by the find-first logic. Accordingly, Raha teaches determining a first position associated with the selected nonzero weight based on the first sparsity bitmap.)
determining a second position of the nonzero valued weight in the compressed weight operand based on the second bitmap; (Raha, col. 12, lines 33-40, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums…”; Col. 12, lines 59-63, “Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.s”, Raha’s second, FL-side sparsity bitmap represents the sparse positional information associated with the compressed FL weight operand. Accordingly, the FL-side bitmap supplies a second positional indication of the selected nonzero weight, which is used together with the sparse-selection logic to access the corresponding FL operand. Thus, Raha teaches determining the claimed second position based on the second bitmap.)
and identifying the nonzero valued weight in the compressed weight operand based on the position, the first position, and the second position. (Raha, col. 12, lines 33-40, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums…”; Col. 12, lines 59-63, “Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.s”; Col. 13 lines 1-5, “…each of the subbanks will be independently accessed by 4 rd pointers based on the combined sparsity of data in each IF subbank with 4 different FL subbanks.” Raha teaches that the combined-bitmap-derived position and the respective positional information provided by the individual IF-side and FL-side sparsity bitmaps are used by the find-first/read-pointer circuitry to select the corresponding compressed IF and FL operands. Accordingly, the nonzero FL weight is identified from the compressed weight operand using the combined sparsity position together with the respective positional sparsity information supplied by the first and second bitmaps, thereby teaching identification of the nonzero-valued weight based on the claimed position, first position, and second position.)
Claim 24: Raha, and Narad teaches the limitations of claim 1, Raha further teaches:
The one or more non-transitory computer-readable media of claim 11, wherein identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand comprises: determining a position of the nonzero valued activation in the compressed activation operand; (Raha, col. 12, lines 36-43, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output.”; Col. 13, line 1, “Due to data dependent sparsity, the number of read ports in each IF subbank may need to be increased from 1 to 4 for the M×M mode of operation as each of the subbanks will be independently accessed by 4 rd pointers based on the combined sparsity of data in each IF subbank with 4 different FL subbanks.”, Raha teaches compressed IF activation operands having accompanying sparsity bitmaps and uses the combined sparsity bitmap to control find-first and read-pointer operation. The read pointer identifies which compressed IF entry is selected for computation and therefore corresponds to determining the position of the nonzero-valued activation within the compressed activation operand.)
and determining a position of the nonzero valued weight in the compressed weight operand, (Raha, col. 12, lines 33-43, “For a sparse ML accelerator, the data in the IF 681 and FL 682 RFs 608 is usually stored in compressed format in input channel (IC) dimension with the accompanying bitmaps stored within dedicated bitmap storage. The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read from the IF 681 and FL 682 RFs 608 that will result in non-zero partial sums towards the accumulated output… In these embodiments, a bitmap (BM) is encoded inline with the compressed data for decoding into dense data.”; Col. 12, lines 56-63, “Within each PE 230, the sparsity bitmap is automatically recreated as seen during the load phase using incoming IF and FL wr_en signals coming from the schedule aware sparse decoders. Subsequently, during the compute phase, the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic to skip 0 data and gain performance and energy improvements for the entire accelerator system 124.”, Raha’s FL operand corresponds to the claimed weight operand, and Raha’s FL-side sparsity bitmap and combined IF/FL sparsity information are used by the find-first circuitry to select the FL entry that participates in a nonzero computation. Accordingly, the bitmap-driven operand-selection circuitry determines the location of the selected nonzero FL weight in the compressed weight operand and therefore teaches determining the claimed position of the nonzero-valued weight.)
wherein executing the neural network model comprises multiplying the nonzero valued activation by the nonzero valued weight. (Raha, col. 10, lines 5-11, “Here, an input activation/input feature map (IF) and weights/filters (FL) are fed into the MAC 606, and the MAC 606 generates an output activations/output feature map (OF).”; Col. 4, lines 20-29, “Input data, known as input activations or feature maps, are also brought into the PE array 210 … Inside the PEs 230, multiply-and-accumulate (MAC) operations are performed in respective MAC operators through multiple input channels … to generate output activations. Multiple sets of weight tensors may be used against a given set of activations to produce an output tensor volume.”,
Col. 12, lines 36-43, “The find-first logic uses a combination of these two bitmaps (AND) to skip RF 608 values that result in zero partial sums (or MAC 606 multiplications) such that only those IF 681 and FL 682 operands are read … that will result in non-zero partial sums towards the accumulated output.”, Raha expressly teaches supplying activation data (IF) and corresponding weight/filter data (FL) to a multiply-and-accumulate unit and performing MAC multiplication between the selected activation and weight operands. Raha’s sparsity logic further ensures that only IF activation and FL weight pairs corresponding to nonzero computation results are read for the MAC operation. Thus, the activation and weight selected through Raha’s sparsity logic correspond to the claimed nonzero-valued activation and nonzero-valued weight, and the MAC operation expressly teaches multiplying those selected values during execution of the neural-network model.)
Claim 25 recites limitations substantially similar to claim 24, as such a similar analysis applies.
Claims 2-4, 12, 15, 18, 20, and 21-22 are rejected under 35 U.S.C. 103 as being unpatentable over Raha in view of Moyer, Mody, and Hall et al., (US20120144089A1), hereafter referred to as Hall.
Claim 2: Raha, Moyer, and Mody teaches the limitations of claim 1, Hall in the same field bit mask analysis, teaches the following limitations which Raha and Narad fail to teach:
The method of claim 1, wherein the bitmap is generated based on a previous bitmap, another nonzero valued activation in the compressed activation operand or another nonzero valued weight in the compressed weight operand was identified based on the previous bitmap, (Hall, paragraph 50, “Mask register 410 may track the completion of the gather operation by monitoring the data stored in destination register 415.”
Hall, paragraph 51, “In one embodiment, a processor may call or execute the gather step instruction, for example, in a ‘while’ loop or repeating ‘if’ statement, until mask register 410 may be completely cleared”, Hall’s mask register 410 is a bitmap-like state vector that is read each iteration to decide which elements remain to be processed (analogous to using a previous bitmap to identify the next candidate element). Updating the mask register across iterations provides the claimed “bitmap generated based on a previous bitmap” concept in an iterative selection workflow.)
and determining whether there is any fault in the sparsity computation comprises: determining a number of one or more nonzero elements in the previous bitmap; and determining whether the number of one or more nonzero elements in the bitmap is not equal to a sum of one plus the number of one or more nonzero elements in the previous bitmap. (Hall, paragraph 50, “For example, a “1” in mask register 410 may indicate that a corresponding data element was not written into destination register 415; otherwise a “0” may be used. In such embodiments, the gather instruction may execute until the sum of the values of the state elements in mask register 410 is equal to a predetermined threshold, for example, the number of data elements to be gathered, which may vary for each gather instruction.”, Hall expressly uses a bit-sum (“sum of the values of the state elements”) as a progress/termination check, i.e., counting set bits in the state/bitmap.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Raha, Moyer, and Mody by further incorporating the teachings of Hall to provide the additional functionality regarding (i) generating a current bitmap based on a previous bitmap and (ii) verifying correctness of the update via a count-based check. Raha already teaches iterative bitmap-guided identification in sparse deep-learning compute, e.g., “the sparsity bitmaps of IF and FL are combined and given as an input to the find-first logic” and “the find-first logic uses a combination of these two bitmaps (AND) … such that only those IF and FL operands are read.” Narad teaches determining mask/bitmap validity using a count of set bits (“population count”) and indicates invalidity when the mask does not have expected structure (“Returns -1 … if the 1-bits … are not …”). Hall is analogous (mask/bitmap-driven iterative selection) and explicitly teaches maintaining and updating a “previous” mask state across iterations: “mask register 410 may track the completion …” and the operation proceeds via repeated steps until the mask state indicates completion (e.g., cleared/thresholded), including a count-based completion check where “the sum of the values of the state elements in mask register 410 is equal to a predetermined threshold” (Hall, paragraph 50). A motivation of which would have been to ensure correct, repeatable stepwise progress (i.e., robust iterative bitmap state updates) during repeated bitmap-guided selection.
Claim 3: Raha, Moyer, Mody and Hall teaches the limitations of claim 2, Hall further teaches:
The method of claim 2, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap and identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand based on the new bitmap. (Hall, paragraph 48, “According to embodiments of the invention, by storing data elements that have been gathered in destination register 415, the data previously collected by an interrupted or stopped gather operation may be preserved and the gather operation may restart in the middle. The interrupted gather operation (e.g., having gathered one or more data elements) may start from the middle, for example, gathering the remaining elements missing from destination register 415.”,
Hall, paragraph 50, “Mask register 410 may track the completion of the gather operation by monitoring the data stored in destination register 415. In one embodiment, there is a one-to-one correspondence between data elements stored in destination register 415 and corresponding state elements stored in mask register 410. State elements or values may include flags, markers, tabs, indicators, signals, and or other numbers, bits and/or codes for indicating whether of not a corresponding data element (e.g., in a corresponding or pointed register location) is stored in destination register 415. For example, a “1” in mask register 410 may indicate that a corresponding data element was not written into destination register 415; otherwise a “0” may be used.”, In Hall, the claimed “previous bitmap” corresponds to the contents of mask register 410 at the time the gather operation is stopped/interrupted (i.e., the bitmap state indicating which corresponding data elements are still not written/loaded). Hall explains that “Mask register 410 may track the completion of the gather operation” and that “a ‘1’ in mask register 410 may indicate that a corresponding data element was not written into destination register 415; otherwise a ‘0’ may be used.” The claimed “new bitmap based on the previous bitmap” corresponds to the updated mask register 410 after at least one element has been gathered/written and the corresponding state bit(s) are cleared. Hall explicitly teaches this update behavior: when an element is gathered and written, “the corresponding … state elements in mask register 410 may be set to ‘0’.” Hall further teaches that after interruption, the operation can “restart in the middle” and continue by “gathering the remaining elements missing from destination register 415,” which corresponds to identifying remaining elements using the updated (new) bitmap state.)
Claim 4: Raha, Moyer, Mody and Hall teaches the limitations of claim 3, Hall further teaches:
The method of claim 3, wherein generating the new bitmap based on the previous bitmap comprises: replacing a nonzero element in the previous bitmap with zero. (Hall, paragraph 55, “For example, if the cache line read has one element to be gathered, then one element may be written into destination register 415 and the corresponding one bit state elements in mask register 410 may be set to “0”.”, Hall expressly teaches clearing the bit corresponding to a processed element (“set to ‘0’”), which teaches “replacing a nonzero element… with zero”.)
Claim 21: Raha, Moyer, and Mody teaches the limitations of claim 1. Hall in the same field bit mask analysis, teaches the following limitations which Raha Moyer, and Mody fail to teach:
The one or more non-transitory computer-readable media of claim 12, wherein identifying the nonzero valued activation in the compressed activation operand comprises: generating a new bitmap based on the previous bitmap; (Hall, paragraph 48, “Storing data gathered from cache memory 416 and/or external memory 435 in destination register 415 may preserve the data, for example, in case the complete gather operation (e.g., gathering all 16 vectors) is interrupted, fails or terminates before completion. According to embodiments of the invention, by storing data elements that have been gathered in destination register 415, the data previously collected by an interrupted or stopped gather operation may be preserved and the gather operation may restart in the middle. The interrupted gather operation (e.g., having gathered one or more data elements) may start from the middle, for example, gathering the remaining elements missing from destination register 415.”, The state of mask register 410 at the time an operation is interrupted corresponds to the claimed previous bitmap because the mask identifies the elements that remain to be processed. As subsequent elements are successfully processed, the corresponding mask-state elements are updated, thereby producing an updated mask state based on the previous mask state. Accordingly, Hall teaches generating a new bitmap based on a previous bitmap.)
and identifying the nonzero valued activation in the compressed activation operand or the nonzero valued weight in the compressed weight operand based on the new bitmap. (Hall, paragraph 48, “Storing data gathered from cache memory 416 and/or external memory 435 in destination register 415 may preserve the data, for example, in case the complete gather operation (e.g., gathering all 16 vectors) is interrupted, fails or terminates before completion. According to embodiments of the invention, by storing data elements that have been gathered in destination register 415, the data previously collected by an interrupted or stopped gather operation may be preserved and the gather operation may restart in the middle. The interrupted gather operation (e.g., having gathered one or more data elements) may start from the middle, for example, gathering the remaining elements missing from destination register 415.”, Hall teaches that after interruption the operation resumes from the preserved state and proceeds with the remaining data elements indicated by the updated mask register. See paragraphs 48-55. Hall therefore teaches using the updated mask state to determine which corresponding data elements remain to be selected and processed. When Hall’s iterative mask-state technique is applied to the bitmap-driven sparsity selection of Raha, the updated bitmap identifies the remaining activation and/or weight entries to be processed. Accordingly, Hall teaches identifying an activation or weight based on the newly generated bitmap.)
The rationale to combine Hall with Rahad/Moyer/Mody is similar to that as applied for claim 2 above.
Claim 22: Raha, Moyer, and Mody teaches the limitations of claim 1. Hall in the same field bit mask analysis, teaches the following limitations which Raha Moyer, and Mody fail to teach:
The one or more non-transitory computer-readable media of claim 21, wherein generating the new bitmap based on the previous bitmap comprises: replacing a nonzero element in the previous bitmap with zero. (Hall, paragraph 55, “In one embodiment, in each operation cycle, the “gather_step” instruction may read a different one of the cache lines from (e.g., L1) cache memory 416 and may fill a maximum number of elements in destination register 415 corresponding to the cache line read. For example, if the cache line read has one element to be gathered, then one element may be written into destination register 415 and the corresponding one bit state elements in mask register 410 may be set to “0”. In some embodiments, when dual or multiple ported cache memory 416 and/or external memory 435 are used, a processor may scatter and/or gather more than one data element per cycle, in which case the scatter and/or gather operation for scattering and/or gathering a predetermined set of data elements may execute in fewer cycles or iterations. It will be also appreciated that by permitting access to different cache banks in multiple data cache lines, as is described in greater detail below with regard to FIGS. 7, 9 and 10, a processor may also scatter and/or gather more than one data element per cycle without requiring a dual or multiple ported cache memory 416.”, Hall teaches that when an element associated with an asserted mask bit is successfully gathered, the corresponding one-bit state element in the mask register “may be set to ‘0’.” Hall therefore expressly teaches updating a binary mask by clearing an asserted, nonzero mask element to zero after the corresponding data element has been processed. The updated mask constitutes the claimed new bitmap, and the clearing of the asserted mask bit teaches replacing a nonzero element in the previous bitmap with zero.)
The rationale to combine Hall with Rahad/Moyer/Mody is similar to that as applied for claim 2 above.
Claims 12 and 18 are substantially similar to claim 2, as such a similar analysis applies.
Claims 15 and 20 are substantially similar to claim 7, as such a similar analysis applies.
Claims 9 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Raha in view of Moyer, Mody and Anders et al., (US 20210397414 A1), hereafter referred to as Anders.
Claim 9: Raha, Moyer, and Mody teaches the limitations of claim 1. Anders in the same field bit mask analysis, teaches the following limitations which Raha and Narad fail to teach:
The method of claim 1, wherein identifying the nonzero valued activation in the compressed activation operand comprises: identifying another nonzero valued activation in the compressed activation operand, wherein the another nonzero valued activation is subsequently next to a previously identified nonzero valued activation in the compressed activation operand. (Anders, paragraph 73, “The non-zero values are compressed and kept adjacent to one another in an IF RF 214. ”, Anders explicitly states that nonzero activations are compressed and stored adjacent in IF RF. Thus, selecting another activation next to a previously selected one is directly supported by the adjacency teaching.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Raha and Narad by further incorporating the teachings of Anders to support recovery behavior of identifying “another” nonzero activation that is “subsequently next” to a previously identified nonzero activation in the compressed activation operand. Raha teaches bitmap-guided sparse operand identification for deep-learning MAC operations (e.g., IF/FL sparsity bitmaps combined into find-first logic to select nonzero work and read only the corresponding operands). Narad provides a known mechanism for detecting invalid/erroneous mask states using bit-count logic (“population count” and invalid indication). Anders explicitly teaches the storage property needed for adjacent selection in compressed activations: “the non-zero values are compressed and kept adjacent to one another in an IF RF,” (Anders, paragraph 73) and further ties the bitmap to zero/nonzero positions (“zero and non-zero positions … are represented by a bit in the bitmap”). A motivation of which would have been to reduce fault-recovery overhead and improve robustness by enabling a simple “next-adjacent-nonzero” fallback in the compressed activation stream.
Claim 16 is substantially similar to claim 9, as such a similar analysis applies.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Wang, Y., Zhang, C., Xie, Z., Guo, C., Liu, Y., & Leng, J. (2021, June). Dual-side sparse tensor core. In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) (pp. 1083-1095). IEEE.
Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., ... & Dally, W. J. (2017). SCNN: An accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH computer architecture news, 45(2), 27-40.
Mishra, A., Latorre, J. A., Pool, J., Stosic, D., Stosic, D., Venkatesh, G., ... & Micikevicius, P. (2021). Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08378.
US 2020/0074287 A1
US 2021/0125070 A1
US 2017/0286830 A1
US 11,507,452 B1
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HYUNGJUN B YI whose telephone number is (703)756-4799. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/H.B.Y./Examiner, Art Unit 2146
/DANIEL T PELLETT/Primary Examiner, Art Unit 2121
Prosecution Insights