DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 2024/01/23. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–20 are directed to patent-eligible subject matter under 35 U.S.C. § 101. The analysis is set forth below.
Regarding independent claims 1, 8, and 15
Step 1 -- whether the claim falls within any statutory category. See MPEP 2106.03.
Claim 1 is drawn to a terminal (machine); claim 8 is drawn to a method (process); claim 15 is drawn to an apparatus (machine). Each of these claims falls within one of the four statutory categories. Step 1 is satisfied.
Step 2A Prong One -- whether the claim recites a judicial exception. See MPEP 2106.04, subsection II.
Regarding independent claim 1, the limitations "comparing a number of elements in the compressed data of the neural network model with a first condition, wherein satisfying the first condition comprises being equal to or smaller than the number of registers in the vector register" and "comparing the number of elements in the compressed data of the neural network model with a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register" recite a comparison of two quantities. Such a comparison is a mental process — a concept capable of being performed in the human mind, including evaluation and judgment (see MPEP § 2106.04(a)(2), subsection III) — and a mathematical concept in the nature of a mathematical relationship (see MPEP § 2106.04(a)(2), subsection I). Claim 1 therefore recites an abstract idea.
Independent claim 8 recites corresponding "receiving a determination whether a number of elements … satisfies a first condition …" and "… a second condition …" limitations and recites an abstract idea for the same reason. Independent claim 15 recites corresponding limitations and recites an abstract idea for the same reason.
Step 2A Prong Two -- whether the claim as a whole integrates the recited judicial exception into a practical application, or whether the claim is "directed to" the judicial exception. See MPEP 2106.04(d).
Regarding independent claim 1, beyond the abstract idea identified above, the claim recites the additional elements of "a host unit," "a memory storing a set of instructions," "one or more processors," "compressing data of the neural network model," "identifying structure information of a vector register, wherein the structure information includes a number of registers included in the vector register," "in response to the number of elements satisfying the first condition, associating the compressed data of the neural network model with the vector register to enable loading the compressed data to the vector register," and "adjusting a structure of the vector register in response to the number of elements satisfying the second condition."
These additional elements, considered individually and in combination, integrate the recited judicial exception into a practical application. The claim does not merely apply the comparison on a generic computer or link it to a technological environment; rather, the result of the comparison is used to load the compressed neural-network data into the vector register and to adjust the structure of the vector register — for example, by reducing the number of registers and correspondingly increasing the register length. As described in the specification, this adjustment balances the size of the neural network model against execution efficiency and model accuracy, allowing more bits to be associated with each remaining element as the number of elements falls (specification ¶¶ [0014]–[0017] describing the deficiencies of conventional model compression; ¶¶ [0043]–[0044], [0052]–[0057] describing the improvement). The claim as a whole thus reflects an improvement to the functioning of a computer and to the technical field of executing neural-network models on vector-register hardware (see MPEP § 2106.05(a) and § 2106.04(d)(1)), consistent with the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, and Example 47 thereof. Accordingly, the claim integrates the recited judicial exception into a practical application.
Independent claims 8 and 15 recite corresponding additional elements — including loading the compressed data to the vector register in response to the determination and adjusting the structure of the vector register in response to the determination — and integrate the recited judicial exception into a practical application for the same reasons.
Because the claims integrate the recited judicial exception into a practical application at Step 2A Prong Two, the claims are not directed to a judicial exception, the analysis does not proceed to Step 2B, and claims 1, 8, and 15 are eligible under 35 U.S.C. § 101.
Regarding dependent claims 2–7, 9–14, and 16–20
The dependent claims recite all of the limitations of their respective independent claims and further recite additional technical limitations, including that the compression comprises pruning (claims 2, 9, 16), that the data is a weight matrix (claims 3, 10, 17), that adjusting the structure of the vector register comprises reducing the number of registers by one half (claims 4, 11, 18), a conditional second operation of compression (claims 5, 12), generating or receiving an instruction set based on an association between the compressed data and the vector register (claims 6, 13, 19), and that the vector register is part of a group of vector registers whose elements are executed simultaneously (claims 7, 14, 20). These limitations further specify the technical implementation by which the compressed neural-network data is fit to and loaded into the vector-register hardware and do not remove the integration into a practical application found for the independent claims. Claims 2–7, 9–14, and 16–20 are therefore eligible under 35 U.S.C. § 101 for the same reasons.
Conclusion
Claims 1–20 are patent-eligible under 35 U.S.C. § 101. No rejection under § 101 is made.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 7-11, 14-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Matveev et al. (Matveev), US 11,195,095 B2, in view of Sen et al. (Sen), US 11,663,001 B2, and further in view of Agarwal et al. (Agarwal), US 5,513,366.
Regarding claim 1, Matveev teaches a terminal comprising the following limitations:
"A terminal, comprising: a host unit; and an apparatus reducing a size of a neural network model communicatively coupled to the host unit, the apparatus comprising: a memory storing a set of instructions; and one or more processors configured to execute the set of instruction to cause the apparatus to perform operations comprising:"
"compressing data of the neural network model;"
"identifying structure information of a vector register, wherein the structure information includes a number of registers included in the vector register;"
"in response to the number of elements satisfying the first condition, associating the compressed data of the neural network model with the vector register to enable loading the compressed data to the vector register;"
As to "A terminal, comprising: a host unit; and an apparatus reducing a size of a neural network model communicatively coupled to the host unit, the apparatus comprising: a memory storing a set of instructions; and one or more processors configured to execute the set of instruction to cause the apparatus to perform operations comprising:", Matveev teaches a computing device for accelerating execution of a neural network model that includes a controller/processor and a memory storing executable code, the processor executing the code to perform the recited operations (Matveev, FIG. 1, computing device 1, controller/processor 2, memory 4, executable code 5; claim 1, col. 31, ll. 65–67, "A method of accelerating execution of a neural network (NN) model, by at least one processor, the method comprising").
As to "compressing data of the neural network model;", Matveev teaches producing a group-sparse matrix A′ from a weight matrix A of the neural network model by pruning (Matveev, claim 2, col. 32, ll. 28–34, "producing group-sparse matrix A′ comprises pruning first matrix A, such that the G groups of elements are consecutive along one or more axes of A′"; col. 31, ll. 24–43, describing pruning of the layer weights so that pruned execution fits the processor).
As to "identifying structure information of a vector register, wherein the structure information includes a number of registers included in the vector register;", Matveev teaches receiving a parameter of the vector operation that specifies the structure of the input vector register, including the number of indices of the input vector register(s) and the number of entries per index (Matveev, claim 1, col. 32, ll. 1–4, "receiving at least one parameter of a vector operation, wherein said parameter comprises a number N2>1, representing a number of entries in an index of an input vector register"; claim 2, col. 32, ll. 28–31, "a first number N1>1, representing a number of indices of one or more input vector registers"). Under the broadest reasonable interpretation, the recited "number of registers included in the vector register" reads on the number of indices N1 of the input vector register(s) of Matveev.
As to "in response to the number of elements satisfying the first condition, associating the compressed data of the neural network model with the vector register to enable loading the compressed data to the vector register;", Matveev teaches associating the compressed (group-sparse) data with the input vector register and loading it by storing/broadcasting the elements into the entries of the vector-register indices (Matveev, claim 1, col. 32, ll. 23–27, "storing N2 elements pertaining to an indexed axis of matrix B in respective entries of an index of an input vector register"; claim 4(b), col. 32, ll. 43–46, "broadcasting the N2 elements of the selected group into respective N2 entries of each of the N1 indices of a first input vector register"). The conditional aspect, "in response to the number of elements satisfying the first condition," is supplied by Sen as set forth below.
Matveev teaches subject matter related to comparing the number of elements to the structure of the vector register, in that Matveev defines the dimensions of the group-sparse data by the number of indices and entries of the input vector register (Matveev, claim 1, col. 32, ll. 9–16). However, Matveev does not teach:
"comparing a number of elements in the compressed data of the neural network model with a first condition, wherein satisfying the first condition comprises being equal to or smaller than the number of registers in the vector register;"
"comparing the number of elements in the compressed data of the neural network model with a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register;"
In the same field of endeavor, Sen teaches "comparing a number of elements in the compressed data of the neural network model with a first condition, wherein satisfying the first condition comprises being equal to or smaller than the number of registers in the vector register;" and "comparing the number of elements in the compressed data of the neural network model with a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register;". Sen teaches a lossy sparse load unit that, for a neural network exhibiting weight/activation sparsity, determines how many non-zero values are present in one or more input vector operands and compares that number of non-zero values to a programmable threshold number, causing the operands to be processed/loaded when the count satisfies the threshold and skipped otherwise (Sen, Abstract; col. 1, ll. 14–26, describing sparsity and zero values in DNN weights/activations; col. 2, ll. 46–62, "the lossy sparse load unit determines how many non-zero values are present in one or more input vector operands … responsive to determining that the number of non-zero values … is greater than or equal to a threshold … Otherwise, if the number of non-zero values … is less than the threshold, then the lossy sparse load unit causes processing … to be skipped … the threshold is programmable"; FIG. 8, decision 815 "Number of Non-Zero Values < Threshold?", step 820 prevent, step 825 cause). The number of non-zero values of Sen corresponds to the recited "number of elements in the compressed data," and Sen's programmable threshold number corresponds to the recited first and second conditions; setting that programmable threshold to the number of registers (first condition) and to one half the number of registers (second condition) would have been obvious as a matter of design, particularly because the vector-register configurations are organized in powers of two such that the next-lower level is one half (see Agarwal below; applicant's own disclosure at spec ¶ [0060], reciting register levels of 16, 32, or 64).
Matveev and Sen are analogous to the claimed invention as both are from the same field of endeavor of executing neural network models on processors using vector/SIMD registers. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the loading of pruned group-sparse neural-network weights into the vector register of Matveev with the count-versus-threshold comparison and conditional load/skip of Sen. The motivation to combine Matveev and Sen is as recited by Sen (col. 1, ll. 14–26; col. 2, ll. 46–62), namely to avoid issuing operations on input vector operands having too few non-zero values, thereby improving processor performance and reducing power consumption when executing a sparse neural network.
The combination of Matveev and Sen, however, does not teach:
"adjusting a structure of the vector register in response to the number of elements satisfying the second condition."
In the same field of endeavor, Agarwal teaches "adjusting a structure of the vector register in response to the number of elements satisfying the second condition." Agarwal teaches a dynamically reconfigurable vector register file in which a size register receives a vector size parameter that specifies the number of registers comprising a vector register, and in which, in response to that parameter, the structure of the vector register — the number of registers and the register length — is reconfigured at a fixed array capacity (Agarwal, Abstract; col. 2, "A size register … receives a selected vector size parameter, which specifies a number of registers comprising a vector register. In response to the vector size parameter, columns … are selected and concatenated to form a vector register … Vector register lengths, and the number of vector registers, may be dynamically configured by setting the vector size parameter"; col. 11–12, describing the "vector register stride" mechanism that "allows trading off a number of vector registers with vector size," FIG. 7). Reducing the number of registers to the next-lower level (one half) and correspondingly increasing the register length is taught by Agarwal's trade-off of the number of vector registers against vector size.
Matveev, Sen, and Agarwal are analogous to the claimed invention. Matveev and Sen are in the same field of endeavor of executing neural networks using vector/SIMD registers, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned — namely, fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the reconfigurable vector-register structure of Agarwal with the system of Matveev and Sen. The motivation to combine Matveev, Sen, and Agarwal is as recited by Agarwal (col. 11–12), namely to trade off the number of vector registers against the register size so as to match the vector-register structure to the number of data elements the application requires, which yields the predictable benefit of using fewer, longer registers to associate more bits with each remaining element once the number of elements has fallen.
Regarding claim 2, claim 2 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected for the same reasons set forth above with respect to claim 1.
Claim 2 further recites the following limitation:
"The terminal of claim 1, wherein compressing the data of the neural network model comprises pruning of the data."
As to "wherein compressing the data of the neural network model comprises pruning of the data," Matveev teaches that the compression of the neural-network data is performed by pruning, in that Matveev produces the group-sparse matrix A′ from the weight matrix A of the neural network model by pruning matrix A (Matveev, claim 2, col. 32, ll. 28–34, "producing group-sparse matrix A′ comprises pruning first matrix A, such that the G groups of elements are consecutive along one or more axes of A′"; col. 31, ll. 24–43, describing that the weights of the layer "may be pruned" — for example, "pruned to 90%" — so that the pruned model is executed efficiently on the processor). The pruning of matrix A by Matveev corresponds to the recited "pruning of the data."
Regarding claim 3, claim 3 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected for the same reasons set forth above with respect to claim 1.
Claim 3 further recites the following limitation:
"The terminal of claim 1, wherein the data is a weight matrix of the neural network model."
As to "wherein the data is a weight matrix of the neural network model," Matveev teaches that the data of the neural network model is a weight matrix, in that Matveev receives a first matrix A representing elements of a kernel K of the neural network model and produces the group-sparse matrix A′ from that weight matrix by pruning (Matveev, claim 1, col. 32, ll. 5–6, "receiving a first matrix A, representing elements of a kernel K of the NN model"; claim 2, col. 32, ll. 28–34, "producing group-sparse matrix A′ comprises pruning first matrix A"; col. 31, ll. 24–43, describing the weight values of the layer arranged as a matrix of shape [C_out, C_in, Kz, Ky, Kx]). The matrix A of weight values of kernel K in Matveev corresponds to the recited "weight matrix of the neural network model."
Regarding claim 4, claim 4 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected under the same rationale as set forth above with respect to claim 1.
Claim 4 further recites the following limitation:
"The terminal of claim 1, wherein adjusting the structure of the vector register comprises: reducing the number of registers in the vector register by one half."
The combination of Matveev and Sen, however, does not teach "wherein adjusting the structure of the vector register comprises: reducing the number of registers in the vector register by one half."
In the same field of endeavor, Agarwal teaches "wherein adjusting the structure of the vector register comprises: reducing the number of registers in the vector register by one half." Agarwal teaches a dynamically reconfigurable register file in which a size register receives a vector size parameter specifying the number of registers comprising a vector register, and in which the structure of the vector register is reconfigured by trading off the number of vector registers against the vector size at a fixed array capacity, such that the number of registers making up the vector register is reduced (Agarwal, Abstract; col. 2, "A size register … receives a selected vector size parameter, which specifies a number of registers comprising a vector register … Vector register lengths, and the number of vector registers, may be dynamically configured by setting the vector size parameter"; col. 11–12, describing the "vector register stride" mechanism that "allows trading off a number of vector registers with vector size," whereby fewer vector registers of larger size are provided, FIG. 7). Because Agarwal's vector-register configurations are organized in discrete levels in which the next-lower level is one half of the current level, reducing the number of registers in the vector register by one half is taught by, or would have been an obvious configuration of, Agarwal's trade-off of the number of vector registers against vector size, consistent with applicant's own disclosure that the register levels are powers of two such that "the next level of fewer number of registers is one half" (spec ¶ [0060]).
Matveev, Sen, and Agarwal are analogous to the claimed invention. Matveev and Sen are from the same field of endeavor of executing neural networks using vector/SIMD registers, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned — namely, fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the reconfigurable vector-register structure of Agarwal with the system of Matveev and Sen. The motivation to combine Matveev, Sen, and Agarwal is as recited by Agarwal (col. 11–12), namely to trade off the number of vector registers against the register size so as to match the vector-register structure to the number of data elements the application requires, which yields the predictable benefit of using fewer, longer registers to associate more bits with each remaining element once the number of elements has fallen.
Regarding claim 7, claim 7 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected under the same rationale as set forth above with respect to claim 1.
Claim 7 further recites the following limitation:
"The terminal of claim 1, wherein the vector register is part of a group of vector registers that include elements that are executed simultaneously."
As to "wherein the vector register is part of a group of vector registers that include elements that are executed simultaneously," Matveev teaches that the input vector register is one of a group of vector registers whose elements are operated on together by a single SIMD vector operation, that is, executed simultaneously (Matveev, Abstract, "The number of elements in each tensor is defined by, or equal to a number of entries in each index of an input tensor register used for a specific Single Instruction Multiple Data (SIMD) tensor operation"; claim 1, col. 32, ll. 17–22, "executing kernel K on input I, by performing at least one computation of the vector operation, the vector operation having as operands elements of a group of the G groups and corresponding elements of the B matrix"; claim 4, col. 32, ll. 41–47, "broadcasting the N2 elements of the selected group into respective N2 entries of each of the N1 indices of a first input vector register; … populating N1*N2 entries of a second input vector register … accumulating the result of said multiplications in corresponding N1 indices of a first output vector register"). The plurality of input and output vector registers operated upon together in the single SIMD vector operation of Matveev, the elements of which are processed simultaneously, corresponds to the recited "group of vector registers that include elements that are executed simultaneously."
Claim 8 is an independent method claim reciting limitations corresponding in substance to those of claim 1. To the extent the limitations of claim 8 correspond to the limitations of claim 1, they are rejected based on the same rationale as for claim 1. The remaining limitations are addressed below.
Regarding claim 8, Matveev teaches a method comprising:
"A method for executing a neural network model by an accelerator, comprising:"
Matveev teaches a method of accelerating execution of a neural network model by at least one processor (Matveev, claim 1, col. 31, ll. 65–67, "A method of accelerating execution of a neural network (NN) model, by at least one processor, the method comprising"; FIG. 1, controller/processor 2).
"receiving compressed data of the neural network model;"
Matveev teaches obtaining, for execution, the group-sparse matrix A′ produced by pruning the weight matrix A of the neural network model, the group-sparse matrix A′ being the compressed data (Matveev, claim 1, col. 32, ll. 9–16, "producing from first matrix A, a group-sparse matrix A′, comprising G groups of elements … wherein all elements of A′ outside said G groups are null"; claim 2, col. 32, ll. 28–34, "producing group-sparse matrix A′ comprises pruning first matrix A").
"in response to the determination that the number of elements satisfies the first condition, loading the compressed data to the vector register;"
Matveev teaches loading the compressed data into the vector register by storing and broadcasting the elements into the entries of the vector-register indices (Matveev, claim 1, col. 32, ll. 23–27, "storing N2 elements pertaining to an indexed axis of matrix B in respective entries of an index of an input vector register"; claim 4(b), col. 32, ll. 43–46, "broadcasting the N2 elements of the selected group into respective N2 entries of each of the N1 indices of a first input vector register"). The conditional aspect, "in response to the determination that the number of elements satisfies the first condition," is supplied by Sen as set forth below.
Matveev teaches subject matter related to a determination as to the number of elements relative to the structure of the vector register, in that Matveev defines the dimensions of the group-sparse data by the number of indices and entries of the input vector register (Matveev, claim 1, col. 32, ll. 9–16). However, Matveev does not teach the following limitations, addressed sub-limitation by sub-limitation:
"receiving a determination whether a number of elements in the compressed data of the neural network model satisfies a first condition, wherein satisfying the first condition comprises being equal to or smaller than a number of registers in a vector register from a host;"
"receiving a determination whether the number of elements in the compressed data of the neural network model satisfies a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register;"
In the same field of endeavor, Sen teaches these limitations as follows.
As to "receiving a determination whether a number of elements in the compressed data of the neural network model satisfies a first condition," Sen teaches that a lossy sparse load unit determines how many non-zero values are present in one or more input vector operands and that this determination is received by the processing/loading stage to govern whether the operands are processed (Sen, col. 2, ll. 46–62, "the lossy sparse load unit determines how many non-zero values are present in one or more input vector operands … responsive to determining that the number of non-zero values … is greater than or equal to a threshold"; FIG. 8, step 810 and decision 815). The number of non-zero values of Sen corresponds to the recited "number of elements in the compressed data."
As to "wherein satisfying the first condition comprises being equal to or smaller than a number of registers in a vector register from a host," Sen teaches comparing the number of non-zero values to a programmable threshold number, where the determination is performed and provided from the host processor (Sen, Abstract; col. 2, ll. 46–62, "the threshold is programmable"; FIG. 8, decision 815 "Number of Non-Zero Values < Threshold?"). Sen's programmable threshold number corresponds to the recited first condition; setting that programmable threshold to the number of registers in the vector register would have been obvious as a matter of design, and the "from a host" recitation reads on the determination being performed by the host processor of the combination and provided to the accelerator that performs the loading.
As to "receiving a determination whether the number of elements in the compressed data of the neural network model satisfies a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register," Sen teaches the same count-versus-threshold determination applied with a further programmable threshold (Sen, col. 2, ll. 46–62; FIG. 8). Sen's programmable threshold number corresponds to the recited second condition; setting that programmable threshold to one half of the number of registers would have been obvious as a matter of design, particularly because the vector-register configurations are organized in powers of two such that the next-lower level is one half (see Agarwal below; applicant's own disclosure at spec ¶ [0060]).
Matveev and Sen are analogous to the claimed invention as both are from the same field of endeavor of executing neural network models on processors using vector/SIMD registers. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the loading of pruned group-sparse neural-network data into the vector register of Matveev with the count-versus-threshold determination of Sen. The motivation to combine Matveev and Sen is as recited by Sen (col. 1, ll. 14–26; col. 2, ll. 46–62), namely to avoid issuing operations on input vector operands having too few non-zero values, thereby improving processor performance and reducing power consumption when executing a sparse neural network.
The combination of Matveev and Sen, however, does not teach "in response to the determination that the number of elements satisfies the second condition, adjusting a structure of the vector register."
In the same field of endeavor, Agarwal teaches "in response to the determination that the number of elements satisfies the second condition, adjusting a structure of the vector register." Agarwal teaches a dynamically reconfigurable register file in which a size register receives a vector size parameter specifying the number of registers comprising a vector register, and in which the structure of the vector register is reconfigured by trading off the number of vector registers against the vector size at a fixed array capacity (Agarwal, Abstract; col. 2, "A size register … receives a selected vector size parameter, which specifies a number of registers comprising a vector register … Vector register lengths, and the number of vector registers, may be dynamically configured by setting the vector size parameter"; col. 11–12, the "vector register stride" mechanism that "allows trading off a number of vector registers with vector size," FIG. 7). Performing this reconfiguration when the number of elements satisfies the second condition corresponds to the recited limitation.
Matveev, Sen, and Agarwal are analogous to the claimed invention. Matveev and Sen are from the same field of endeavor of executing neural networks using vector/SIMD registers, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned, namely fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the reconfigurable vector-register structure of Agarwal with the method of Matveev and Sen. The motivation to combine Matveev, Sen, and Agarwal is as recited by Agarwal (col. 11–12), namely to trade off the number of vector registers against the register size so as to match the vector-register structure to the number of data elements the application requires, which yields the predictable benefit of using fewer, longer registers to associate more bits with each remaining element once the number of elements has fallen.
Claim 9 is rejected based on the same rationale as for claim 2.
Claim 10 is rejected based on the same rationale as for claim 3.
Claim 11 is rejected based on the same rationale as for claim 4.
Claim 14 is rejected based on the same rationale as for claim 7.
Claim 15 is an independent apparatus claim reciting operative limitations corresponding in substance to those of claim 8. To the extent the operative limitations of claim 15 correspond to the limitations of claim 8, they are rejected based on the same rationale as for claim 8. The preamble and the corresponding mapping are set forth below in accordance with the required format.
Regarding claim 15, Matveev teaches an apparatus comprising:
"An apparatus for reducing a size of a neural network model, comprising: a memory storing a set of instructions, and one or more processors configured to execute the set of instruction to cause the apparatus to perform operations comprising:"
Matveev teaches an apparatus for reducing the size of a neural network model that includes a memory storing executable code and one or more processors that execute the code to perform the recited operations (Matveev, FIG. 1, computing device 1, controller/processor 2, memory 4, executable code 5; claim 1, col. 31, ll. 65–67, "A method of accelerating execution of a neural network (NN) model, by at least one processor"; col. 31, ll. 24–43, describing pruning of the layer weights to reduce the model so that it fits the processor).
"receiving compressed data of the neural network model;"
Matveev teaches obtaining, for execution, the group-sparse matrix A′ produced by pruning the weight matrix A of the neural network model, the group-sparse matrix A′ being the compressed data (Matveev, claim 1, col. 32, ll. 9–16, "producing from first matrix A, a group-sparse matrix A′, comprising G groups of elements … wherein all elements of A′ outside said G groups are null"; claim 2, col. 32, ll. 28–34, "producing group-sparse matrix A′ comprises pruning first matrix A").
"in response to the determination that the number of elements satisfies the first condition, loading the compressed data to the vector register;"
Matveev teaches loading the compressed data into the vector register by storing and broadcasting the elements into the entries of the vector-register indices (Matveev, claim 1, col. 32, ll. 23–27, "storing N2 elements pertaining to an indexed axis of matrix B in respective entries of an index of an input vector register"; claim 4(b), col. 32, ll. 43–46, "broadcasting the N2 elements of the selected group into respective N2 entries of each of the N1 indices of a first input vector register"). The conditional aspect, "in response to the determination that the number of elements satisfies the first condition," is supplied by Sen as set forth below.
Matveev teaches subject matter related to a determination as to the number of elements relative to the structure of the vector register, in that Matveev defines the dimensions of the group-sparse data by the number of indices and entries of the input vector register (Matveev, claim 1, col. 32, ll. 9–16). However, Matveev does not teach the following limitations, addressed sub-limitation by sub-limitation:
"receiving a determination whether a number of elements in the compressed data of the neural network model satisfies a first condition, wherein satisfying the first condition comprises being equal to or smaller than a number of registers in a vector register from a host;"
"receiving a determination whether the number of elements in the compressed data of the neural network model satisfies a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register;"
In the same field of endeavor, Sen teaches these limitations as follows.
As to "receiving a determination whether a number of elements in the compressed data of the neural network model satisfies a first condition," Sen teaches that a lossy sparse load unit determines how many non-zero values are present in one or more input vector operands and that this determination is received by the processing/loading stage to govern whether the operands are processed (Sen, col. 2, ll. 46–62, "the lossy sparse load unit determines how many non-zero values are present in one or more input vector operands … responsive to determining that the number of non-zero values … is greater than or equal to a threshold"; FIG. 8, step 810 and decision 815). The number of non-zero values of Sen corresponds to the recited "number of elements in the compressed data."
As to "wherein satisfying the first condition comprises being equal to or smaller than a number of registers in a vector register from a host," Sen teaches comparing the number of non-zero values to a programmable threshold number, where the determination is performed and provided from the host processor (Sen, Abstract; col. 2, ll. 46–62, "the threshold is programmable"; FIG. 8, decision 815 "Number of Non-Zero Values < Threshold?"). Sen's programmable threshold number corresponds to the recited first condition; setting that programmable threshold to the number of registers in the vector register would have been obvious as a matter of design, and the "from a host" recitation reads on the determination being performed by the host processor of the combination and provided to the apparatus that performs the loading.
As to "receiving a determination whether the number of elements in the compressed data of the neural network model satisfies a second condition, wherein satisfying the second condition comprises being equal to or smaller than one half of the number of registers in the vector register," Sen teaches the same count-versus-threshold determination applied with a further programmable threshold (Sen, col. 2, ll. 46–62; FIG. 8). Sen's programmable threshold number corresponds to the recited second condition; setting that programmable threshold to one half of the number of registers would have been obvious as a matter of design, particularly because the vector-register configurations are organized in powers of two such that the next-lower level is one half (see Agarwal below; applicant's own disclosure at spec ¶ [0060]).
Matveev and Sen are analogous to the claimed invention as both are from the same field of endeavor of executing neural network models on processors using vector/SIMD registers. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the loading of pruned group-sparse neural-network data into the vector register of Matveev with the count-versus-threshold determination of Sen. The motivation to combine Matveev and Sen is as recited by Sen (col. 1, ll. 14–26; col. 2, ll. 46–62), namely to avoid issuing operations on input vector operands having too few non-zero values, thereby improving processor performance and reducing power consumption when executing a sparse neural network.
The combination of Matveev and Sen, however, does not teach "in response to the determination that the number of elements satisfies the second condition, adjusting a structure of the vector register."
In the same field of endeavor, Agarwal teaches "in response to the determination that the number of elements satisfies the second condition, adjusting a structure of the vector register." Agarwal teaches a dynamically reconfigurable register file in which a size register receives a vector size parameter specifying the number of registers comprising a vector register, and in which the structure of the vector register is reconfigured by trading off the number of vector registers against the vector size at a fixed array capacity (Agarwal, Abstract; col. 2, "A size register … receives a selected vector size parameter, which specifies a number of registers comprising a vector register … Vector register lengths, and the number of vector registers, may be dynamically configured by setting the vector size parameter"; col. 11–12, the "vector register stride" mechanism that "allows trading off a number of vector registers with vector size," FIG. 7). Performing this reconfiguration when the number of elements satisfies the second condition corresponds to the recited limitation.
Matveev, Sen, and Agarwal are analogous to the claimed invention. Matveev and Sen are from the same field of endeavor of executing neural networks using vector/SIMD registers, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned, namely fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to further combine the reconfigurable vector-register structure of Agarwal with the apparatus of Matveev and Sen. The motivation to combine Matveev, Sen, and Agarwal is as recited by Agarwal (col. 11–12), namely to trade off the number of vector registers against the register size so as to match the vector-register structure to the number of data elements the application requires, which yields the predictable benefit of using fewer, longer registers to associate more bits with each remaining element once the number of elements has fallen.
Claim 16 is rejected based on the same rationale as for claim 2.
Claim 17 is rejected based on the same rationale as for claim 3.
Claim 18 is rejected based on the same rationale as for claim 4.
Claim 20 is rejected based on the same rationale as for claim 7.
Claims 5 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Matveev et al. (Matveev), US 11,195,095 B2, in view of Sen et al. (Sen), US 11,663,001 B2, in view of Agarwal et al. (Agarwal), US 5,513,366, and further in view of Litvak et al. (Litvak), US 2020/0401895 A1.
Regarding claim 5, claim 5 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected under the same rationale as set forth above with respect to claim 1.
Claim 5 further recites the following limitation:
"The terminal of claim 1, wherein compressing the data of the neural network model comprises a first operation of compression, and the one or more processors are configured to execute the set of instruction to cause the apparatus to further perform a second operation of compression of the data in response to the number of elements not satisfying the first condition."
The combination of Matveev, Sen, and Agarwal, however, does not teach the above limitation.
In the same field of endeavor, Litvak teaches the above limitation, as follows.
As to "wherein compressing the data of the neural network model comprises a first operation of compression," Litvak teaches a first iteration of pruning performed on the weights of a layer of the neural network, in which the layer is pruned by setting weights to zero (Litvak, FIG. 13, "First Iteration 1300"; ¶ [0115], "using the same example filter as in first iteration 1300, the weights selected for processing are different"; ¶ [0120], "At step 1410, the system may prune the layer based at least in part on the target ratio of zero weights by setting one or more weights of the layer to zero"). Litvak's first pruning iteration corresponds to the recited "first operation of compression."
As to "the one or more processors are configured to execute the set of instruction to cause the apparatus to further perform a second operation of compression of the data," Litvak teaches that a processor executing a set of codes performs a further, second iteration of pruning on the weights (Litvak, ¶ [0117], "these operations may be performed by a processor executing a set of codes to control functional elements of an apparatus"; ¶ [0114], "the second iteration 1320"; FIG. 13, "Second Iteration 1320"; ¶ [0122], "At step 1420, the system may reset at least one weight within each sequence of zero weights that violates the LZS condition to a non-zero value"). Litvak's second pruning iteration, executed by the processor, corresponds to the recited "second operation of compression of the data."
As to "in response to the number of elements not satisfying the first condition," Litvak teaches that the further pruning is performed in response to a determination that the result of the prior pruning does not satisfy a condition based on the number of weights, namely the limited zero sequence (LZS) condition, which is based on the number of weights the zero-skipping circuit is configured to evaluate and skip in a single cycle (Litvak, ¶ [0121], "At step 1415, the system may determine whether each sequence of zero weights in the layer violates the LZS condition after the pruning"; Abstract, "The LZS condition may be based on a number of weights that a zero-skipping circuit of a computing system for processing the ANN is configured to evaluate and skip in a single cycle"; ¶ [0122], the reset of step 1420 being performed "within each sequence of zero weights that violates the LZS condition"). Litvak's performance of the further pruning in response to the pruned result violating the LZS condition — a condition based on the number of weights — corresponds to the recited "in response to the number of elements not satisfying the first condition," where the antecedent "first condition" is as established by Sen in the rejection of claim 1.
Matveev, Sen, Agarwal, and Litvak are analogous to the claimed invention. Matveev, Sen, and Litvak are from the same field of endeavor of executing pruned/sparse neural networks on hardware exploiting vector/SIMD parallelism, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned, namely fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the iterative pruning of Litvak, conditioned on a prior pruning result not satisfying a condition on the number of elements, with the system of Matveev, Sen, and Agarwal. The motivation to combine Matveev, Sen, Agarwal, and Litvak is as recited by Litvak (Abstract; ¶ [0132]), namely to perform structured pruning with consideration of the hardware limitations of the underlying zero-skipping/SIMD circuit so that the resulting pruned neural network can be processed efficiently and the computing system can avoid processing zero weights, thereby speeding up neural-network processing and reducing the power required for processing.
Claim 12 is rejected based on the same rationale as for claim 5.
Claims 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Matveev et al. (Matveev), US 11,195,095 B2, in view of Sen et al. (Sen), US 11,663,001 B2, in view of Agarwal et al. (Agarwal), US 5,513,366, and further in view of Azizi et al. (Azizi), US 10,929,503 B2.
Regarding claim 6, claim 6 depends from claim 1 and recites all of the limitations of claim 1. Those limitations are rejected under the same rationale as set forth above with respect to claim 1.
Claim 6 further recites the following limitation:
"The terminal of claim 1, wherein the operations further comprise: generating an instruction set based on an association between the compressed data and the vector register to load the compressed data to the vector register."
The combination of Matveev, Sen, and Agarwal, however, does not teach the above limitation.
In the same field of endeavor, Azizi teaches the above limitation. Azizi teaches generating, for neural network pruning operations, a matrix-multiplication-with-masking (GEMM) instruction that is decoded and that identifies the registers holding the matrix data and a matrix mask, the matrix mask reflecting the pruned data, and that loads the matrix data into the identified registers for execution (Azizi, title, "Apparatus and Method for a Masked Multiply Instruction to Support Neural Network Pruning Operations"; Abstract, "a decoder to decode a matrix multiplication with masking (GEMM) instruction identifying a destination matrix register to store a result, and source registers storing an A-matrix, a B-matrix, and a matrix mask; execution circuitry to execute the GEMM instruction, the execution circuitry to multiply a plurality of B-matrix elements with a plurality of A-matrix elements, each of the B-matrix elements associated with a mask value in the matrix mask"). Azizi's generation of the GEMM-with-masking instruction, in which the matrix mask establishes the association between the masked (pruned/compressed) matrix data and the matrix registers identified by the instruction, and which loads the matrix data into those registers, corresponds to the recited "generating an instruction set based on an association between the compressed data and the vector register to load the compressed data to the vector register."
Matveev, Sen, Agarwal, and Azizi are analogous to the claimed invention. Matveev, Sen, and Azizi are from the same field of endeavor of executing pruned/sparse neural networks on hardware using matrix/vector registers, and Agarwal is reasonably pertinent to the particular problem with which the inventor was concerned, namely fitting a varying number of data elements into a vector register by adjusting the register structure. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the generation of a masking instruction that associates the pruned matrix data with the matrix registers and loads that data into the registers, as taught by Azizi, with the system of Matveev, Sen, and Agarwal. The motivation to combine Matveev, Sen, Agarwal, and Azizi is as recited by Azizi (title; Abstract), namely to support neural network pruning operations by generating an instruction that uses a matrix mask to associate the pruned matrix data with the registers, thereby enabling the pruned data to be loaded and processed efficiently while skipping masked-out elements.
Claim 13 is rejected based on the same rationale as for claim 6.
Claim 19 is rejected based on the same rationale as for claim 6.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HUNG VAN LE whose telephone number is (571)270-0164. The examiner can normally be reached 8 a.m. - 5 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached at (571) 272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HUNG VAN LE/Examiner, Art Unit 2145
/CHAU T NGUYEN/Primary Examiner, Art Unit 2145
Prosecution Insights