DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
Claim 25 limitations;
means for receiving inputs
means for defining a first parameter ... and the second parameter
means for adapting the inputs ... and a second set of compute engine parameters
means for generating an approximation of an AI workload
Claim 27 limitations;
means for truncating
Claim 28 limitations;
means for adaptively selecting
Claim 29 limitations;
means for rounding
Claim 20 limitations;
means for defining
invoke 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, and the written description discloses the corresponding structure, material, or acts for performing the entire claimed function and clearly link the structure, material, or acts to the function.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
According to the first part of the analysis, in the instant case, claims 1-8 are directed to an apparatus, claims 9-16 are directed to a processor, claims 17-24 are directed to a non-transitory computer-readable medium, and claims 25-30 are directed to an apparatus. Each of these claims fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
For claim 1,
Step 2A Prong One
define a first parameter for the inputs and a second parameter for the AI compute engine, the first parameter indicating a first portion of the first precision to use for computations by the AI compute engine and the second parameter indicating a second portion of the second precision to use for computation by the AI compute engine;
(This step for defining parameters for a computation is a mathematical concept)
Step 2A Prong Two
An apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to:
(This step for performing the methods of the disclosure using a generic computer is mere-instructions to apply an exception. See MPEP § 2106.05(f))
receive inputs to be processed using an artificial intelligence (AI) compute engine, the inputs having a first precision and the AI compute engine being configured for processing in a second precision that is different from the first precision; adapt the inputs according to the first parameter to generate a first representation of the inputs and a second set of compute engine parameters according to the second parameter to generate a second representation of the second set of compute engine parameters; and generate an approximation of an AI workload corresponding to the inputs based on the first representation and the second representation.
(These steps for receiving and defining inputs, preprocessing data, and generating output are considered insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are insignificant extra-solution activity and mere-instructions to apply an exception recited at a high level of generality.
For claim 2,
Step 2A Prong One
(Claim 2 depends on claim 1, which has been determined to recite abstract ideas including mathematical concepts. Therefore, claim 2 also recites an abstract idea.)
Step 2A Prong Two
The apparatus of claim 1, in which the first precision is greater than the second precision.
(This step for defining inputs is considered insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are insignificant extra-solution activity recited at a high level of generality.
For claim 3,
Step 2A Prong One
(Claim 3 depends on claim 1, which has been determined to recite abstract ideas including mathematical concepts. Therefore, claim 3 also recites an abstract idea.)
Step 2A Prong Two
in which the first parameter specifies a first number of bits of the inputs having the first precision to use for computations by the AI compute engine and the second parameter specifies a second number of bits of the compute engine parameters having the second precision to use for computations by the AI compute engine.
(This step for selecting precisions for inputs and compute engine parameters based on the defined parameters is insignificant extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are insignificant extra-solution activity recited at a high level of generality.
For claim 4,
Step 2A Prong One
(Claim 4 depends on claim 1, which has been determined to recite abstract ideas including mathematical concepts. Therefore, claim 4 also recites an abstract idea.)
Step 2A Prong Two
in which the at least one processor is further configured to
(This step for using a generic computer is mere instructions to apply an exception. See MPEP § 2106.05(f))
truncate the inputs based on the first parameter to generate the first representation.
(This step for truncating the inputs based on the defined parameters is insignificant-extra-solution activity. See MPEP § 2106.05(g))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are mere instructions to apply an exception and insignificant extra-solution activity recited at a high level of generality.
For claim 5,
Step 2A Prong One
adaptively select one of most significant bits or least significant bits for each of the first representation and the second representation based on the first parameter and the second parameter, respectively.
(This step for adaptively selecting bits based on the defined parameters is a mental process)
Step 2A Prong Two
which the at least one processor is further configured to
(This step for using a generic computer is mere instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts (from claim 1) and mental processes while the additional elements are mere instructions to apply a judicial exception recited at a high level of generality.
For claim 6,
Step 2A Prong One
round one or more of the inputs based on the first parameter or the compute engine parameters based on the second parameter.
(This step for rounding inputs is a mathematical concept)
Step 2A Prong Two
in which the at least one processor is further configured to
(This step for using a generic computer is mere instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are mere instructions to apply a judicial exception recited at a high level of generality.
For claim 7,
Step 2A Prong One
(Claim 7 depends on claim 1, which has been determined to recite abstract ideas including mathematical concepts. Therefore, claim 7 also recites an abstract idea.)
Step 2A Prong Two
in which the compute engine parameters comprise one or more of weights or activations.
(This step for defining the compute engine parameters to weights or activations is a field of use and technological environment limitation. See MPEP § 2106.05(h))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are field of use and technological environment limitations recited at a high level of generality.
For claim 8,
Step 2A Prong One
define a third parameter defining a third number of times a summation operation is repeated to generate the approximation.
(This step for defining a parameter indicating a number of times a summation operation is repeated is a mathematical concept)
Step 2A Prong Two
in which the at least one processor is further configured to
(This step performing on a generic computer is mere-instructions to apply an exception. See MPEP § 2106.05(f))
Step 2B
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception because, when considered individually and in combination, they do not add significantly more (also known as an inventive concept) to the exception. The claim recites mathematical concepts while the additional elements are mere-instructions to apply an exception recited at a high level of generality.
For claims 9-16,
Claims 9-16 are processor implemented method claims that are substantially similar to apparatus claims 1-8, and are rejected using the same reasoning.
For claims 17-24,
Claims 17-24 are non-transitory computer-readable medium claims that are substantially similar to apparatus claims 1-8, and are rejected using the same reasoning.
For claims 25-30,
Claims 25-30 are apparatus claims that are substantially similar to apparatus claims 1, 3-6, 8, respectively, and are rejected using the same reasoning.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7-13, 15-21, 23-28, 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Avishaii Abuhatzera et al. (hereinafter Abuhatzera) (US 20200320375 A1, 2020-10-08) in view of Gregory Henry et al. (hereinafter Henry) (US 20190042244 A1, 2019-02-07) further in view of Tung Thanh Hoang et al. (hereinafter Hoang) (US 20210397974 A1, 2021-12-23).
Regarding claim 1, Abuhatzera teaches;
An apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to:
PNG
media_image1.png
600
884
media_image1.png
Greyscale
receive inputs to be processed using an artificial intelligence (AI) compute engine, the inputs having a first precision and the AI compute engine being configured for processing in a second precision that is different from the first precision;
([Abstract] An apparatus to facilitate accelerating neural networks with low precision-based multiplication and exploiting sparsity in higher order bits is disclosed. The apparatus includes a processor comprising a re-encoder to re-encode a first input number of signed input numbers represented in a first precision format as part of a machine learning model, the first input number re-encoded into two signed input numbers of a second precision format, wherein the first precision format is a higher precision format than the second precision format.)
NOTE: Abuhatzera teaches receiving machine learning inputs of a first precision format and re-encoding the inputs into a lower, second precision for processing by low-precision neural network acceleration hardware.
Thus, Abuhatzera teaches receiving inputs to be processed using an artificial intelligence (AI) compute engine (the low-precision neural network acceleration hardware), the inputs having a first precision and the AI compute engine being configured for processing in a second precision that is different from the first precision.
Define a first parameter for the inputs and a second parameter for the AI compute engine, the first parameter indicating a first portion of the first precision to use for computations by the AI compute engine and the second parameter indicating a second portion of the engine parameters according to the second parameter to generate a second representation of the second set of compute engine parameters;
([0073] Operand selector 210 may include two multiplexers, one each for weights and input activations, with 1-bit select control (value 0 indicates 4-bit LSB to be chosen for multiplier and otherwise 4-bit MSB is selected) … SelW and SelA are bit select control bits)
NOTE: Abuhatzera teaches weights and input activations both having the aforementioned first (higher) precision, and a first and second parameter SelW (for the weights) and SelA (for the input activations) for determining whether to represent the weights and inputs as the 4 least significant bits (LSBs) or the 4 most significant bits (MSBs) for use in the multiplier.
Thus, Abuhatzera teaches defining a first parameter for the inputs (SelA, for the input activations) and a second parameter for the AI compute engine (SelW, for the weights used by the AI compute engine), the first parameter indicating a first portion of the first precision to use for computations by the AI compute engine (SelA indicates whether to use the MSB or LSB portion of the input precision for the multiplier) and the second parameter indicating a second portion of the first precision to use for computation by the AI compute engine (SelW indicates whether to use the MSB or LSB portion of the weight precision for the multiplier), and adapting the inputs according to the first parameter to generate a first representation of the inputs (for the multiplier, the input activations are represented by the MSB or LSB portions depending on SelA) and adapting a second set of compute engine parameters according to the second parameter to generate a second representation of the second set of compute engine parameters (for the multiplier, the set of compute engine weights are represented by the MSB or LSB portions depending on SelW).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, for the second parameter to represent a portion of the second precision rather than the first, further explained later.
and generate an
([0024] example approaches disclosed herein convert the weights and activations of a neural network topology for inference … to be mathematically-equivalent represented in a lower precision format. For example, the weight and activations may be mathematically-equivalent represented by 4 bit (4b) integers (also referred to as “nibbles”) and perform multiplications using 4-bit multipliers.)
NOTE: Abuhatzera teaches representing inputs and weights at a lower precision format (as previously taught) to perform multiplications of the neural network inference workload.
Thus, Abuhatzera teaches generating an AI workload corresponding to the inputs based on the first representation and the second representation.
Abuhatzera fails to teach but Henry teaches;
and generate an approximation of an AI workload
([0131] The lower precision values may use the hardware circuits to generate weight to a neural network; and they may also use the same or different hardware circuits to generate input values to an activation function… [0049] Converting a single floating-point value … format to multiple values in bfloat16 format or integer formats … often results in the multiple values in a lower precision in each of the multiple values than the original floating-point value, because less bits are used to represent each of the multiple values ... embodiments of the invention may result in … approximately same or even better accuracy than the arithmetic operations without the conversion.)
NOTE: Henry teaches representing weights and inputs of a neural network at a lower precision to approximate higher precision arithmetic operations.
Thus, Henry teaches generating an approximation of an AI workload (neural network arithmetic) corresponding to the inputs and weights represented at a lower precision.
OBVIOUSNESS TO COMBINE HENRY WITH ABUHARZERA:
Henry and Abuhatzera are both analogous art to the present disclosure as they pertain to performing higher precision computations using decomposed operations.
Abuhatzera teaches defining parameters for guiding the conversion of inputs and compute engine parameters (weights) to a lower precision for AI compute engine computations, while Henry teaches converting inputs and weights to a lower precision to generate an approximation of an AI workload.
Additionally, Henry states;
([0049] Converting a single floating-point value … to multiple values in bfloat16 format or integer formats … often results in the multiple values in a lower precision in each of the multiple values than the original floating-point value, because less bits are used to represent each of the multiple values … Yet faster hardware units/circuits for the bfloat16 and integer formats … exist in some computer architecture, and the converted values may be used in multiple arithmetic operations … so that the additional cost … of conversion and reconstruction is spread over the multiple arithmetic operations, embodiments of the invention may result in faster overall arithmetic operation time (thus yielding higher throughput) as well as the approximately same or even better accuracy than the arithmetic operations without the conversion.)
NOTE: Henry states that converting the values to a lower precision to generate an approximation of an arithmetic operation results in faster computation time, higher throughput, and approximately the same accuracy.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the first and second lower precision representations disclosed by Abuhatzera to generate an approximation of an AI workload as disclosed by Henry to improve the computation time and throughput of the AI compute engine computations, without sacrificing accuracy.
Abuhatzera and Henry fail to teach but Hoang teaches;
Weight values having the same precision that the AI compute engine is configured to process
([0121] The memory cells of each of the first memory arrays are each configured to store a bit of a multi-bit value of a weight of a neural network… independently configure each of the first memory arrays to a corresponding one of a plurality precision levels; store weight values of neural network in one or more of the first memory arrays with the corresponding level of precision; and perform one or more in-array multiplication and accumulation operations between multi-bit input values of the neural network and the weight values neural network in each of the first memory arrays with the corresponding level of precision.)
OBVIOUSNESS TO COMBINE HOANG WITH ABUHARZERA AND HENRY:
Hoang is analogous art to the present disclosure as it pertains to an AI compute engine having configurable precision.
Abuhatzera already teaches accelerating neural network workloads using low precision multiplication, and Hoang teaches mapping neural network weight matrices to blocks configured with corresponding precision levels.
Hoang states;
([0112] Under the arrangement of FIG. 22, the weight matrices of the neural layers are mapped to the blocks that can be configured with different precisions which are pre-determined after a quantization-aware training phase. Digital logic on the host, on the memory circuit, or a combination of these can perform bit extension or bit truncation to compromise the precision of activation required for intra-block data propagation in an inference operation. This arrangement can use layers of a neural network to be quantized at different bit precisions in order to improve computation throughput, memory utilization and energy efficiency with acceptable or zero accuracy loss.)
NOTE: The arrangement of Hoang is capable of configuring different layers to different precision levels, thereby allowing different neural network layers to be quantized at different bit precisions to improve throughput, memory utilization and energy efficiency with acceptable or zero accuracy loss.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the low precision neural network accelerator and workload approximation system of the Abuhatzera-Henry combination such that the weight values are stored or provided at the same precision level used by the compute engine using the arrangement taught by Hoang. A person of ordinary skill would have been motivated to make this modification because the arrangement of Hoang is capable of configuring different layers to different precision levels which improves computation throughput, memory utilization and energy efficiency with acceptable or zero accuracy loss.
From this reasoning, Abuhatzera in view of Henry and Hoang reasonably teaches;
the second parameter indicating a second portion of the second precision to use for computation by the AI compute engine
Abuhatzera teaches a first precision representing neural network inputs and a second precision that an AI compute engine is configured to operate with. Abuhatzera further teaches a first and second parameter (selA and selW) indicating a portion of the precision to use for the input activations and weights, respectively. The gap with Abuhatzera is that the second parameter (selW) does not indicate a portion of the second precision (i.e. the precision of the AI compute engine), but instead indicates a portion of the precision of the weights.
Hoang fills this gap by providing a teaching that the precision of the weights is the same as the precision utilized by an AI compute engine.
This, in combination, teaches the second parameter (selW) indicating a second portion of the second precision (selW indicates a portion of the weight precision, which, as modified in view of Hoang, is the same as the AI compute engine precision) to use for computation by the AI compute engine.
Regarding claim 2, Abuhatzera teaches;
the first precision is greater than the second precision.
([Abstract] wherein the first precision format is a higher precision format than the second precision format.)
Regarding claim 3, Abuhatzera teaches;
inputs having the first precision … second precision [being the precision that the AI compute engine is configured for processing in] … inputs / compute engine parameters … used for computations by the AI compute engine
([Abstract] An apparatus to facilitate accelerating neural networks with low precision-based multiplication and exploiting sparsity in higher order bits is disclosed. The apparatus includes a processor comprising a re-encoder to re-encode a first input number of signed input numbers represented in a first precision format as part of a machine learning model, the first input number re-encoded into two signed input numbers of a second precision format, wherein the first precision format is a higher precision format than the second precision format. The processor further includes a multiply-add circuit to perform operations in the first precision format using the two signed input numbers of the second precision format)
Abuhatzera fails to teach but Henry teaches;
the first parameter specifies a first number of bits of the inputs
([0093] A1 and B1 ... are approximation of the ones of A and B ... and have lower precision than the ones of A and B ... A1 and B1 has values represented by the most significant 8-bit mantissa of the ones in A and B... the reminders of A and B after subtracting A1 and B1 are approximated ... as A2 and B2, respectively… [0131] The lower precision values may use the hardware circuits to generate weight to a neural network; and they may also use the same or different hardware circuits to generate input values to an activation function [0130] each value (the original values or ones resulting from the quantization of the original values) is represented using a plurality of bits including at least a first and second number of bits... first number of bits ... identified to approximate the each value ... second number of bits ... for a second lower precision value is identified to represent a remainder of the approximation of the first lower precision value, where the first and second lower precision values are used to perform the arithmetic operations with lower precision values from one or more other operands)
NOTE: Henry teaches representing values A and B using lower precision approximations, A1, A2, B1, and B2 (which can represent neural network weights or input activations). Henry teaches identifying numbers of bits for the lower precision approximations of each value. The lower precision approximations defined by the identified numbers of bits are then used in arithmetic computations.
Thus, Henry teaches a first parameter that specifies a first number of bits of the inputs (an identified number of bits for a first value, which can represent input activations) to use for computations (a number of bits specifies the number of bits of a [input] value to use for the approximate representation used in computations) and the second parameter specifies a second number of bits of the compute engine parameters (an identified number of bits for a second value, which can represent neural network weights) to use for computations (the number of bits specifies the number of bits of a [weight] value to use for the approximate representation used in computations).
OBVIOUSNESS: As previously taught, Abuhatzera and Henry are analogous art to the present disclosure.
Abuhatzera teaches parameters (selA and selW) specifying a portion of bits for both the inputs and compute engine parameters (weights) for low precision approximations to use for computations by an AI compute engine.
Henry provides a parameter for specifying a number of bits for low precision approximations of operands to be used in computations.
Henry further states;
([0071] one may quantize a FP128 value into an 8-long string of uint16s, but software double-double or software quad precision may operate faster than arithmetic operations using the 8-long string of uint16s. The determination of whether to quantization and using how many bits to quantize may depend on the available of hardware circuitry performing fast integer arithmetic operations and how much faster the hardware circuitry is.)
NOTE: Henry indicates that selecting the number of bits allows the system to exploit available fast integer arithmetic circuitry, thereby improving the speed / throughput while maintaining the desired accuracy level.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the first and second parameter of Abuhatzera to further specify the number of bits to use in the low precision approximations to match operand precisions to available fast arithmetic hardware, thereby improving computation speed / throughout while allowing the system to manage the accuracy-speed tradeoff.
Abuhatzera and Henry fail to teach but Hoang teaches;
compute engine parameters [neural network weights] having the second precision [i.e., the precision that the AI compute engine is configure to operate on]
([0121] The memory cells of each of the first memory arrays are each configured to store a bit of a multi-bit value of a weight of a neural network… independently configure each of the first memory arrays to a corresponding one of a plurality precision levels; store weight values of neural network in one or more of the first memory arrays with the corresponding level of precision; and perform one or more in-array multiplication and accumulation operations between multi-bit input values of the neural network and the weight values neural network in each of the first memory arrays with the corresponding level of precision.)
OBVIOUSNESS:
Using the same reasoning from claim 1, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify the low precision neural network accelerator and workload approximation system of the Abuhatzera-Henry combination such that the weight values are stored or provided at the same precision level used by the compute engine (the second precision) using the arrangement taught by Hoang. A person of ordinary skill would have been motivated to make this modification because the arrangement of Hoang is capable of configuring different layers to different precision levels which improves computation throughput, memory utilization and energy efficiency with acceptable or zero accuracy loss.
Regarding claim 4, Abuhatzera teaches;
the at least one processor is further configured to truncate the inputs based on the first parameter to generate the first representation.
([0073] Operand selector 210 may include two multiplexers, one each for weights and input activations, with 1-bit select control (value 0 indicates 4-bit LSB to be chosen for multiplier and otherwise 4-bit MSB is selected) … SelW and SelA are bit select control bits)
NOTE: Abuhatzera teaches selecting only the 4-bit MSB or LSB portion of the input activation to be used in computation based on the first parameter (selA). The selected 4-bit portion is a truncated representation of the original, full, input activation.
This, Abuhatzera teaches the previously taught at least one processor being further configured to truncate the inputs based on the first parameter (selA) to generate the first representation (the 4-bit portion of the original input activation).
Regarding claim 5, Abuhatzera teaches;
the at least one processor is further configured to adaptively select one of most significant bits or least significant bits for each of the first representation and the second representation based on the first parameter and the second parameter, respectively.
([0073] Operand selector 210 may include two multiplexers, one each for weights and input activations, with 1-bit select control (value 0 indicates 4-bit LSB to be chosen for multiplier and otherwise 4-bit MSB is selected) … SelW and SelA are bit select control bits)
NOTE: Abuhatzera teaches the previously taught at least one processor is further configured to adaptively select one of most significant bits or least significant bits for each of the first representation and the second representation (selecting 4-bit LSB or MSB portion for the first / input representation and for the second / weight representation) based on the first parameter and the second parameter (based on selA and selW), respectively.
Regarding claim 7, Abuhatzera teaches;
the compute engine parameters comprise one or more of weights or activations.
([0024] approaches disclosed herein convert the weights and activations of a neural network topology for inference (e.g., INT8 or INT16-based neural network) to be mathematically-equivalent represented in a lower precision format… weight and activations may be mathematically-equivalent represented by 4 bit (4b) integers (also referred to as “nibbles”) and perform multiplications using 4-bit multipliers.)
Regarding claim 8, Abuhatzera teaches;
the at least one processor
As previously taught
Abuhatzera fails to teach but Henry teaches;
define a third parameter defining a third number of times a summation operation is repeated to generate the approximation.
([0101] For example, we may refer the four inner-products of bfloat16 multiplications as the following: bx2_4p: given by A1*B1+A1*B2+A2*B1+A2*B2 [0102] where “bx” denotes the bfloat16 operations, “2” denotes each FP32 value is split into two bfloat16 values, and “4p” denotes that four terms are included in the computation.)
NOTE: Henry teaches an approximation for the product of operands A and B using a summation of a plurality of products of the aforementioned lower precision approximations of the operands. The number of products to be summed in the approximation is defined by the parameter #p (which is 4p in this example, indicating that 4 products are summed).
Thus, Henry teaches defining a third parameter (#p) defining a third number of times a summation operation is repeated to generate the approximation (#p defines the number of products that are summed in the approximation).
OBVIOUSNESS:
As previously taught, Abuhatzera and Henry are analogous art to the present disclosure.
Henry states;
([0103] To reduce the workload, we may skip the lower-order term, A2*B2, so that the workload becomes the following: bx2_3p: given by A1*B1+A1*B2+A2*B1)
NOTE: Henry indicates that by lowering the #p parameter from 4p to 3p, one less product is summed in the approximation, reducing the workload.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to include the third parameter taught by Henry in the low precision neural network accelerator and workload approximation system of the Abuhatzera-Henry combination to specify the number of terms to be summed in the approximation thereby allowing the approximation to be more configurable (i.e., configuring the approximation to prioritize workload or accuracy).
Regarding claims 9-13, 15-16,
Claims 9-13, 15-16 are processor implemented method claims that are substantially similar to apparatus claims 1-5, 7-8, respectively, and are rejected using the same reasoning.
Regarding claim 17,
Claim 17 is a non-transitory computer-readable medium claim that is substantially similar to apparatus claim 1, with one added limitation, which is taught by Abuhatzera;
A non-transitory computer-readable medium having program code recorded thereon, the program code executed by a processor and comprising:
PNG
media_image2.png
610
882
media_image2.png
Greyscale
([0108] In embodiments, the processor cores 718 are capable of executing machine-readable instruction sets 714)
NOTE: The computing device for performing the methods disclosed by Abuhatzera includes a non-transitory computer-readable medium (RAM) having program code recorded thereon (instructions), the program code executed by a processor (processor cores).
The remaining limitations are taught using the same reasoning from claim 1.
Regarding claims 18-21, 23-24,
Claims 18-21, 23-24 are non-transitory computer-readable medium claims that are substantially similar to apparatus claims 2-5, 7-8, respectively, and are rejected using the same reasoning.
Regarding claims 25-28, 30,
Claims 25-28, 30 are apparatus claims that are substantially similar to apparatus claims 1, 3-5, 8, respectively, and are rejected using the same reasoning.
Claim(s) 6, 14, 22, 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Abuhatzera (US 20200320375 A1, 2020-10-08) in view of Henry (US 20190042244 A1, 2019-02-07) further in view of Hoang (US 20210397974 A1, 2021-12-23) as applied to claim 1 above, and further in view of Victor Dmitriyevich Podlozhnyuk (hereinafter Podlozhnyuk) (US 20230153374 A1, 2021-11-16).
Regarding claim 6, Abuhatzera teaches;
at least one processor
As previously taught
[defining lower precision representations of inputs and compute engine parameters] based on the first parameter and the second parameter
([0073] Operand selector 210 may include two multiplexers, one each for weights and input activations, with 1-bit select control (value 0 indicates 4-bit LSB to be chosen for multiplier and otherwise 4-bit MSB is selected) … SelW and SelA are bit select control bits)
Abuhatzera, Henry, and Hoang fail to teach but Podlozhnyuk teaches;
round one or more of the inputs
([0045] In at least one embodiment, mathematical operations, such as matrix multiply operations, can be performed for tasks such as may relate to training of one or more neural networks… values can be received 402 for a matrix multiply operation, where those values have high precision … these received values can be selected 404 for decomposition… a decomposition process can be performed as illustrated in FIG. 1A that performs rounding on this FP32 value to obtain a lower precision value ... In at least one embodiment, these lower precision values resulting from this rounding can then be used for mathematical operations, such as matrix multiply operations)
NOTE: Podlozhnyuk teaches input to a neural network matrix multiplication being rounded to obtain lower precision values.
OBVIOUSNESS TO COMBINE PODLOZHNYUK WITH ABUHATZERA, HENRY, AND HOANG:
Podlozhnyuk is analogous art to the present disclosure as it pertains to converting values of a higher precision to a lower precision using rounding.
Abuhatzera provides the base of using a first and second parameter to define how input and compute engine parameters are to be converted into a lower precision representation to be used in computations while Podlozhnyuk provides a means for converting values into a lower precision using rounding.
Podlozhnyuk further states;
([0047] In at least one embodiment, such a process can decompose input data into sums of lower precision values, such as values of BF16 or TF32 data types. In at least one embodiment, a round-to-nearest rounding mode can be used, which can improve accuracy across values.)
Podlozhnyuk indicates that their rounding method improves accuracy across values. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use the rounding technique taught by Podlozhnyuk to convert the inputs and compute engine parameters to a lower precision as specified by the first and second parameters taught by Abuhatzera, to improve accuracy across values.
Regarding claim 14,
Claim 14 is a processor implemented method claim that is substantially similar to apparatus claim 6, and is rejected using the same reasoning.
Regarding claim 22,
Claim 22 is a non-transitory computer-readable medium claim that is substantially similar to apparatus claim 6, and is rejected using the same reasoning.
Regarding claim 29,
Claim 29 is an apparatus claim that is substantially similar to apparatus claim 6, and is rejected using the same reasoning.
CONCLUSION
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Matthew Alan Cady whose telephone number is (571) 272-7229. The examiner can normally be reached Monday - Friday, 7:30 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Cesar Paula can be reached on (571)272-4128. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC)
at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW ALAN CADY/ Examiner, Art Unit 2145
/CHAU T NGUYEN/ Primary Examiner, Art Unit 2145
Prosecution Insights