DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 9, 15 and 18 have been amended. Claims 2, 6 and 10-14 have been canceled. Claims 20-27 have been added. Claims 1, 3-5, 7-9 and 15-27 remain pending and have been examined.
Response to Arguments/Amendments
On pp. 3-4 of the 2/24/2026 remarks, Applicant generally argues that cited art of record Qiu, Kovvuri and Ghodrati fail to teach various claimed limitations. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
In response to applicant's argument that the examiner's conclusion of obviousness is based upon improper hindsight reasoning, it must be recognized that any judgment on obviousness is in a sense necessarily a reconstruction based upon hindsight reasoning. But so long as it takes into account only knowledge which was within the level of ordinary skill at the time the claimed invention was made, and does not include knowledge gleaned only from the applicant's disclosure, such a reconstruction is proper. See In re McLaughlin, 443 F.2d 1392, 170 USPQ 209 (CCPA 1971). The rejections include cited portions of each of the references to illustrate the knowledge which was within the level of ordinary skill at the time the claimed invention was made.
Applicant’s arguments, see pp. 2-4, filed 2/24/2026, with respect to the rejection(s) of claim(s) 1, 9 and 15 under 35 USC § 103 have been fully considered and are persuasive. Therefore, the rejections have been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of previously cited art Qiu, Kovvuri and Ghodrati, along with newly cited “Compute-Efficient Neural-Network Acceleration” by Wu et al. (“Wu”) and U.S. Patent 10338919 to Boswell et al.
Claim Objections
Claim 15 is objected to because of the following informalities: Lines 9-10 include: “computation units of matrix multiply accelerator.” This is a typo which should read: “computation units of a matrix multiply accelerator.”
Claims 22-27 are objected to because of the following informalities: Each of the claims include the phrase “The method according to any of claims …,” which is a typo that should be “The method according to claim …,” Appropriate correction is required.
Claims 22 and 26-27 are objected to because of the following informalities: Each of the claims include the word “whereinthe” which is a typo that should be “wherein the.” Appropriate correction is required.
Claim 23 is objected to because of the following informalities: The claim includes the phrase “the transformed exclusively” which is a typo that should be “the transformed component exclusively” Appropriate correction is required.
Claims 26-27 are objected to because of the following informalities: Claim 27 is presented in-line with the end of claim 26. That is, claim 27 does not appear on a new line. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-5, 7-9, 20-22 and 24-27 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement.
The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1, 9, 21, 22 and 26 have been amended to include the limitation “dynamic range.” The as-filed disclosure includes references to “bit range” (e.g. ¶ 0025), “full range” (e.g. ¶ 0058), and “upper range” (e.g. ¶ 0059), but the term “dynamic range” was not found. The amendment perhaps appears to be based upon “loss in a bit range” as described in ¶ 0043, and will be interpreted accordingly for the purpose of further examination.
Claims 3-5, 7-8, 20-22 and 24-27 are each rejected as carrying the limitations of a rejected base claim.
Claims 3, 8 and 23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 8 recite the limitation "The method according to claim 2," in line 1. There is insufficient antecedent basis for this limitation in the claims. It is noted that claim 2 has been canceled. For the purpose of further examination, claims 3 and 8 will be interpreted as being dependent upon claim 1.
Claim 3 recites the limitation "the ADC" in line 3. There is insufficient antecedent basis for this limitation in the claims. For the purpose of further examination, claims 3 and 8 will be interpreted as “an ADC.”
Claim 23 recites the limitation "the reconstructing or mapping" in line 2. There is insufficient antecedent basis for this limitation in the claim. It is noted that claim 1 includes limitations directed to “reconstructing” and “mapping.” For the purpose of further examination, claim 23 will be interpreted as being dependent upon claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5, 7-8 and 21-25 are rejected under 35 U.S.C. 103 as being unpatentable over “FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks” by Qiu et al. (hereinafter “Qiu”) in view of U.S. Patent Application Publication 20190286973 by Kovvuri et al. ("Kovvuri") U.S. Patent Application Publication 20220350662 by Ghodrati et al. ("Ghodrati"), and “Compute-Efficient Neural-Network Acceleration” by Wu et al. (“Wu”).
In regard to claim 1, Qiu discloses:
1. A method for transforming one or more components of a computation network graph, the method comprising: See Qiu, at least p. 1224, section II(A), “By redesigning the rectified point of ReLU as a learnable parameter, we propose flexible rectified linear unit (FReLU) to improve flexibility on the horizontal and vertical axes.”
receiving, … , a computation network … for an artificial neural network, wherein a plurality of activation operation nodes and a plurality of branches define the computation network …, See Qiu, p. 1223, left column, “The activation function is an important component in neural networks. It provides the non-linear property for deep neural networks and controls the information propagation through adjacent layers. Therefore, the design of the activation function usually plays an important role to achieve good learning behavior and performance.”
Qiu discloses the broad notion of network analysis and activation operation identification. See Qiu, p. 1224, left column, “Since the activation function generally follows convolutional/linear layer, the arbitrary input value for FReLU can be learned by the preceding convolutional/linear layer.”
Qiu does not expressly disclose the following limitations, but they are taught by Kovvuri and Ghodrati:
receiving … by a compiler program, a … graph …; identifying, by the compiler program, a sub-graph associated with a matrix multiply accelerator (MMA) implemented as an … computation unit of a … integrated circuit; See Kovvuri, ¶ 0066, “For example, a compiler can identify the subgraph” Also Kovvuri, ¶ 0046, e.g. “integrated circuit.” Also ¶ 0051, “… neural processing cores for performing operations, including matrix and vector multiplication …” Also ¶ 0066-0067, “For example, a compiler can identify the subgraph. … Once the subgraph 320 has been identified, the neural network model 310 can be partitioned such that the subgraph 320 is evaluated with a neural network hardware accelerator. For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s subgraph compiler with Qiu’s network in order to provider higher performance as suggested by Kovvuri (see ¶ 0020).
Qiu and Kovvuri do not expressly teach: analog computation of mixed-signal circuit. This is taught by Ghodrati. See Ghodrati, ¶ 0005, “accumulate a result of the multiplication operation as an analog signal.” ¶ 0078, “The compiler takes in a caffe-2 specification of the DNN model, finds the optimum tiling and cutting for each layer, and maps it to the ISA of the BIHIWE architecture.” ¶ 0097, “This bit-partitioned arithmetic enables rearranging the highly parallel MACC operations in modern DNNs into wide low-bitwidth computations that map efficiently to low-bitwidth mixed-signal units. Further, these units operate in charge domain using switched-capacitor circuitry and reduce the rate of A/D conversion by accumulating partial results in the analog domain.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Ghodrati’s analog mixed-signal computation with the network compilation of Qiu and Kovvuri in order to provide high noise robustness while reducing energy overhead as suggested by Ghodrati (see ¶ 0034).
Kovvuri also teaches:
determining that the sub-graph includes at least one of: (i) a data type restriction incompatible with the analog computation unit, or (ii) a loss of dynamic range resulting from an activation function applied to outputs of the MMA; See Kovvuri, ¶ 0042, “The training can be fine-tuned using computations where the precision matches the precision of the hardware executing each of the subgraphs.” ¶ 0066, “For example, a compiler can identify the subgraph.” Also see ¶ 0071-0072, e.g. “For example, portions of the neural network model that are to be executed on a CPU or GPU can be partitioned from portions of the neural network model that are to be executed on a neural network accelerator.” ¶ 0090, “In particular, when the subgraph 320 is executed on a general-purpose CPU, the computations of the subgraph 320 can be truncated or rounded to match a precision of the accelerator.” ¶ 0094, “Specifically, the marker nodes can be used to truncate or round values destined for and/or leaving from the subgraph 320. In this manner, a precision of the values entering and/or leaving the subgraph 320 can be matched to a precision of the neural network accelerator.” ¶ 0116, “For example, the marker nodes can be used to mark a boundary of the subgraph. The marker nodes can include interface information for communicating between a general-purpose CPU and the neural network accelerator. The marker nodes can include quantization information so that a change in precision of the underlying hardware can be properly accounted for.”
transforming, by the compiler program at compile time, the sub-graph generating a modified sub-graph configured for execution on the analog computation unit, Kovvuri, ¶ 0067, “For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” Also see Ghodrati as cited above.
Qiu discloses:
the transforming comprising: replacing a first activation function with a second activation function configured to preserve a full bit range of the analog computation unit; and Qiu, p. 1224, section II.C, “By adding a learnable bias term, the output range of FReLU is [b, +∞), which is helpful to ensure efficient learning.”
Qiu does not expressly disclose:
adjusting a bias term of the MMA based on a bit width of the analog computation unit to compensate for the replacing of the first activation function; and This is taught by Kovvuri and Wu. See Wu p. 198, section 3.6, “Processor 𝑃 uses 16 significant bits for activations and input biases. The other processors use just eight bits. All weights and biases are quantized to 8 bits (again with 5-bit exponents).” Also see Kovvuri, ¶ 0047, “In some examples, the dot product can be adjusted by a bias value before it is used as an input to an activation function.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Wu’s bias compensation with the activation functions of Qiu and the circuitry of Kovvuri and Ghodrati in order to ensure compatibility as essentially suggested by Wu.
Kovvuri and Ghodrati also teach:
reconstructing the computation network graph to assign the modified sub-graph to the analog computation unit of the mixed-signal integrated circuit. Kovvuri, ¶ 0052, “The control unit 160 can schedule the data transfer between the CPU 165 and the plurality 110 of neural processing cores so that a latency between the CPU 165 and the plurality 110 of neural processing cores is optimized for the particular division of the subgraphs on the different hardware components.” Also see Ghodrati, ¶ 0005, “convert a subset of digital-domain bits partitioned from a first input vector to a first analog-domain signal and a second digital-to-analog convertor configured to convert a subset of digital-domain bits partitioned from a second input vector to a second analog-domain signal.”
In regard to claim 5, Qiu discloses:
5. The method according to claim 1, wherein at runtime, the activation operation node comprising the second activation function is applied following a first computation operation of the computation network graph that outputs the dataset having the negative data values. Qiu is directed to a neural network whereby neuron activation nodes utilize input data, either from first layer inputs or from other activation functions. Also see at least p. 1223, left column, “negative inputs.”
In regard to claim 7, Qiu does not expressly disclose:
7. The method according to claim 5, wherein the computation operation is implemented by a streaming arithmetic logic unit. This is taught by Ghodrati. See at least Fig. 3. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Ghodrati’s arithmetic logic with Qiu’s activation functions in order to accelerate a DNN as suggested by Ghodrati (see ¶ 0031).
In regard to claim 8, Qiu discloses:
8. The method according to [claim 1], wherein the adjusting the weights bias value of the computation operation is further based on one or more of a gain and a scaling factor. See Qiu, p. 1225, top left column, “… representation restoration (scale γ and bias β) in BN.”
In regard to claim 21, Qiu and Kovvuri also teach:
21. The method according to claim 1, wherein the determining of the data type restriction or loss of dynamic range is performed based on hardware configuration parameters of the analog computation unit, including an analog bit width or analog-to-digital converter range. See Kovvuri ¶ 0090, ¶ 0094 and ¶ 0116 as cited above.
In regard to claim 22, Qiu also discloses:
22. The method according to [claim] 1, whereinthe second activation function is selected to preserve a full unsigned dynamic range of the analog computation unit without truncating negative values to zero. Qiu, p. 1223, bottom right column, “… adjust the ReLU output by a learnable rectified point to capture the negative information.”
In regard to claim 23, Qiu, Kovvuri and Ghodrati also teach:
23. The method according to [claim 1], wherein the reconstructing or mapping comprises assigning the transformed [component] exclusively to the analog domain and preventing execution of the modified sub-graph in the digital domain. See Kovvuri, ¶ 0080 and ¶ 0089 as cited above. Also see Ghodrati ¶ 0074, 0078, and 0097 as cited above.
In regard to claim 24, Qiu, Kovvuri and Ghodrati also teach:
24. The method according to [claim] 1, wherein the transforming is performed prior to deployment of the artificial neural network onto the mixed-signal integrated circuit and independent of runtime execution of the artificial neural network. See Kovvuri, ¶ 0038, “According to another aspect of the disclosed technology, an installer, which programs the specialized hardware accelerator, and a runtime environment can further optimize the subgraph, because they use code and metadata generated by a neural network compiler.” Also see Ghodrati, ¶ 0073, e.g. “When cuts and tiles are determined, the compiler generates the binary code that contains the communication and computation instruction blocks. In some implementations, all the instructions are statically scheduled. The static scheduling can be extended to cluster coordination, and data communication and transfer.”
In regard to claim 25, Qiu and Kovvuri also teach:
25. The method according to [claim] 1, wherein the identifying further comprises extracting the sub-graph from the computation network graph for a distinct hardware-specific optimization procedure prior to reconstructing the computation network graph. See Kovvuri, ¶ 0066, “The subgraph 320 can be identified in a number of different ways. For example, a compiler can identify the subgraph.”
Claims 3 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Wu as applied above, and further in view of “How do you calculate a decimal number to 8-bit excess binary?” by Ryan et al. (“Ryan”).
In regard to claim 3, Qiu does not expressly disclose:
3. The method according to [claim 1], wherein the adjusting the weights bias value of the computation operation includes reducing the weights bias value by one-half the bit range of [an] ADC. However, this is taught by Ghodrati and Ryan.
See Ghodrati ¶ 0060, e.g. “ADC.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Ghodrati’s ADC with Qiu’s network in order to provide data processing which provides balance between speed and resolution as suggested by Ghodrati (see ¶ 0061).
Also see Ryan, p. 1, e.g. “Since it's typical to select a bias equal to half the available range in magnitude, for 8 bits we 'll pick -127 as the bias.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide Qiu’s data values using Ryan’s number representation since the use of such representation is typical as suggested by Ryan.
In regard to claim 26, Qiu does not expressly disclose:
26. The method according to [claim] 1, whereinthe adjusting of the bias term comprises calculating an adjustment value derived from one-half of an input range of the analog MMA and applying the adjustment value to restore dynamic range within the analog computation unit. This is taught by Kovvuri, Wu and Ryan. See Kovvuri, ¶ 0090, 0094 and 0116 and Wu as cited above. Also see Ryan, p. 1, e.g. “Since it's typical to select a bias equal to half the available range in magnitude, for 8 bits we 'll pick -127 as the bias.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide Qiu’s data values using Ryan’s number representation since the use of such representation is typical as suggested by Ryan.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Wu as applied above, and further in view of U.S. Patent Application 20200226468 by Hamilton et al. ("Hamilton").
In regard to claim 4, Qiu discloses:
4. The method according to claim 1, wherein the first activation function comprises a rectifier linear unit, the second activation function comprises a … [FReLU], and after the transformation, by the compiler program, the activation operation node comprises the … [FReLU]. See Qiu, p. 1227, section III(B), e.g. “To compare the compatibility of FReLU and ELU with BN, we first investigate the performances with simply replacing the ReLU with FReLU and ELU, that is using the architecture in Fig. 5(a).”
Qiu does not expressly disclose: hard sigmoid. However, this is taught by Hamilton. See Hamilton, ¶ 0057, “FIG. 4 is a graph that illustrates a hard sigmoid activation function … and thereby simplifies node processing as compared to using the s-shaped Sigmoid.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Hamilton’s hard sigmoid for Qiu’s activation function in order to simplify node processing as suggested by Hamilton.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri and Ghodrati.
In regard to claim 9, Qiu discloses:
9. A method for transforming one or more components of a computation graph, the method comprising: See Qiu, at least p. 1224, section II(A), “By redesigning the rectified point of ReLU as a learnable parameter, we propose flexible rectified linear unit (FReLU) to improve flexibility on the horizontal and vertical axes,”
receiving, … a computation [network] … for an artificial neural network, wherein … a plurality of … computation nodes and a plurality of branches define the computation …, See Qiu, p. 1223, left column, “The activation function is an important component in neural networks. It provides the non-linear property for deep neural networks and controls the information propagation through adjacent layers. Therefore, the design of the activation function usually plays an important role to achieve good learning behavior and performance.”
Qiu does not expressly disclose: … by a graph compiler, … graph … graphical representations of … graphically-represented … graph. See Kovvuri, ¶ 0066, “For example, a compiler can identify the subgraph” Also Kovvuri, ¶ 0046, e.g. “integrated circuit.” Also ¶ 0051, “… neural processing cores for performing operations, including matrix and vector multiplication …” Also ¶ 0066-0067, “For example, a compiler can identify the subgraph. … Once the subgraph 320 has been identified, the neural network model 310 can be partitioned such that the subgraph 320 is evaluated with a neural network hardware accelerator. For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s subgraph compiler with Qiu’s network in order to provider higher performance as suggested by Kovvuri (see ¶ 0020).
wherein the graph compiler, at compile time: detects a graph part of the computation graph that is incompatible with execution on an … matrix multiply accelerator (MMA) … integrated circuit due to data type or dynamic range limitations, Kovvuri, ¶ 0042, “The training can be fine-tuned using computations where the precision matches the precision of the hardware executing each of the subgraphs.” ¶ 0066, “For example, a compiler can identify the subgraph.” Also see ¶ 0071-0072, e.g. “For example, portions of the neural network model that are to be executed on a CPU or GPU can be partitioned from portions of the neural network model that are to be executed on a neural network accelerator.” ¶ 0090, “In particular, when the subgraph 320 is executed on a general-purpose CPU, the computations of the subgraph 320 can be truncated or rounded to match a precision of the accelerator.” ¶ 0094, “Specifically, the marker nodes can be used to truncate or round values destined for and/or leaving from the subgraph 320. In this manner, a precision of the values entering and/or leaving the subgraph 320 can be matched to a precision of the neural network accelerator.” ¶ 0116, “For example, the marker nodes can be used to mark a boundary of the subgraph. The marker nodes can include interface information for communicating between a general-purpose CPU and the neural network accelerator. The marker nodes can include quantization information so that a change in precision of the underlying hardware can be properly accounted for.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s precision matching with Qiu’s network in order to achieve more accurate results in addition to reducing development time and costs as suggested by Kovvuri (see ¶ 0042).
Kovvuri does not expressly disclose: analog … of a mixed-signal This is taught by Ghodrati. See Ghodrati, ¶ 0005, “accumulate a result of the multiplication operation as an analog signal.” ¶ 0078, “The compiler takes in a caffe-2 specification of the DNN model, finds the optimum tiling and cutting for each layer, and maps it to the ISA of the BIHIWE architecture.” ¶ 0097, “This bit-partitioned arithmetic enables rearranging the highly parallel MACC operations in modern DNNs into wide low-bitwidth computations that map efficiently to low-bitwidth mixed-signal units. Further, these units operate in charge domain using switched-capacitor circuitry and reduce the rate of A/D conversion by accumulating partial results in the analog domain.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Ghodrati’s analog mixed-signal computation with the network compilation of Qiu and Kovvuri in order to provide high noise robustness while reducing energy overhead as suggested by Ghodrati (see ¶ 0034).
Qiu also discloses:
modifies one or more activation functions and one or more bias terms … See Qiu, p. 1223, right column, “These methods suggest that the negative part is helpful for neural networks.” Also p. 1227, section III(B), e.g. “To compare the compatibility of FReLU and ELU with BN, we first investigate the performances with simply replacing the ReLU with FReLU and ELU, that is using the architecture in Fig. 5(a).” Also see Qiu, p. 1224, section II.A., e.g. “Since the activation function generally follows convolutional/linear layer, the arbitrary input value for FReLU can be learned by the preceding convolutional/linear layer.” Also section II.C., “By adding a learnable bias term, the output range of FReLU is [b, +∞), which is helpful to ensure efficient learning.”
Qiu does not expressly disclose: … producing an analog-executable graph part, map the analog-executable graph part to the analog MMA. and This is taught by Ghodrati. Ghodrati, ¶ 0005, “accumulate a result of the multiplication operation as an analog signal.” ¶ 0078, “The compiler takes in a caffe-2 specification of the DNN model, finds the optimum tiling and cutting for each layer, and maps it to the ISA of the BIHIWE architecture.” ¶ 0097, “This bit-partitioned arithmetic enables rearranging the highly parallel MACC operations in modern DNNs into wide low-bitwidth computations that map efficiently to low-bitwidth mixed-signal units. Further, these units operate in charge domain using switched-capacitor circuitry and reduce the rate of A/D conversion by accumulating partial results in the analog domain.”
Claims 15 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and U.S. Patent 10338919 to Boswell et al. ("Boswell").
In regard to claim 15, Qiu discloses:
15. A method for transforming one or more components of a computation network graph, the method comprising: See Qiu, p, 1223 section 1, e.g. “The activation function is an important component in neural networks.” Also see at least p. 1224, section II(A), “By redesigning the rectified point of ReLU as a learnable parameter, we propose flexible rectified linear unit (FReLU) to improve flexibility on the horizontal and vertical axes.”
receiving, … a computation [network] … for an artificial neural network, wherein … a plurality of … computation nodes and a plurality of branches define the computation …, See Qiu, p. 1223, left column, “The activation function is an important component in neural networks. It provides the non-linear property for deep neural networks and controls the information propagation through adjacent layers. Therefore, the design of the activation function usually plays an important role to achieve good learning behavior and performance.”
Qiu does not expressly disclose: by a graph compiler, … graph … graphical representations of … graphically-represented … graph. See Kovvuri, ¶ 0066, “For example, a compiler can identify the subgraph” Also Kovvuri, ¶ 0046, e.g. “integrated circuit.” Also ¶ 0051, “… neural processing cores for performing operations, including matrix and vector multiplication …” Also ¶ 0066-0067, “For example, a compiler can identify the subgraph. … Once the subgraph 320 has been identified, the neural network model 310 can be partitioned such that the subgraph 320 is evaluated with a neural network hardware accelerator. For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s subgraph compiler with Qiu’s network in order to provider higher performance as suggested by Kovvuri (see ¶ 0020).
identifying, by the graph compiler, a component of the computation graph including a first computation operation node of a computation graph that outputs a dataset having negative data values; Qiu discloses the broad notion of network analysis and activation operation identification. See Qiu, p. 1224, left column, “Since the activation function generally follows convolutional/linear layer, the arbitrary input value for FReLU can be learned by the preceding convolutional/linear layer.” Also see Qiu, p. 1223 section 1, e.g. “The activation function is an important component in neural networks. It provides the non-linear property for deep neural networks and controls the information propagation through adjacent layers. … Moreover, ReLU is computational efficient by just outputting zero for negative inputs.”
Qui does not expressly disclose: operation of … one or more computation units of … matrix multiply accelerator (MMA). This is taught by Kovvuri. See Kovvuri, ¶ 0066, “For example, a compiler can identify the subgraph” Also Kovvuri, ¶ 0046, e.g. “integrated circuit.” Also ¶ 0051, “… neural processing cores for performing operations, including matrix and vector multiplication …” Also ¶ 0066-0067, “For example, a compiler can identify the subgraph. … Once the subgraph 320 has been identified, the neural network model 310 can be partitioned such that the subgraph 320 is evaluated with a neural network hardware accelerator. For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s subgraph compiler with Qiu’s network in order to provider higher performance as suggested by Kovvuri (see ¶ 0020).
Qiu does not expressly disclose a graph component that cannot be assigned to … MMA of a mixed-signal integrated circuit based on the component including an MMA data type restriction; This is taught by Kovvuri and Ghodrati as indicated below.
Kovvuri, teaches identification of subgraphs that include a data type restriction. See Fig. 3A. Also ¶ 0057, “each individual node can have an activation function” ¶ 0062, “The sum of the dot product can be used as an input to an optional activation function.” ¶ 0072, “The compiler can generate interfaces between the portions of the neural network model that are executed in the different environments. The compiler can generate a configuration bitstream and/or code for programing the neural network accelerator.” ¶ 0080, “Specifically, the compiler 420 can identify subgraphs of the neural network model and determine which of those subgraphs will be executed on the server 410 and which of those subgraphs will be executed on the accelerator 450.” ¶ 0089, “For example, many software-implemented neural network models may model node and other network value using 32-bit values. The neural network accelerator 450 may model subgraphs using a fewer number of bits, for example 16, 8, 5, 4, or other number of bits. The provided interface can implement this quantization by converting values to and from the appropriate formats when passing between the server and the accelerator.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to analyze Qiu’s graph using Kovvuri’s subgraph identification in order to provide appropriate formats to various execution environments as suggested by Kovvuri.
Ghodrati teaches the use of a mixed-signal accelerator. See ¶ 0046, “As such, it is possible to integrate a larger number of mixed-signal compute units on a chip with a given power budget compared to a digital architecture.” ¶ 0047, “In some embodiments, BIHIWE is a hierarchically clustered architecture that allocates multiple accelerator cores as a cluster to each vault. FIG. 2B depicts a single accelerator core in accordance with one or more embodiments of the present technology. As shown in FIG. 2B, each core (e.g., as shown in FIG. 2C) is self-sufficient and packs a mixed-signal systolic array of MS-WAGGs as well as the digital units that perform pooling, activation, normalization, etc. The mixed-signal array is responsible for the convolutional and fully connected layers.” Also see Fig. 6 and ¶ 0073, “In some embodiments, as FIG. 6 shows, DNNs are compiled to BIHIWE through a multi-stage process, e.g., beginning with a Caffe2 DNN specification file. The high-level specification provided in the Caffe2 file is translated to a layer DataFlow Graph (DFG) that preserves the structure of the network. The DFG goes through an algorithm that cuts the DFG and tiles the data to map the DNN computation to the accelerator clusters and cores.” Also ¶ 0074, “The BIHIWE compiler stack can statically assign communication and compute instruction blocks to accelerator clusters, shifting the complexity from hardware to the compiler. By splitting the data transfer and on-chip data processing into separate instructions, the BIHIWE ISA enables software pipelining between clusters and allows the memory accesses to run ahead and fetch data for the next tile while processing the current tile.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Ghodrati’s analog mixed-signal computation with the network compilation of Qiu and Kovvuri in order to provide high noise robustness while reducing energy overhead as suggested by Ghodrati (see ¶ 0034).
wherein the mixed-signal integrated circuit comprises: an analog domain including the MMA, and a digital domain including one or more digital circuits; and See Ghodrati, ¶ 0005, “Each of the plurality of mixed-signal units comprises a first digital-to-analog convertor configured to convert a subset of digital-domain bits partitioned from a first input vector to a first analog-domain signal and a second digital-to-analog convertor configured to convert a subset of digital-domain bits partitioned from a second input vector to a second analog-domain signal. … The apparatus also includes an analog-to-digital converter coupled to the circuitry to convert the analog-domain result into a digital-domain result.”
Qiu does not expressly disclose: wherein the identifying further comprises detecting, at compile time, a graph component of the computation graph that cannot be executed in the analog domain due to a signed data restriction; This is taught by Boswell. See Boswell col. 30, lines 20-25, “For example, input vectors
A
→
&
B
→
can be provided in 16-bit floating-point, 8-bit signed/unsigned integer, 16-bit signed/unsigned integer, 32-bit fixed-point format, etc. The conversion/encoding logic 1315 is configured to convert all input values to a half-precision, floating-point value format for compatibility with the rest of the datapath 1300.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Boswell’s conversion with Kovvuri’s accelerator in order to ensure data compatibility as suggested by Boswell.
… transforming the component of the computation graph, by one or more computer processors executing the graph compiler, See Qiu, p. 1227, section III(B), e.g. “To compare the compatibility of FReLU and ELU with BN, we first investigate the performances with simply replacing the ReLU with FReLU and ELU, that is using the architecture in Fig. 5(a).”
Qiu does not expressly disclose transforming … at compile time, This is taught by Kovvuri. See ¶ 0066, “For example, a compiler can identify the subgraph” Also Kovvuri, ¶ 0046, e.g. “integrated circuit.” Also ¶ 0051, “… neural processing cores for performing operations, including matrix and vector multiplication …” Also ¶ 0066-0067, “For example, a compiler can identify the subgraph. … Once the subgraph 320 has been identified, the neural network model 310 can be partitioned such that the subgraph 320 is evaluated with a neural network hardware accelerator. For example, the subgraph 320 can be mapped to specialized neural network hardware implemented with an FPGA, an ASIC, a neural network processor, a digital signal processor, a graphics processing unit (GPU), or other suitable acceleration hardware.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Kovvuri’s subgraph compiler with Qiu’s network in order to provider higher performance as suggested by Kovvuri (see ¶ 0020).
Qiu also discloses:
… the transforming includes: (i) installing an activation function within the first computation operation node of the computation graph, the activation function, when applied to the output of the first computation operation node, converts the negative data values of the output of the first computation operation node to positive data values. or (ii) replacing a first activation function of the first computation operation node of the computation graph with a second activation function, See Qiu, p. 1227, section III(B), e.g. “To compare the compatibility of FReLU and ELU with BN, we first investigate the performances with simply replacing the ReLU with FReLU and ELU, that is using the architecture in Fig. 5(a).”
… the second activation function, when applied to the output of the first computation operation node, converts the negative data values of the output of the first computation operation node to positive data values. See Qiu, p. 1224,
PNG
media_image1.png
166
791
media_image1.png
Greyscale
Also see p. 1224, bottom right column, “Considering a layer with n units, FReLU with b = 0 (equal to ReLU) or b > 0 can only generate 2n output states, while FReLU with b < 0 can generate 3n output states. As shown in Table III, the learnable biases tend to negative and bring the improvement in the network by a successful training.” Note that Table III on p. 1227 displays the results of utilizing b>0. Also see p. 1225, bottom left column, “When b > 0, FReLU tends to move the output distribution of ReLU to larger positive areas, which is unnecessary for state extension.”
In regard to claim 18, Qiu as modified above also teaches:
18. The method according to claim 15, wherein the transforming, by the graph compiler, further includes: adjusting the weights bias value of a second computation operation node based on an input range of a matrix multiply accelerator implementing the second computation operation node. See Qiu, p. 1223 and p. 1224, section II.C as cited above. Also see Kovvuri, ¶ 0042 and 0090 and 0094 as cited above.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Boswell as applied above, and further in view of Hamilton.
In regard to claim 16, Qiu does not expressly disclose:
16. The method according to claim 15, wherein the first computation operation node is implemented by a streaming arithmetic logic unit, and the activation function comprises a hard sigmoid activation. However, these limitations are taught by Ghodrati and Hamilton as indicated above with respect to claims 4 and 7.
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Boswell as applied above, and further in view of Hamilton and U.S. Patent Application 20210133565 by Shamir et al. ("Shamir").
In regard to claim 17, Qiu does not expressly disclose:
17. The method according to claim 15, wherein the first computation operation node is implemented by a streaming arithmetic logic unit or a matrix multiply accelerator, and … and the second activation function replacing the first activation function comprises a hard sigmoid activation. However, this is taught by Ghodrati and Hamilton as indicated above with respect to claims 4, 7 and 16.
Qiu does not expressly disclose: the first activation function comprises a hardtanh activation. However, this is taught by Shamir. See Shamir, ¶ 0035, “HardTanh activations.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Shamir’s HardTanh activation in Qiu’s computations since HardTanh is “deployable with limited functionality hardware” as suggested by Shamir (see ¶ 0035).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Boswell as applied above, and further in view of Ryan.
In regard to claim 19, Qiu and Ryan teach:
19. The method according to claim 18, wherein the adjusting the weights bias value includes reducing the weights bias value by one-half of the input range of the matrix multiply accelerator. See Qiu and Ryan as indicated above with respect to claim 3.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Wu as applied above, and further in view of Boswell and Ryan.
In regard to claim 20, Qiu does not expressly disclose the claimed limitations. They are taught by Boswell and Ryan:
20. The method according to claim 1, wherein transforming further comprises:
converting signed outputs of the MMA to unsigned values for compatibility with the analog computation unit; and See Boswell, ¶ 0137, “The pre-processing may involve converting the elements from one format to the half-precision, floating-point value format. For example, input vectors {right arrow over (A)} & {right arrow over (B)} can be provided in 16-bit floating-point, 8-bit signed/unsigned integer, 16-bit signed/unsigned integer, 32-bit fixed-point format, etc. The conversion/encoding logic 1315 is configured to convert all input values to a half-precision, floating-point value format for compatibility with the rest of the datapath 1300.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Boswell’s conversion with Kovvuri’s accelerator in order to ensure data compatibility as suggested by Boswell.
subtracting one-half of an input bit range of the analog computation unit from the bias term or subtracting a dot product of a weight vector and a constant vector having elements equal to one-half of the input bit range. However, this is taught by Ryan. See Ryan, p. 1, e.g. “Since it's typical to select a bias equal to half the available range in magnitude, for 8 bits we 'll pick -127 as the bias.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to provide Qiu’s data values using Ryan’s number representation since the use of such representation is typical as suggested by Ryan.
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Qiu in view of Kovvuri, Ghodrati and Wu as applied above, and further in view of Boswell.
In regard to claim 27, Qiu also discloses:
27. The method according to [claim] 1, whereinthe transforming converts … outputs of a computation operation to … outputs by installing or replacing an activation function that shifts a signed output range into a positive-only range … . Qiu, p. 1224, section II.C, “By adding a learnable bias term, the output range of FReLU is [b, +∞), which is helpful to ensure efficient learning.”
Qiu does not expressly disclose: signed … unsigned … compatible with the analog MMA. This is taught by Ghodrati and Boswell. See Ghodrati, ¶ 0005, 0047 and 0097 as cited above. Also see Boswell col. 30, lines 20-25, “For example, input vectors
A
→
&
B
→
can be provided in 16-bit floating-point, 8-bit signed/unsigned integer, 16-bit signed/unsigned integer, 32-bit fixed-point format, etc. The conversion/encoding logic 1315 is configured to convert all input values to a half-precision, floating-point value format for compatibility with the rest of the datapath 1300.” It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Boswell’s conversion with Kovvuri’s accelerator in order to ensure data compatibility as suggested by Boswell.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James D Rutten whose telephone number is (571)272-3703. The examiner can normally be reached M-F 9:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached on (571)272-3768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/James D. Rutten/Primary Examiner, Art Unit 2121
Prosecution Insights