DETAILED ACTION
This Office action is responsive to the Request for Continued Examination (RCE) filed under 37 CFR §1.53(d) for the instant application on March 5, 2026. The Applicants have properly set forth the RCE, which has been entered into the application, and an examination on the merits follows herewith.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 2, 4, 5, 8-10, 12, 13, 16-19 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over the article entitled, “CxDNN: Hardware-software Compensation Methods for Deep Neural Networks on Resistive Crossbar Systems” by Jain et al. (“Jain”), over the article entitled, “Mitigating Adversarial Attack for Compute-in-Memory Accelerator Utilizing On-chip Finetune” by Huang et al. (“Huang”), over U.S. Patent Application Publication No. 2019/0189174 to Hu et al. (“Hu”), and also over U.S. Patent Application Publication No. 2021/0303982 to Lee et al. (“Lee”).
Regarding claim 1, Jain generally presents “CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs [deep neural networks] on crossbar systems by compensating for errors due to nonidealities, greatly mitigating the degradation in accuracy.” (Page 113:1). Like claimed, Jain particularly teaches a system comprising:
a processor and a resistive processing unit array coupled to the processor, the resistive processing unit array comprising an array of cells, the cells respectively comprising resistive memory devices which are programmable to store weight values (see e.g. section 3.1 “Resistive Crossbar System”: Jain describes a system comprising a host processor, a main memory and a resistive crossbar processing unit, which comprises a resistive crossbar array of synaptic elements that each comprise resistive memory devices that are programmable to store DNN weights:
Figure 2 (left side) depicts a representative resistive crossbar–based accelerator, viz., Resistive crossbar processing unit (XPU), connected to a host processor and main memory via a system bus. The host processor off-loads execution of compute kernels having vector-matrix multiplications (e.g., convolution and fully-connected layers in DNNs) to the XPU, which in turn realizes them using multiple Crossbar processing tiles (XPTs) and a scheduler. XPTs are composed of Resistive Crossbar Arrays (RCAs) that store weights and compute vector-matrix multiplications, local memories in the form of input and output buffers to store activations, a local bus, a controller to orchestrate various operations, and a special function unit (SFU) to execute other DNN operations such as ReLU, max pooling, and so on. Next, we describe RCAs, the fundamental compute units of resistive crossbar systems, in detail.
As shown in Figure 2 (right side), a resistive crossbar array is composed of a 2D array of synaptic elements, Analog-to-Digital Converters (ADCs), Digital-to-Analog Converters (DACs), registers, a decoder, a column MUX, a reduce unit, and write circuits. The synaptic elements are programmable resistors that store DNN weights as conductances. The number of available conductance levels determines the precision of the synaptic element. They can be realized using emerging NVMs technologies, viz., PCM, ReRAM, and spintronics [18, 28, 33]. RCA supports two main operations: (i) vector-matrix multiplication and (ii) programming. A vector-matrix multiplication is performed by driving all wordlines (WL) to analog voltages using DACs and sensing the resultant currents flowing through the columns (BL) using ADCs. The vector-matrix multiplication ideally realized in an RCA can be expressed as
I
o
u
t
i
d
e
a
l
=
V
i
n
i
d
e
a
l
*
G
i
d
e
a
l
, where
V
i
n
i
d
e
a
l
represents input voltages,
G
i
d
e
a
l
denotes programmed conductances, and
I
o
u
t
i
d
e
a
l
represents output currents. In contrast, the programming operations, i.e., write operations on synaptic elements, are performed row-wise, wherein the write circuitry applies the necessary currents and sets them to the desired conductance.
(Section 3.1 “Resistive Crossbar System”’; emphasis added).
PNG
media_image1.png
307
439
media_image1.png
Greyscale
Jain thus describes a system comprising a host processor and a resistive processing unit array, i.e. a resistive crossbar processing unit, which comprises an array of cells, i.e. synaptic elements, that each comprise resistive memory devices that are programmable to store DNN weight values.);
wherein the processor is configured to:
obtain a matrix comprising target weight values (Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
:
Figure 5 outlines the CxDNN compensation flow, which consists of a quantization and conversion algorithm, hardware-independent re-training, and hardware compensation. CxDNN takes a readily available floating-point (FP32) DNN as an input. It first quantizes the weights and activations of the FP32 network to match the precisions of the synaptic devices and ADCs/DACs, resulting in a fixed-point (FxP) DNN. Subsequently, it transforms the weights into conductances and activations to voltages to obtain a DNN for an “ideal” crossbar (with no device and circuit non-idealities other than limited precision), denoted
D
N
N
X
I
. CxDNN then uses a fast hardware-instance-independent re-training method to recover accuracy lost during the conversion process. Next, CxDNN maps the
D
N
N
X
I
to a resistive crossbar system that suffers from crossbar non-idealities (denoted
D
N
N
X
N
I
), which results in further degradation in accuracy. CxDNN overcomes this accuracy drop using a hardware compensation scheme. Note that
D
N
N
X
I
can also be obtained using end-to-end DNN training, i.e., used for designing FP32 DNNs. However, training is computationally very expensive, requiring exa-ops of compute and hence days to weeks to complete. In contrast, CxDNN designs
D
N
N
X
I
by converting FP32 models and utilizing a fast re-training method to recover accuracy loss during conversion with very few iterations. We next describe CxDNN’s software and hardware methods in detail.
(Section 4.1 “CxDNN: Approach and Overview”; emphasis added and footnote omitted).
PNG
media_image2.png
205
423
media_image2.png
Greyscale
The host processor thus obtains a matrix that comprises target weight values, i.e. that comprises conductances/weights of a DNN for an ideal crossbar,
D
N
N
X
I
.); and
program cells of the array of cells to store weight values in the resistive processing unit array, which correspond to respective target weight values of the matrix, and perform a calibration process using multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines of the resistive processing unit array (Jain discloses that the CxDNN methodology comprises using a “Compensation Factor” (CF) for each crossbar column, wherein the Compensation Factor is used to compensate for variance across crossbar columns of the resistive crossbar processing unit:
As mentioned in Section 4.1, DNNs executed on crossbar-based systems will suffer from accuracy degradation due to data-dependent and hardware-instance-specific non-idealities. CxDNN employs a hardware compensation method to overcome this degradation. Our hardware compensation is motivated by a key observation that errors in vector-matrix multiplications realized using resistive crossbars result from the cumulative effect of all non-idealities. It is not possible to isolate the effect of individual non-idealities on the executed vector-matrix multiplication, as all non-idealities kick in simultaneously (as described in Section 3.1). Therefore, we propose error compensation at the crossbar-level, where the outputs of the realized vector-matrix multiplications are compensated using Compensation Factors (CFs). CxDNN uses a separate compensation factor for each crossbar column of each crossbar instance, which allows many degrees of freedom in tuning the compensation process. The use of per-column compensation factors is motivated by the error characteristics in Figure 3(c) that show a significant variance across both crossbar instances and columns within an instance.
(Section 4.4 “Hardware Compensation”; emphasis added).
Jain further discloses that the CF for each column is identified through a calibration process that occurs after the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, are mapped to the resistive crossbar processing unit, whereby the calibration process entails: (1) the host processor sending predetermined inputs to resistive crossbar arrays (RCAs) of the resistive crossbar processing unit; (2) the RCAs performing vector-matrix multiplication using the predetermined inputs and the stored weights to obtain actual output vectors; and (3) the host processor computing the compensation factors using the actual outputs and ideal outputs:
CxDNN’s hardware compensation method consists of two phases—a calibration phase to determine CFs and a runtime phase to mitigate errors using these CFs. The host processor initiates the calibration phase after the
D
N
N
X
I
is mapped and programmed to the resistive crossbar system. The calibration phase consists of three main steps: In step 1, the host sends predetermined inputs (Inp) to each RCA in the resistive crossbar system. In step 2, RCAs perform vector-matrix multiplications using these calibration inputs (Inp) and stored weights to obtain the actual (erroneous) output vectors (Out-act) and subsequently send them to the host. In step 3, the host processor computes compensation factors using the actual outputs (Out-act) and the ideal outputs (Out-idl). The host processor could either store or generate the calibration inputs (Inp) and expected outputs (Out-idl). We generate the calibration inputs using a characterization dataset. We note that our method does not restrict the nature or size of the characterization dataset—it can be as large as required, subject to the characterization time being acceptable. In our experiments, we found that a small subset of the validation set was sufficient to compute compensation factors that led to the accuracy improvements reported in the article. Figure 10 depicts the CF computation process for an RCA using actual and ideal outputs (Out-idl). As shown in the figure, we compute a vector of CFs, one for each column. The computed CFs are then sent back to the respective crossbars to be stored locally in the compensation logic. While all RCAs execute vector-matrix multiplications in parallel, the host processor computes the compensation factor for each RCA sequentially. Therefore, the overall calibration time is dominated by the host processor. Further, the calibration time depends on the total number of RCAs utilized by the DNN. In our evaluation, the overall calibration time was 0.2 s to 1.3 s across our benchmark DNNs for a single core Intel-Atom C2350 as our host processor. We invoke the calibration phase whenever a new DNN is mapped to the XPU system and subsequently execute millions of inference operations during the runtime phase. We can also periodically invoke the calibration phase to cope with aging issues and drift in synaptic conductances. Aging and synaptic drifts are very slow phenomena, hence the calibration phase will only need to be performed rarely (e.g., once every 10 K–100 K inference operations). The execution time required for calibration is negligible when amortized over 10 K–100 K inference operations. We also note that the computed CFs are not exact but approximate factors that compensate the errors effectively with limited hardware overhead, as discussed next.
(Section 4.4 “Hardware Compensation”; emphasis added).
Accordingly, Jain is further considered to teach configuring the processor to (i) program cells of the array of cells to store weight values, i.e. the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, in the resistive processing unit array, which correspond to respective target weight values of the matrix, and (ii) perform a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data, i.e. actual output data, that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines, i.e. columns, of the resistive processing unit array.).
Jain thus teaches a system similar to that of claim 1. Jain, however, does not disclose or suggests that the calibration process is an iterative process, wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the target weight values of the matrix based on the determined error for each output line, and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values, as is required by claim 1. Jain also does not disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 1.
Similar to Jain, Huang describes a compute-in-memory architecture that comprises a processing unit array comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values:
As DNNs are generally data and compute intensive, frequent data movements between logic and memory units limit the energy efficiency on traditional Von Neumann architecture. In recent years, there are increasingly efforts on developing specific hardware accelerators to run large-scale DNN models from the cloud to the edge. For example, systolic architecture such as TPU [5] employs many digital multiply and accumulate (MAC) engines close to a large global buffer (i.e., SRAM) to reduce the cost of data movement. As a more aggressive approach, compute-in-memory (CIM) architecture [4] merges the computation directly into the memory sub-arrays that ideally addresses the memory-wall problem. The weights of a DNN model could be mapped as the conductance of the memory cells in the sub-array, while the input vector is loaded in parallel as the voltage to the rows, then the multiplication is done in analog fashion, and the current summation along columns represents weighted sum. In principle, CIM could be implemented by different device technologies. SRAM with modified bit-cell and array periphery could enable parallel access as demonstrated in recent silicon prototype chips [6]. Emerging non-volatile memory (eNVM) technologies also provide promising solutions due to a smaller cell size and potential of multi-bit per cell, yielding a higher integration density at the same technology node [7]. Besides, because of the non-volatile nature and near-zero leakage, the eNVM-based CIM is more attractive to edge devices. No matter which kind of memory technologies is used, ADC is commonly essential as an important part of periphery circuitry to convert the analog partial sum back to digital signal for further processing. In other words, CIM is essentially mixed-signal compute, thus the variations are unavoidable. As reported in prior work [8], inference accuracy measured in CIM prototypes generally is degraded from the software baseline. The primary variation sources include the cell-to-cell variation for eNVMs and the intrinsic ADC offset. Cell-to-cell variation could be minimized by iterative write-verify technique with tolerable overhead for inference engine [9]. A more critical challenge is the intrinsic ADC offset introduced by the manufacturing process variation. As a result, the ADC offset may noticeably degrade the inference accuracy and cause different chip instances having different inference results even for the same input. It is noted that when ADC offset introduces quantization error because of the process variation, these offset patterns are static once the chip is fabricated.
(Section I “Introduction”; emphasis added).
The crossbar nature of memory array is a natural substrate for implementing VMM in a highly parallel manner. As shown in Fig. 1, the crossbar array consists of perpendicular rows and columns with the memory cell located at each cross-point. Weights in the filters are mapped as the content of the cells. The VMM operation is performed as follows: read voltages representing the input feature map are applied to all the rows so that the read voltages are multiplied by the memory cells at each cross-point. The current through each device is summed up along columns. Different columns represent filters for different output channels, who should see the same input thus all the columns work at the same time in parallel. Typically, ADCs are needed at the end of the column to convert the analog current to the digital output so that the subsequent processing such as activation and pooling could be performed in the digital domain. In principle, VMM could be done in fully parallel fashion if asserting all the rows and all the columns simultaneously. In practice, multiple rows/columns could be partially turned on due to the sensing resolution of ADCs or the mismatch of column pitch to the peripheral circuitry’s dimension.
(Section II.A “Pricinple [sic] of CIM”; emphasis added).
Huang particularly teaches fine-tuning the weights of the processing unit array to reduce a variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array:
Compute-in-memory (CIM) has been proposed to accelerate the convolution neural network (CNN) computation by implementing parallel multiply and accumulation in analog domain. However, the subsequent processing is still preferred to be performed in digital domain. This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples. Our evaluation results show that by adapting the model weights to the specific ADC offset pattern to each chip, the transferability of the adversarial attack is suppressed. For a chip being attacked by the C&W method, the classification for CIFAR-10 dataset will drop to almost 0%. However, when applying the similarly generated adversarial examples to other chips, the accuracy could still maintain more than 62% and 85% accuracy for VGG-8 and DenseNet-40, respectively.
(Abstract; emphasis added).
In this work, we leverage the ADC offset pattern (which is believed to be detrimental to the inference accuracy) but finetune the model weights to take its advantage against the adversarial attack on the CIM accelerator. In our evaluation, we find that the accuracy drop could be compensated by finetuning DNN parameters that adapt to ADC offset. This finetune, while recovering the inference accuracy, makes the DNN parameters slightly different from chip to chip, which brings us a byproduct: the chip will be robust to the adversarial examples generated by attacking other chips or software baseline. Explicitly, even if the adversary attacks one chip instance by manipulating the adversarial input, he/she could not use the same adversarial input to attack all the other chip instances due to the uniqueness of the DNN model for each chip.
(Section I “Introduction”).
In this work, instead of trying to defend against adversarial examples of a certain model in software, we aim to reduce the transferability of the adversarial examples among actual chips. It works like the aforementioned software defense method of introducing randomness into the network parameters. For CIM architecture that employs ADCs, there are intrinsic process variations that will introduce quantization error. As shown in previous prototype chip measurement results, even with precisely designed ADCs, the accuracy will be low if the ADCs all share the same references [24]. To achieve high accuracy, the references of each ADC need to be adjusted independently. This requirement for ADC reference adjustment and independent reference for each ADC brings additional hardware overhead. Alternatively, we could adjust the weights with several retrain epoch, namely “finetune” process. When the model is adapted to the ADC offset pattern, the inference accuracy could be recovered. The overhead of model finetune in software is much less than the implementation of adjustable ADC references on-chip. Now we could take advantages of the model finetune to reduce the transferability of the adversarial examples from software baseline to actual chips, and from one chip to another.
(Section II.C “Adversarial Defense”).
We now discuss the on-chip/off-chip hybrid finetune procedure to mitigate adversarial attack from the hardware standpoint. During the retraining, the feedforward propagation (inference) is first performed on-chip, and then the backpropagation and weight update are done off-chip by software. The detailed process is as follows: we will run the inference on a specific chip that captures its specific ADC offset pattern, then the prediction of the inference will be compared with the ideal label for the loss function; after obtaining the estimated loss, weights are updated through backpropagation in software; finally, the memory cells will be reprogramed to the new weights possibly with write-verify. In our evaluation, the partial sum will sample the ADC offset from the estimated distribution as in Fig. 3(a), and the distorted partial sum will be used as output feature map and saved for error and gradient calculation. The backpropagation and weight update are all directly use floating-point calculation as done in software.
(Section III.B “Procedure of weights finetune”).
We evaluate the proposed hybrid finetune defense method with VGG-8 and DenseNet-40 networks for CIFAR-10 dataset. The precision setting is 8-bit activation and 2-bit weight for VGG-8, and 8-bit activation and 8-bit weight for DenseNet-40. The software baseline accuracy is ~92% for both networks. For the weight finetune process, the batch size of retrain is 200, which means there are 250 iterations to finish the finetune in one epoch. Fig. 5 (a) & (b) shows the retrain curve of one specific chip that implements VGG-8 which uses Flash-ADC and SARADC, respectively. Chip with either Flash-ADC or SAR-ADC could recover the accuracy, however Flash-ADC has less initial accuracy drop and easier to be retrained to recover the high accuracy. This is consistent with the analysis in Section Ⅲ on the possible compensation of SA offsets for Flash-ADC. It is also seen that as the W/L decreases, it will be more difficult to retrain the model to recover the accuracy under process variations. When the W/L is small, which means the sense pass rate is also low, the accuracy may could not be fully recovered. It needed to be pointed out that the W/L reported here appears high since we use the minimum length (
L
m
i
n
) as the L in our simulation. Generally, two to three times of
L
m
i
n
will be used in the analog circuit to avoid very large process variation. We did not optimize the ADC with advanced offset cancellation techniques. Here our goal is just to show that by changing W/L, we could achieve different degrees of the process variation. Fig. 5 (c) & (d) presents the accuracy distribution of several retrain tests before finetune and after finetune collected from multiple chips, respectively. From the plot, we could observe that the accuracy recovery from finetune is generally achievable.
(Section IV. “Evaluation Results”).
As suggested by the above excerpts, such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values.
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain and Huang before the effective filing date of the claimed invention, to modify the calibration process taught by Jain so as to additionally or alternatively comprise such fine-tuning like taught by Huang, which would entail an iterative process that uses output data generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error between expected output data and the generated output data, iteratively adjusting the target weight values of the matrix based on the determined error, and reprogramming the stored weight values of the matrix in the processing unit array based on the respective adjusted weight values. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array without requiring additional hardware modifications, as is evident from Huang (see e.g. the first paragraph of section III.A “ADC offset modeling”). Accordingly, Jain and Huang are considered to teach a system similar to that of claim 1, but do not explicitly disclose that each iteration of the calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is further required by claim 1. Jain and Huang also do not explicitly disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 1.
Similar to Jain and Huang, Hu describes a compute-in-memory architecture that comprises a processing unit array (i.e. a crossbar array) comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values (see e.g. paragraphs 0016 and 0022-0023). Hu particularly teaches increasing computational accuracy in the processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061). Hu further teaches that the error that is determined for a given output line comprises a difference between a target offset value (i.e. ideal output signal or value) and a measured offset value (i.e. actual output signal or value) associated with the multiply-and-accumulate distribution data obtained for the given output line (see e.g. paragraphs 0060-0061).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang and Hu before the effective filing date of the claimed invention, to modify the calibration process taught by Jain and Huang such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on the error for each output line, and wherein the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the resistive processing unit array, as is suggested by Hu (see e.g. paragraph 0011). Accordingly, Jain, Huang and Hu are considered to teach a system similar to that of claim 1, but do not explicitly teach that the target offset value corresponds to a multiply-and-accumulate value of zero, as is required by claim 1.
Lee nevertheless describes a process for calibrating a resistive processing unit array (i.e. a MAC – an analog multiplier and accumulator) by inputting various values of calibration data to the resistive processing unit array, and by determining a gain and offset based on the output values of the resistive processing unit array (see e.g. paragraphs 0006, 0008, 0014, 0023, 0054 and 0056). Regarding the claimed invention, Lee particularly suggests that the range of calibration input data can particularly comprise an input value of zero, whereby the associated target offset value would correspond to a multiply-and-accumulate value of zero (see e.g. paragraphs 0077-0079).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu and Lee before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to employ a range of input values like taught by Lee, including an input value of zero, which would have a target offset value of zero. It would have been advantageous to one of ordinary skill to utilize such a combination because it would ensure that the processing unit array can more accurately process a plurality of different input values, as is suggested by Lee (see e.g. paragraphs 0006, 0014 and 0054). Accordingly, Jain, Huang, Hu and Lee are considered to teach, to one of ordinary skill in the art, a system like that of claim 1.
As per claim 2, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such a calibration process can converge the measured offset values of multiply-and-accumulate distribution data of respective output lines to the target offset value (see e.g. the abstract, which recites: “This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples.”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 2.
As per claim 4, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such fine-tuning would reduce the respective spreads of multiply-and-accumulate distribution data (i.e. make the outputs closer to the ideal output), which are output from the respective output lines (see e.g. section III.V “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 4.
As per claim 5, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such a calibration process entails: (i) applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which results from performing analog multiplication operations by multiplying each of the known input vectors by the matrix in the processing unit array (i.e. performing feedforward propagation/inference); (ii) determining, for a given output line of the processing unit array, an offset associated with the generated set of multiply-and-accumulate distribution data for the given output line; (iii) determining, for the given output line, an error between the determined offset of the generated set of multiply-and-accumulate distribution data, and a target offset associated with a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target weight values of the matrix; (iv) adjusting the target weight values of the matrix which correspond to the stored weight values of the given output line, to counteract the error between the determined offset and the target offset; and (v) reprogramming the stored weight values of the given output line of the processing unit array based on the adjusted weight values (see e.g. section III.B “Procedure of weights finetune and section IV “Evaluation Results”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 5.
As per claim 8, Jain teaches that the obtained matrix comprises one of a computational matrix utilized to perform matrix computations for a linear system, and a trained synaptic weight matrix of a trained artificial neural network to perform inference processing (see e.g. section 4.1 “CxDNN: Approach and Overview:” Jain suggests that the obtained matrix can comprise a trained synaptic matrix of a trained artificial neural network, i.e. a DNN, to perform inference processing.). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 8.
Regarding claim 9, Jain generally presents “CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs [deep neural networks] on crossbar systems by compensating for errors due to nonidealities, greatly mitigating the degradation in accuracy.” (Page 113:1). Like claimed, Jain particularly teaches (i) obtaining a matrix comprising target weight values; (ii) programming an array of cells of a resistive processing unit array to store weight values which correspond to respective target weight values of the matrix; and (iii) performing a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines of the resistive processing unit array (Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
:
Figure 5 outlines the CxDNN compensation flow, which consists of a quantization and conversion algorithm, hardware-independent re-training, and hardware compensation. CxDNN takes a readily available floating-point (FP32) DNN as an input. It first quantizes the weights and activations of the FP32 network to match the precisions of the synaptic devices and ADCs/DACs, resulting in a fixed-point (FxP) DNN. Subsequently, it transforms the weights into conductances and activations to voltages to obtain a DNN for an “ideal” crossbar (with no device and circuit non-idealities other than limited precision), denoted
D
N
N
X
I
. CxDNN then uses a fast hardware-instance-independent re-training method to recover accuracy lost during the conversion process. Next, CxDNN maps the
D
N
N
X
I
to a resistive crossbar system that suffers from crossbar non-idealities (denoted
D
N
N
X
N
I
), which results in further degradation in accuracy. CxDNN overcomes this accuracy drop using a hardware compensation scheme. Note that
D
N
N
X
I
can also be obtained using end-to-end DNN training, i.e., used for designing FP32 DNNs. However, training is computationally very expensive, requiring exa-ops of compute and hence days to weeks to complete. In contrast, CxDNN designs
D
N
N
X
I
by converting FP32 models and utilizing a fast re-training method to recover accuracy loss during conversion with very few iterations. We next describe CxDNN’s software and hardware methods in detail.
(Section 4.1 “CxDNN: Approach and Overview”; emphasis added and footnote omitted).
PNG
media_image2.png
205
423
media_image2.png
Greyscale
The set of conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, is considered a matrix comprising target weight values. Jain further discloses that the CxDNN methodology comprises using a “Compensation Factor” (CF) for different crossbar columns, wherein the Compensation Factor is used to compensate for variance across crossbar columns of a resistive crossbar processing unit:
As mentioned in Section 4.1, DNNs executed on crossbar-based systems will suffer from accuracy degradation due to data-dependent and hardware-instance-specific non-idealities. CxDNN employs a hardware compensation method to overcome this degradation. Our hardware compensation is motivated by a key observation that errors in vector-matrix multiplications realized using resistive crossbars result from the cumulative effect of all non-idealities. It is not possible to isolate the effect of individual non-idealities on the executed vector-matrix multiplication, as all non-idealities kick in simultaneously (as described in Section 3.1). Therefore, we propose error compensation at the crossbar-level, where the outputs of the realized vector-matrix multiplications are compensated using Compensation Factors (CFs). CxDNN uses a separate compensation factor for each crossbar column of each crossbar instance, which allows many degrees of freedom in tuning the compensation process. The use of per-column compensation factors is motivated by the error characteristics in Figure 3(c) that show a significant variance across both crossbar instances and columns within an instance.
(Section 4.4 “Hardware Compensation”; emphasis added).
Jain further discloses that the CF for each column is identified through a calibration process that occurs after the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, are mapped to the resistive crossbar processing unit, whereby the calibration process entails: (1) a host processor sending predetermined inputs to resistive crossbar arrays (RCAs) of the resistive crossbar processing unit; (2) the RCAs performing vector-matrix multiplication using the predetermined inputs and the stored weights to obtain actual output vectors; and (3) the host processor computing the compensation factors using the actual outputs and ideal outputs:
CxDNN’s hardware compensation method consists of two phases—a calibration phase to determine CFs and a runtime phase to mitigate errors using these CFs. The host processor initiates the calibration phase after the
D
N
N
X
I
is mapped and programmed to the resistive crossbar system. The calibration phase consists of three main steps: In step 1, the host sends predetermined inputs (Inp) to each RCA in the resistive crossbar system. In step 2, RCAs perform vector-matrix multiplications using these calibration inputs (Inp) and stored weights to obtain the actual (erroneous) output vectors (Out-act) and subsequently send them to the host. In step 3, the host processor computes compensation factors using the actual outputs (Out-act) and the ideal outputs (Out-idl). The host processor could either store or generate the calibration inputs (Inp) and expected outputs (Out-idl). We generate the calibration inputs using a characterization dataset. We note that our method does not restrict the nature or size of the characterization dataset—it can be as large as required, subject to the characterization time being acceptable. In our experiments, we found that a small subset of the validation set was sufficient to compute compensation factors that led to the accuracy improvements reported in the article. Figure 10 depicts the CF computation process for an RCA using actual and ideal outputs (Out-idl). As shown in the figure, we compute a vector of CFs, one for each column. The computed CFs are then sent back to the respective crossbars to be stored locally in the compensation logic. While all RCAs execute vector-matrix multiplications in parallel, the host processor computes the compensation factor for each RCA sequentially. Therefore, the overall calibration time is dominated by the host processor. Further, the calibration time depends on the total number of RCAs utilized by the DNN. In our evaluation, the overall calibration time was 0.2 s to 1.3 s across our benchmark DNNs for a single core Intel-Atom C2350 as our host processor. We invoke the calibration phase whenever a new DNN is mapped to the XPU system and subsequently execute millions of inference operations during the runtime phase. We can also periodically invoke the calibration phase to cope with aging issues and drift in synaptic conductances. Aging and synaptic drifts are very slow phenomena, hence the calibration phase will only need to be performed rarely (e.g., once every 10 K–100 K inference operations). The execution time required for calibration is negligible when amortized over 10 K–100 K inference operations. We also note that the computed CFs are not exact but approximate factors that compensate the errors effectively with limited hardware overhead, as discussed next.
(Section 4.4 “Hardware Compensation”; emphasis added).
Accordingly, Jain is further considered to teach (i) programming an array of cells of a resistive processing unit array to store weight values, i.e. the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, in the resistive processing unit array, which correspond to respective target weight values of the matrix, and (ii) performing a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data, i.e. actual output data, that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines, i.e. columns, of the resistive processing unit array.). Jain suggests that such tasks are performed via a host processor, understandably by executing program instructions (see e.g. section 4.4 “Hardware Compensation.”). The computer readable storage media necessary to store the program instructions to implement the above-described teachings of Jain is considered a computer program product similar to that of claim 9. Jain, however, does not disclose or suggests that the calibration process is an iterative process, wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the target weight values of the matrix based on the determined errors for each output line, and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values, as is required by claim 9. Jain also does not disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 9.
Nevertheless, like noted above (see e.g. the rejection for claim 1), Huang teaches fine-tuning the weights of a processing unit array to reduce a variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array; such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values.
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain and Huang before the effective filing date of the claimed invention, to modify the calibration process taught by Jain so as to additionally or alternatively comprise such fine-tuning like taught by Huang, which would entail an iterative process that uses output data generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error between expected output data and the generated output data, iteratively adjusting the target weight values of the matrix based on the determined error, and reprogramming the stored weight values of the matrix in the processing unit array based on the respective adjusted weight values. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array without requiring additional hardware modifications, as is evident from Huang (see e.g. the first paragraph of section III.A “ADC offset modeling”). Accordingly, Jain and Huang are considered to teach a computer program product similar to that of claim 9, but do not explicitly disclose that each iteration of the calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is further required by claim 9. Jain and Huang also do not explicitly disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 9.
Nevertheless, like noted above, Hu similarly describes a compute-in-memory architecture that comprises a processing unit array (i.e. a crossbar array) comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values (see e.g. paragraphs 0016 and 0022-0023). Hu particularly teaches increasing computational accuracy in the processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061). Hu further teaches that the error that is determined for a given output line comprises a difference between a target offset value (i.e. ideal output signal or value) and a measured offset value (i.e. actual output signal or value) associated with the multiply-and-accumulate distribution data obtained for the given output line (see e.g. paragraphs 0060-0061).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang and Hu before the effective filing date of the claimed invention, to modify the calibration process taught by Jain and Huang such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, and wherein the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the resistive processing unit array, as is suggested by Hu (see e.g. paragraph 0011). Accordingly, Jain, Huang and Hu are considered to teach a computer program product similar to that of claim 9, but do not explicitly teach that the target offset value corresponds to a multiply-and-accumulate value of zero, as is required by claim 9.
Lee nevertheless describes a process for calibrating a resistive processing unit array (i.e. a MAC – an analog multiplier and accumulator) by inputting various values of calibration data to the resistive processing unit array, and by determining a gain and offset based on the output values of the resistive processing unit array (see e.g. paragraphs 0006, 0008, 0014, 0023, 0054 and 0056). Regarding the claimed invention, Lee particularly suggests that the range of calibration input data can particularly comprise an input value of zero, whereby the associated target offset value would correspond to a multiply-and-accumulate value of zero (see e.g. paragraphs 0077-0079).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu and Lee before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to employ a range of input values like taught by Lee, including an input value of zero, which would have a target offset value of zero. It would have been advantageous to one of ordinary skill to utilize such a combination because it would ensure that the processing unit array can more accurately process a plurality of different input values, as is suggested by Lee (see e.g. paragraphs 0006, 0014 and 0054). Accordingly, Jain, Huang, Hu and Lee are considered to teach, to one of ordinary skill in the art, a computer program product like that of claim 9.
As per claim 10, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such a calibration process can converge respective offsets of multiply-and-accumulate distribution data, which are output from respect output lines (i.e. ADCs), to a target offset (see e.g. the abstract, which recites: “This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples.”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a computer program product like that of claim 10.
As per claim 12, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such fine-tuning would reduce the respective spreads of the multiply-and-accumulate distribution data (i.e. make the outputs closer to the ideal output), which are output from the respective output lines (see e.g. section III.V “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a computer program product like that of claim 12.
As per claim 13, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such a calibration process entails: (i) applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which results from performing analog multiplication operations by multiplying each of the known input vectors by the matrix in the processing unit array (i.e. performing feedforward propagation/inference); (ii) determining, for a given output line of the processing unit array, an offset associated with the generated set of multiply-and-accumulate distribution data for the given output line; (iii) determining, for the given output line, an error between the determined offset of the generated set of multiply-and-accumulate distribution data, and a target offset associated with a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target weight values of the matrix; (iv) adjusting the target weight values of the matrix which correspond to the stored weight values of the given output line, to counteract the error between the determined offset and the target offset; and (v) reprogramming the stored weight values of the given output line of the processing unit array based on the adjusted weight values (see e.g. section III.B “Procedure of weights finetune and section IV “Evaluation Results”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach computer program product like that of claim 13.
As per claim 16, Jain teaches that the obtained matrix comprises one of a computational matrix utilized to perform matrix computations for a linear system, and a trained synaptic weight matrix of a trained artificial neural network to perform inference processing (see e.g. section 4.1 “CxDNN: Approach and Overview:” Jain suggests that the obtained matrix can comprise a trained synaptic matrix of a trained artificial neural network, i.e. a DNN, to perform inference processing.). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a computer program product like that of claim 16.
Regarding claim 17, Jain generally presents “CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs [deep neural networks] on crossbar systems by compensating for errors due to nonidealities, greatly mitigating the degradation in accuracy.” (Page 113:1). Like claimed, Jain particularly teaches a system comprising:
a neuromorphic computing system comprising a resistive processing unit array which comprises an array of resistive processing unit cells, a plurality of input lines extending in a first direction across the resistive processing unit array, a plurality of output lines extending in a second direction across the resistive processing unit array, wherein each resistive processing unit cell is coupled at an intersection of one of the input lines and one of the output lines, and wherein the resistive processing unit cells respectively comprise resistive memory devices which are programmable to store weight values (see e.g. section 3.1 “Resistive Crossbar System”: Jain describes a system comprising a host processor, a main memory and a resistive crossbar processing unit comprising a resistive crossbar array of synaptic elements, a plurality of input lines extending in a first direction across the resistive crossbar array, a plurality of output lines, i.e. columns, extending in a second direction across the resistive crossbar array, wherein each synaptic element is coupled at an intersection of one of the input lines and one of the output lines, and wherein each of the synaptic elements comprises resistive memory devices which are programmable to store weight values:
Figure 2 (left side) depicts a representative resistive crossbar–based accelerator, viz., Resistive crossbar processing unit (XPU), connected to a host processor and main memory via a system bus. The host processor off-loads execution of compute kernels having vector-matrix multiplications (e.g., convolution and fully-connected layers in DNNs) to the XPU, which in turn realizes them using multiple Crossbar processing tiles (XPTs) and a scheduler. XPTs are composed of Resistive Crossbar Arrays (RCAs) that store weights and compute vector-matrix multiplications, local memories in the form of input and output buffers to store activations, a local bus, a controller to orchestrate various operations, and a special function unit (SFU) to execute other DNN operations such as ReLU, max pooling, and so on. Next, we describe RCAs, the fundamental compute units of resistive crossbar systems, in detail.
As shown in Figure 2 (right side), a resistive crossbar array is composed of a 2D array of synaptic elements, Analog-to-Digital Converters (ADCs), Digital-to-Analog Converters (DACs), registers, a decoder, a column MUX, a reduce unit, and write circuits. The synaptic elements are programmable resistors that store DNN weights as conductances. The number of available conductance levels determines the precision of the synaptic element. They can be realized using emerging NVMs technologies, viz., PCM, ReRAM, and spintronics [18, 28, 33]. RCA supports two main operations: (i) vector-matrix multiplication and (ii) programming. A vector-matrix multiplication is performed by driving all wordlines (WL) to analog voltages using DACs and sensing the resultant currents flowing through the columns (BL) using ADCs. The vector-matrix multiplication ideally realized in an RCA can be expressed as
I
o
u
t
i
d
e
a
l
=
V
i
n
i
d
e
a
l
*
G
i
d
e
a
l
, where
V
i
n
i
d
e
a
l
represents input voltages,
G
i
d
e
a
l
denotes programmed conductances, and
I
o
u
t
i
d
e
a
l
represents output currents. In contrast, the programming operations, i.e., write operations on synaptic elements, are performed row-wise, wherein the write circuitry applies the necessary currents and sets them to the desired conductance.
(Section 3.1 “Resistive Crossbar System”’; emphasis added).
PNG
media_image1.png
307
439
media_image1.png
Greyscale
The resistive crossbar array described by Jain is considered a resistive processing unit array like claimed.);
a digital processing system, coupled to the neuromorphic computing system, wherein the digital processing system comprises one or more processors, and memory to store program instructions that are executed by the one or more processors to configure the digital processing system to control operations of the neuromorphic computing system (Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
:
Figure 5 outlines the CxDNN compensation flow, which consists of a quantization and conversion algorithm, hardware-independent re-training, and hardware compensation. CxDNN takes a readily available floating-point (FP32) DNN as an input. It first quantizes the weights and activations of the FP32 network to match the precisions of the synaptic devices and ADCs/DACs, resulting in a fixed-point (FxP) DNN. Subsequently, it transforms the weights into conductances and activations to voltages to obtain a DNN for an “ideal” crossbar (with no device and circuit non-idealities other than limited precision), denoted
D
N
N
X
I
. CxDNN then uses a fast hardware-instance-independent re-training method to recover accuracy lost during the conversion process. Next, CxDNN maps the
D
N
N
X
I
to a resistive crossbar system that suffers from crossbar non-idealities (denoted
D
N
N
X
N
I
), which results in further degradation in accuracy. CxDNN overcomes this accuracy drop using a hardware compensation scheme. Note that
D
N
N
X
I
can also be obtained using end-to-end DNN training, i.e., used for designing FP32 DNNs. However, training is computationally very expensive, requiring exa-ops of compute and hence days to weeks to complete. In contrast, CxDNN designs
D
N
N
X
I
by converting FP32 models and utilizing a fast re-training method to recover accuracy loss during conversion with very few iterations. We next describe CxDNN’s software and hardware methods in detail.
(Section 4.1 “CxDNN: Approach and Overview”; emphasis added and footnote omitted).
PNG
media_image2.png
205
423
media_image2.png
Greyscale
The computer system necessary for generating the DNN and for quantizing and transforming the weights of the DNN is considered a digital processing system like claimed, which is coupled to the neuromorphic computing system and necessarily comprises one or more processors and memory to store program instructions that are executed by the one or more processors to configure the digital processing system to control operations of the neuromorphic computing system), wherein the digital processing system is configured to:
train an artificial neural network in a digital domain, wherein the trained artificial neural network comprises at least one trained synaptic weight matrix with target synaptic weight values that are learned (Like noted above, Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
. The input DNN is understood to be an artificial neural network trained in the digital domain, and comprises at least one trained synaptic weight matrix with target synaptic weight values that are learned.); and
program the array of resistive processing unit cells to store weight values which correspond to respective target synaptic weight values of the trained synaptic weight matrix, and perform a calibration process using multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines of the resistive processing unit array (Jain discloses that the CxDNN methodology comprises using a “Compensation Factor” (CF) for each crossbar column, wherein the Compensation Factor is used to compensate for variance across crossbar columns of the resistive crossbar processing unit:
As mentioned in Section 4.1, DNNs executed on crossbar-based systems will suffer from accuracy degradation due to data-dependent and hardware-instance-specific non-idealities. CxDNN employs a hardware compensation method to overcome this degradation. Our hardware compensation is motivated by a key observation that errors in vector-matrix multiplications realized using resistive crossbars result from the cumulative effect of all non-idealities. It is not possible to isolate the effect of individual non-idealities on the executed vector-matrix multiplication, as all non-idealities kick in simultaneously (as described in Section 3.1). Therefore, we propose error compensation at the crossbar-level, where the outputs of the realized vector-matrix multiplications are compensated using Compensation Factors (CFs). CxDNN uses a separate compensation factor for each crossbar column of each crossbar instance, which allows many degrees of freedom in tuning the compensation process. The use of per-column compensation factors is motivated by the error characteristics in Figure 3(c) that show a significant variance across both crossbar instances and columns within an instance.
(Section 4.4 “Hardware Compensation”; emphasis added).
Jain further discloses that the CF for each column is identified through a calibration process that occurs after the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, are mapped to the resistive crossbar processing unit, whereby the calibration process entails: (1) the host processor sending predetermined inputs to resistive crossbar arrays (RCAs) of the resistive crossbar processing unit; (2) the RCAs performing vector-matrix multiplication using the predetermined inputs and the stored weights to obtain actual output vectors; and (3) the host processor computing the compensation factors using the actual outputs and ideal outputs:
CxDNN’s hardware compensation method consists of two phases—a calibration phase to determine CFs and a runtime phase to mitigate errors using these CFs. The host processor initiates the calibration phase after the
D
N
N
X
I
is mapped and programmed to the resistive crossbar system. The calibration phase consists of three main steps: In step 1, the host sends predetermined inputs (Inp) to each RCA in the resistive crossbar system. In step 2, RCAs perform vector-matrix multiplications using these calibration inputs (Inp) and stored weights to obtain the actual (erroneous) output vectors (Out-act) and subsequently send them to the host. In step 3, the host processor computes compensation factors using the actual outputs (Out-act) and the ideal outputs (Out-idl). The host processor could either store or generate the calibration inputs (Inp) and expected outputs (Out-idl). We generate the calibration inputs using a characterization dataset. We note that our method does not restrict the nature or size of the characterization dataset—it can be as large as required, subject to the characterization time being acceptable. In our experiments, we found that a small subset of the validation set was sufficient to compute compensation factors that led to the accuracy improvements reported in the article. Figure 10 depicts the CF computation process for an RCA using actual and ideal outputs (Out-idl). As shown in the figure, we compute a vector of CFs, one for each column. The computed CFs are then sent back to the respective crossbars to be stored locally in the compensation logic. While all RCAs execute vector-matrix multiplications in parallel, the host processor computes the compensation factor for each RCA sequentially. Therefore, the overall calibration time is dominated by the host processor. Further, the calibration time depends on the total number of RCAs utilized by the DNN. In our evaluation, the overall calibration time was 0.2 s to 1.3 s across our benchmark DNNs for a single core Intel-Atom C2350 as our host processor. We invoke the calibration phase whenever a new DNN is mapped to the XPU system and subsequently execute millions of inference operations during the runtime phase. We can also periodically invoke the calibration phase to cope with aging issues and drift in synaptic conductances. Aging and synaptic drifts are very slow phenomena, hence the calibration phase will only need to be performed rarely (e.g., once every 10 K–100 K inference operations). The execution time required for calibration is negligible when amortized over 10 K–100 K inference operations. We also note that the computed CFs are not exact but approximate factors that compensate the errors effectively with limited hardware overhead, as discussed next.
(Section 4.4 “Hardware Compensation”; emphasis added).
Accordingly, Jain is further considered to teach configuring the digital processing system to (i) program the resistive crossbar array to store synaptic weight values, i.e. the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, which correspond to respective target synaptic weight values of the trained synaptic weight matrix, i.e. the DNN, and (ii) perform a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data, i.e. actual output data, that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines, i.e. columns, of the resistive processing unit array.).
Jain thus teaches a system similar to that of claim 17. Jain, however, does not disclose or suggests that the calibration process is an iterative process, wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the synaptic target weight values of the trained synaptic weight matrix based on the determined errors for each output line, and reprogramming the stored synaptic weight values of the synaptic weight matrix in the resistive processing unit array based on the respective adjusted target synaptic weight values, as is required by claim 17. Jain also does not disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 17.
Similar to Jain, Huang describes a compute-in-memory architecture that comprises a processing unit array comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values:
As DNNs are generally data and compute intensive, frequent data movements between logic and memory units limit the energy efficiency on traditional Von Neumann architecture. In recent years, there are increasingly efforts on developing specific hardware accelerators to run large-scale DNN models from the cloud to the edge. For example, systolic architecture such as TPU [5] employs many digital multiply and accumulate (MAC) engines close to a large global buffer (i.e., SRAM) to reduce the cost of data movement. As a more aggressive approach, compute-in-memory (CIM) architecture [4] merges the computation directly into the memory sub-arrays that ideally addresses the memory-wall problem. The weights of a DNN model could be mapped as the conductance of the memory cells in the sub-array, while the input vector is loaded in parallel as the voltage to the rows, then the multiplication is done in analog fashion, and the current summation along columns represents weighted sum. In principle, CIM could be implemented by different device technologies. SRAM with modified bit-cell and array periphery could enable parallel access as demonstrated in recent silicon prototype chips [6]. Emerging non-volatile memory (eNVM) technologies also provide promising solutions due to a smaller cell size and potential of multi-bit per cell, yielding a higher integration density at the same technology node [7]. Besides, because of the non-volatile nature and near-zero leakage, the eNVM-based CIM is more attractive to edge devices. No matter which kind of memory technologies is used, ADC is commonly essential as an important part of periphery circuitry to convert the analog partial sum back to digital signal for further processing. In other words, CIM is essentially mixed-signal compute, thus the variations are unavoidable. As reported in prior work [8], inference accuracy measured in CIM prototypes generally is degraded from the software baseline. The primary variation sources include the cell-to-cell variation for eNVMs and the intrinsic ADC offset. Cell-to-cell variation could be minimized by iterative write-verify technique with tolerable overhead for inference engine [9]. A more critical challenge is the intrinsic ADC offset introduced by the manufacturing process variation. As a result, the ADC offset may noticeably degrade the inference accuracy and cause different chip instances having different inference results even for the same input. It is noted that when ADC offset introduces quantization error because of the process variation, these offset patterns are static once the chip is fabricated.
(Section I “Introduction”; emphasis added).
The crossbar nature of memory array is a natural substrate for implementing VMM in a highly parallel manner. As shown in Fig. 1, the crossbar array consists of perpendicular rows and columns with the memory cell located at each cross-point. Weights in the filters are mapped as the content of the cells. The VMM operation is performed as follows: read voltages representing the input feature map are applied to all the rows so that the read voltages are multiplied by the memory cells at each cross-point. The current through each device is summed up along columns. Different columns represent filters for different output channels, who should see the same input thus all the columns work at the same time in parallel. Typically, ADCs are needed at the end of the column to convert the analog current to the digital output so that the subsequent processing such as activation and pooling could be performed in the digital domain. In principle, VMM could be done in fully parallel fashion if asserting all the rows and all the columns simultaneously. In practice, multiple rows/columns could be partially turned on due to the sensing resolution of ADCs or the mismatch of column pitch to the peripheral circuitry’s dimension.
(Section II.A “Pricinple [sic] of CIM”; emphasis added).
Huang particularly teaches fine-tuning the weights of the processing unit array to reduce a variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array:
Compute-in-memory (CIM) has been proposed to accelerate the convolution neural network (CNN) computation by implementing parallel multiply and accumulation in analog domain. However, the subsequent processing is still preferred to be performed in digital domain. This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples. Our evaluation results show that by adapting the model weights to the specific ADC offset pattern to each chip, the transferability of the adversarial attack is suppressed. For a chip being attacked by the C&W method, the classification for CIFAR-10 dataset will drop to almost 0%. However, when applying the similarly generated adversarial examples to other chips, the accuracy could still maintain more than 62% and 85% accuracy for VGG-8 and DenseNet-40, respectively.
(Abstract; emphasis added).
In this work, we leverage the ADC offset pattern (which is believed to be detrimental to the inference accuracy) but finetune the model weights to take its advantage against the adversarial attack on the CIM accelerator. In our evaluation, we find that the accuracy drop could be compensated by finetuning DNN parameters that adapt to ADC offset. This finetune, while recovering the inference accuracy, makes the DNN parameters slightly different from chip to chip, which brings us a byproduct: the chip will be robust to the adversarial examples generated by attacking other chips or software baseline. Explicitly, even if the adversary attacks one chip instance by manipulating the adversarial input, he/she could not use the same adversarial input to attack all the other chip instances due to the uniqueness of the DNN model for each chip.
(Section I “Introduction”).
In this work, instead of trying to defend against adversarial examples of a certain model in software, we aim to reduce the transferability of the adversarial examples among actual chips. It works like the aforementioned software defense method of introducing randomness into the network parameters. For CIM architecture that employs ADCs, there are intrinsic process variations that will introduce quantization error. As shown in previous prototype chip measurement results, even with precisely designed ADCs, the accuracy will be low if the ADCs all share the same references [24]. To achieve high accuracy, the references of each ADC need to be adjusted independently. This requirement for ADC reference adjustment and independent reference for each ADC brings additional hardware overhead. Alternatively, we could adjust the weights with several retrain epoch, namely “finetune” process. When the model is adapted to the ADC offset pattern, the inference accuracy could be recovered. The overhead of model finetune in software is much less than the implementation of adjustable ADC references on-chip. Now we could take advantages of the model finetune to reduce the transferability of the adversarial examples from software baseline to actual chips, and from one chip to another.
(Section II.C “Adversarial Defense”).
We now discuss the on-chip/off-chip hybrid finetune procedure to mitigate adversarial attack from the hardware standpoint. During the retraining, the feedforward propagation (inference) is first performed on-chip, and then the backpropagation and weight update are done off-chip by software. The detailed process is as follows: we will run the inference on a specific chip that captures its specific ADC offset pattern, then the prediction of the inference will be compared with the ideal label for the loss function; after obtaining the estimated loss, weights are updated through backpropagation in software; finally, the memory cells will be reprogramed to the new weights possibly with write-verify. In our evaluation, the partial sum will sample the ADC offset from the estimated distribution as in Fig. 3(a), and the distorted partial sum will be used as output feature map and saved for error and gradient calculation. The backpropagation and weight update are all directly use floating-point calculation as done in software.
(Section III.B “Procedure of weights finetune”).
We evaluate the proposed hybrid finetune defense method with VGG-8 and DenseNet-40 networks for CIFAR-10 dataset. The precision setting is 8-bit activation and 2-bit weight for VGG-8, and 8-bit activation and 8-bit weight for DenseNet-40. The software baseline accuracy is ~92% for both networks. For the weight finetune process, the batch size of retrain is 200, which means there are 250 iterations to finish the finetune in one epoch. Fig. 5 (a) & (b) shows the retrain curve of one specific chip that implements VGG-8 which uses Flash-ADC and SARADC, respectively. Chip with either Flash-ADC or SAR-ADC could recover the accuracy, however Flash-ADC has less initial accuracy drop and easier to be retrained to recover the high accuracy. This is consistent with the analysis in Section Ⅲ on the possible compensation of SA offsets for Flash-ADC. It is also seen that as the W/L decreases, it will be more difficult to retrain the model to recover the accuracy under process variations. When the W/L is small, which means the sense pass rate is also low, the accuracy may could not be fully recovered. It needed to be pointed out that the W/L reported here appears high since we use the minimum length (
L
m
i
n
) as the L in our simulation. Generally, two to three times of
L
m
i
n
will be used in the analog circuit to avoid very large process variation. We did not optimize the ADC with advanced offset cancellation techniques. Here our goal is just to show that by changing W/L, we could achieve different degrees of the process variation. Fig. 5 (c) & (d) presents the accuracy distribution of several retrain tests before finetune and after finetune collected from multiple chips, respectively. From the plot, we could observe that the accuracy recovery from finetune is generally achievable.
(Section IV. “Evaluation Results”).
As suggested by the above excerpts, such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values.
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain and Huang before the effective filing date of the claimed invention, to modify the calibration process taught by Jain so as to additionally or alternatively comprise such fine-tuning like taught by Huang, which would entail an iterative process that uses output data generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error between expected output data and the generated output data, iteratively adjusting the synaptic target weight values of the synaptic weight matrix based on the determined error, and reprogramming the stored synaptic weight values of the synaptic weight matrix in the processing unit array based on the respective adjusted weight values. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array without requiring additional hardware modifications, as is evident from Huang (see e.g. the first paragraph of section III.A “ADC offset modeling”). Accordingly, Jain and Huang are considered to teach a system similar to that of claim 17, but do not explicitly disclose that each iteration of the calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is further required by claim 17. Jain and Huang also do not explicitly disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 17.
Similar to Jain and Huang, Hu describes a compute-in-memory architecture that comprises a processing unit array (i.e. a crossbar array) comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values (see e.g. paragraphs 0016 and 0022-0023). Hu particularly teaches increasing computational accuracy in the processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061). Hu further teaches that the error that is determined for a given output line comprises a difference between a target offset value (i.e. ideal output signal or value) and a measured offset value (i.e. actual output signal or value) associated with the multiply-and-accumulate distribution data obtained for the given output line (see e.g. paragraphs 0060-0061).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang and Hu before the effective filing date of the claimed invention, to modify the calibration process taught by Jain and Huang such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, and wherein the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the resistive processing unit array, as is suggested by Hu (see e.g. paragraph 0011). Accordingly, Jain, Huang and Hu are considered to teach a system similar to that of claim 17, but do not explicitly teach that the target offset value corresponds to a multiply-and-accumulate value of zero, as is required by claim 17.
Lee nevertheless describes a process for calibrating a resistive processing unit array (i.e. a MAC – an analog multiplier and accumulator) by inputting various values of calibration data to the resistive processing unit array, and by determining a gain and offset based on the output values of the resistive processing unit array (see e.g. paragraphs 0006, 0008, 0014, 0023, 0054 and 0056). Regarding the claimed invention, Lee particularly suggests that the range of calibration input data can particularly comprise an input value of zero, whereby the associated target offset value would correspond to a multiply-and-accumulate value of zero (see e.g. paragraphs 0077-0079).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu and Lee before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to employ a range of input values like taught by Lee, including an input value of zero, which would have a target offset value of zero. It would have been advantageous to one of ordinary skill to utilize such a combination because it would ensure that the processing unit array can more accurately process a plurality of different input values, as is suggested by Lee (see e.g. paragraphs 0006, 0014 and 0054). Accordingly, Jain, Huang, Hu and Lee are considered to teach, to one of ordinary skill in the art, a system like that of claim 17.
As per claim 18, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target weight values of the matrix and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values. Huang suggests that such a calibration process can converge the measured offset values of multiply-and-accumulate distribution data of respective output lines to the target offset value (see e.g. the abstract, which recites: “This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples.”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 18.
As per claim 19, it would have been obvious, as is described above, to modify the calibration process taught by Jain so as to additionally or alternatively comprise fine-tuning like taught by Huang, which would entail iteratively adjusting the target synaptic weight values of the matrix and reprogramming the stored synaptic weight values of the matrix in the resistive processing unit array based on the respective adjusted target synaptic weight values. Huang suggests that such a calibration process entails: (i) applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which results from performing analog multiplication operations by multiplying each of the known input vectors by the matrix in the processing unit array (i.e. performing feedforward propagation/inference); (ii) determining, for a given output line of the processing unit array, an offset associated with the generated set of multiply-and-accumulate distribution data for the given output line; (iii) determining, for the given output line, an error between the determined offset of the generated set of multiply-and-accumulate distribution data, and a target offset associated with a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target weight values of the matrix; (iv) adjusting the target synaptic weight values of the synaptic weight matrix which correspond to the stored synaptic weight values of the given output line, to counteract the error between the determined offset and the target offset; and (v) reprogramming the stored synaptic weight values of the given output line of the processing unit array based on the adjusted synaptic weight values (see e.g. section III.B “Procedure of weights finetune and section IV “Evaluation Results”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Accordingly, the above-described combination of Jain, Huang, Hu and Lee is further considered to teach a system like that of claim 19.
Regarding claim 22, Jain generally presents “CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs [deep neural networks] on crossbar systems by compensating for errors due to nonidealities, greatly mitigating the degradation in accuracy.” (Page 113:1). Like claimed, Jain particularly teaches (i) obtaining a matrix comprising target weight values; (ii) programming an array of cells of a resistive processing unit array to store weight values which correspond to respective target weight values of the matrix; and (iii) performing a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines of the resistive processing unit array (Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
:
Figure 5 outlines the CxDNN compensation flow, which consists of a quantization and conversion algorithm, hardware-independent re-training, and hardware compensation. CxDNN takes a readily available floating-point (FP32) DNN as an input. It first quantizes the weights and activations of the FP32 network to match the precisions of the synaptic devices and ADCs/DACs, resulting in a fixed-point (FxP) DNN. Subsequently, it transforms the weights into conductances and activations to voltages to obtain a DNN for an “ideal” crossbar (with no device and circuit non-idealities other than limited precision), denoted
D
N
N
X
I
. CxDNN then uses a fast hardware-instance-independent re-training method to recover accuracy lost during the conversion process. Next, CxDNN maps the
D
N
N
X
I
to a resistive crossbar system that suffers from crossbar non-idealities (denoted
D
N
N
X
N
I
), which results in further degradation in accuracy. CxDNN overcomes this accuracy drop using a hardware compensation scheme. Note that
D
N
N
X
I
can also be obtained using end-to-end DNN training, i.e., used for designing FP32 DNNs. However, training is computationally very expensive, requiring exa-ops of compute and hence days to weeks to complete. In contrast, CxDNN designs
D
N
N
X
I
by converting FP32 models and utilizing a fast re-training method to recover accuracy loss during conversion with very few iterations. We next describe CxDNN’s software and hardware methods in detail.
(Section 4.1 “CxDNN: Approach and Overview”; emphasis added and footnote omitted).
PNG
media_image2.png
205
423
media_image2.png
Greyscale
The set of conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, is considered a matrix comprising target weight values. Jain further discloses that the CxDNN methodology comprises using a “Compensation Factor” (CF) for different crossbar columns, wherein the Compensation Factor is used to compensate for variance across crossbar columns of a resistive crossbar processing unit:
As mentioned in Section 4.1, DNNs executed on crossbar-based systems will suffer from accuracy degradation due to data-dependent and hardware-instance-specific non-idealities. CxDNN employs a hardware compensation method to overcome this degradation. Our hardware compensation is motivated by a key observation that errors in vector-matrix multiplications realized using resistive crossbars result from the cumulative effect of all non-idealities. It is not possible to isolate the effect of individual non-idealities on the executed vector-matrix multiplication, as all non-idealities kick in simultaneously (as described in Section 3.1). Therefore, we propose error compensation at the crossbar-level, where the outputs of the realized vector-matrix multiplications are compensated using Compensation Factors (CFs). CxDNN uses a separate compensation factor for each crossbar column of each crossbar instance, which allows many degrees of freedom in tuning the compensation process. The use of per-column compensation factors is motivated by the error characteristics in Figure 3(c) that show a significant variance across both crossbar instances and columns within an instance.
(Section 4.4 “Hardware Compensation”; emphasis added).
Jain further discloses that the CF for each column is identified through a calibration process that occurs after the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, are mapped to the resistive crossbar processing unit, whereby the calibration process entails: (1) a host processor sending predetermined inputs to resistive crossbar arrays (RCAs) of the resistive crossbar processing unit; (2) the RCAs performing vector-matrix multiplication using the predetermined inputs and the stored weights to obtain actual output vectors; and (3) the host processor computing the compensation factors using the actual outputs and ideal outputs:
CxDNN’s hardware compensation method consists of two phases—a calibration phase to determine CFs and a runtime phase to mitigate errors using these CFs. The host processor initiates the calibration phase after the
D
N
N
X
I
is mapped and programmed to the resistive crossbar system. The calibration phase consists of three main steps: In step 1, the host sends predetermined inputs (Inp) to each RCA in the resistive crossbar system. In step 2, RCAs perform vector-matrix multiplications using these calibration inputs (Inp) and stored weights to obtain the actual (erroneous) output vectors (Out-act) and subsequently send them to the host. In step 3, the host processor computes compensation factors using the actual outputs (Out-act) and the ideal outputs (Out-idl). The host processor could either store or generate the calibration inputs (Inp) and expected outputs (Out-idl). We generate the calibration inputs using a characterization dataset. We note that our method does not restrict the nature or size of the characterization dataset—it can be as large as required, subject to the characterization time being acceptable. In our experiments, we found that a small subset of the validation set was sufficient to compute compensation factors that led to the accuracy improvements reported in the article. Figure 10 depicts the CF computation process for an RCA using actual and ideal outputs (Out-idl). As shown in the figure, we compute a vector of CFs, one for each column. The computed CFs are then sent back to the respective crossbars to be stored locally in the compensation logic. While all RCAs execute vector-matrix multiplications in parallel, the host processor computes the compensation factor for each RCA sequentially. Therefore, the overall calibration time is dominated by the host processor. Further, the calibration time depends on the total number of RCAs utilized by the DNN. In our evaluation, the overall calibration time was 0.2 s to 1.3 s across our benchmark DNNs for a single core Intel-Atom C2350 as our host processor. We invoke the calibration phase whenever a new DNN is mapped to the XPU system and subsequently execute millions of inference operations during the runtime phase. We can also periodically invoke the calibration phase to cope with aging issues and drift in synaptic conductances. Aging and synaptic drifts are very slow phenomena, hence the calibration phase will only need to be performed rarely (e.g., once every 10 K–100 K inference operations). The execution time required for calibration is negligible when amortized over 10 K–100 K inference operations. We also note that the computed CFs are not exact but approximate factors that compensate the errors effectively with limited hardware overhead, as discussed next.
(Section 4.4 “Hardware Compensation”; emphasis added).
Accordingly, Jain is further considered to teach (i) programming an array of cells of a resistive processing unit array to store weight values, i.e. the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, in the resistive processing unit array, which correspond to respective target weight values of the matrix, and (ii) performing a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data, i.e. actual output data, that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines, i.e. columns, of the resistive processing unit array.). Jain thus teaches a method similar to that of claim 22. Jain, however, does not disclose or suggests that the calibration process is an iterative process, wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the synaptic target weight values of the trained synaptic weight matrix based on the determined errors for each output line, and reprogramming the stored synaptic weight values of the synaptic weight matrix in the resistive processing unit array based on the respective adjusted target synaptic weight values, as is required by claim 22. Jain also does not disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 22.
Nevertheless, like noted above (see e.g. the rejection for claim 1), Huang teaches fine-tuning the weights of a processing unit array to reduce a variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array; such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values.
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain and Huang before the effective filing date of the claimed invention, to modify the calibration process taught by Jain so as to additionally or alternatively comprise such fine-tuning like taught by Huang, which would entail an iterative process that uses output data generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error between expected output data and the generated output data, iteratively adjusting the target weight values of the matrix based on the determined error, and reprogramming the stored weight values of the matrix in the processing unit array based on the respective adjusted weight values. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array without requiring additional hardware modifications, as is evident from Huang (see e.g. the first paragraph of section III.A “ADC offset modeling”). Accordingly, Jain and Huang are considered to teach method similar to that of claim 22, but do not explicitly disclose that each iteration of the calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is further required by claim 22. Jain and Huang also do not explicitly disclose that the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line, the target offset value corresponding to a multiply-and-accumulate value of zero, as is further required by claim 22.
Nevertheless, like noted above, Hu similarly describes a compute-in-memory architecture that comprises a processing unit array (i.e. a crossbar array) comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values (see e.g. paragraphs 0016 and 0022-0023). Hu particularly teaches increasing computational accuracy in the processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061). Hu further teaches that the error that is determined for a given output line comprises a difference between a target offset value (i.e. ideal output signal or value) and a measured offset value (i.e. actual output signal or value) associated with the multiply-and-accumulate distribution data obtained for the given output line (see e.g. paragraphs 0060-0061).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang and Hu before the effective filing date of the claimed invention, to modify the calibration process taught by Jain and Huang such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, and wherein the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the resistive processing unit array, as is suggested by Hu (see e.g. paragraph 0011). Accordingly, Jain, Huang and Hu are considered to teach a method similar to that of claim 22, but do not explicitly teach that the target offset value corresponds to a multiply-and-accumulate value of zero, as is required by claim 22.
Lee nevertheless describes a process for calibrating a resistive processing unit array (i.e. a MAC – an analog multiplier and accumulator) by inputting various values of calibration data to the resistive processing unit array, and by determining a gain and offset based on the output values of the resistive processing unit array (see e.g. paragraphs 0006, 0008, 0014, 0023, 0054 and 0056). Regarding the claimed invention, Lee particularly suggests that the range of calibration input data can particularly comprise an input value of zero, whereby the associated target offset value would correspond to a multiply-and-accumulate value of zero (see e.g. paragraphs 0077-0079).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu and Lee before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to employ a range of input values like taught by Lee, including an input value of zero, which would have a target offset value of zero. It would have been advantageous to one of ordinary skill to utilize such a combination because it would ensure that the processing unit array can more accurately process a plurality of different input values, as is suggested by Lee (see e.g. paragraphs 0006, 0014 and 0054). Accordingly, Jain, Huang, Hu and Lee are considered to teach, to one of ordinary skill in the art, a method like that of claim 22.
Claims 3, 6, 11, 14, 20 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Jain, Huang, Hu and Lee, which is described above, and also over the article entitled, “GENIEx: A Generalized Approach to Emulating Non-Ideality in Memristive Xbars using Neural Networks” by Chakraborty et al. (“Chakraborty”).
Regarding claims 3 and 11, Jain, Huang, Hu and Lee teach a system like that of claim 1 and a computer program product like that of claim 9, as is described above, which entail performing a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Jain, Huang, Hu and Lee, however, do not explicitly disclose that the calibration process comprises converging respective slopes of the multiply-and-accumulate distribution data, which are output from the respective output lines, to a target slope like required by claims 3 and 11.
Chakraborty nevertheless generally teaches that the slopes of multiply-and-accumulate data (i.e. current) actually output by a resistive crossbar differ from the slopes of the ideal multiply-and-accumulate data output by such a crossbar (see e.g. section 3 “Analysis of NVM Non-Idealities” and particularly FIG. 2(a) therein). Chakraborty suggests that mitigation strategies (e.g. retraining of the neural network weights) can mitigate such non-idealities (see e.g. section 1 “Introduction,” which states: “To address this accuracy degradation, there have been efforts towards exploring techniques to model non-idealities and subsequently mitigating them [9-11]. The efficacy of these mitigation techniques strongly depend upon the modelling [9, 10, 12] approach to exhaustively capture the sources of the non-idealities and retraining of the neural network weights.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Chakraborty before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee so as to mitigate the divergent slopes of multiply-and-accumulate data output by the respective output lines of the resistive processing unit array (i.e. to converge the respective slopes of multiply-and-accumulate data to a target, ideal slope), like generally taught by Chakraborty. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array, as is evident from Chakraborty (see e.g. section 1 “Introduction”). Accordingly, Jain, Huang, Hu, Lee and Chakraborty are considered to teach, to one of ordinary skill in the art, a method like that of claim 3 and a computer program product like that of claim 11.
Regarding claims 6 and 14, Jain, Huang, Hu and Lee teach a system like that of claim 1 and a computer program product like that of claim 9, as is described above, which entail performing a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claims 6 and 14, Huang particularly teaches that the calibration process includes applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which result from performing analog multiplication operations by multiplying each of the known input vectors by the matrix in the resistive processing unit array (see e.g. section III.B “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Jain, Huang, Hu and Lee, however, do not explicitly teach: (i) determining, for a given output line of the resistive processing unit array, a slope of a straight line fitted to the generated set of multiply-and-accumulate distribution data for the given output line; (ii) determining, for the given output line, a weight scaling factor based on a difference between the determined slope of the straight line fitted to the generated set of multiply-and-accumulate distribution data for the given output line, and a target slope of a straight line fitted to a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target weight values of the matrix; (iii) scaling the target weight values of the matrix, which correspond to the stored weight values of the given output line, based on the weight scaling factor; and (iv) reprogramming the stored weight values of the given output line of the resistive processing unit array based on the scaled target weight values, as is further required by claims 6 and 14.
Chakraborty nevertheless generally teaches that the slopes of multiply-and-accumulate data (i.e. current) actually output by a resistive crossbar differ from the slopes of the ideal multiply-and-accumulate data output by such a crossbar (see e.g. section 3 “Analysis of NVM Non-Idealities” and particularly FIG. 2(a) therein). Chakraborty suggests that mitigation strategies (e.g. retraining of the neural network weights) can mitigate such non-idealities (see e.g. section 1 “Introduction,” which states: “To address this accuracy degradation, there have been efforts towards exploring techniques to model non-idealities and subsequently mitigating them [9-11]. The efficacy of these mitigation techniques strongly depend upon the modelling [9, 10, 12] approach to exhaustively capture the sources of the non-idealities and retraining of the neural network weights.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Chakraborty before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee so as to mitigate the slopes of multiply-and-accumulate data output by the respective output lines of the resistive processing unit array that differ from the slope of multiply-and-accumulate data ideally output (i.e. by determining, for each output line, a weight scaling factor based on a difference between the slope of a straight line fitted to the generated set of multiply-and-accumulate distribution data for the output line, and a target slope of an ideal strait line that is fitted to a known set of multiply-and-accumulate distribution data that is obtained by performing digital analog vector-matrix multiplication operation using the known input vectors and the target weight values of the matrix; by scaling the target weight values of the matrix that correspond to the stored weight values of the given output line, based on the weight scaling factor; and by reprogramming the stored weight values of the given output line of the resistive processing unit array based on the scaled target weight values), like generally taught by Chakraborty. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array, as is evident from Chakraborty (see e.g. section 1 “Introduction”). Accordingly, Jain, Huang, Hu, Lee and Chakraborty are considered to teach, to one of ordinary skill in the art, a method like that of claim 6 and a computer program product like that of claim 14.
Regarding claim 20, Jain, Huang, Hu and Lee teach a system like that of claim 17, as is described above, which entails performing a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claim 20, Huang particularly teaches that the calibration process includes applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which result from performing analog multiplication operations by multiplying each of the known input vectors by the synaptic weight matrix in the resistive processing unit array (see e.g. section III.B “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Jain, Huang, Hu and Lee, however, do not explicitly teach: (i) determining, for a given output line of the resistive processing unit array, a slope of a straight line fitted to the generated set of multiply-and-accumulate distribution data for the given output line; (ii) determining, for the given output line, a weight scaling factor based on a difference between the determined slope of the straight line fitted to the generated set of multiply-and-accumulate distribution data for the given output line, and a target slope of a straight line fitted to a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target synaptic weight values of the synaptic weight matrix; (iii) scaling the target synaptic weight values of the synaptic weight matrix, which correspond to the stored synaptic weight values of the given output line, based on the weight scaling factor; and (iv) reprogramming the stored synaptic weight values of the given output line of the resistive processing unit array based on the scaled target synaptic weight values, as is further required by claim 20.
Chakraborty nevertheless generally teaches that the slopes of multiply-and-accumulate data (i.e. current) actually output by a resistive crossbar differ from the slopes of the ideal multiply-and-accumulate data output by such a crossbar (see e.g. section 3 “Analysis of NVM Non-Idealities” and particularly FIG. 2(a) therein). Chakraborty suggests that mitigation strategies (e.g. retraining of the neural network weights) can mitigate such non-idealities (see e.g. section 1 “Introduction,” which states: “To address this accuracy degradation, there have been efforts towards exploring techniques to model non-idealities and subsequently mitigating them [9-11]. The efficacy of these mitigation techniques strongly depend upon the modelling [9, 10, 12] approach to exhaustively capture the sources of the non-idealities and retraining of the neural network weights.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Chakraborty before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee so as to mitigate the slopes of multiply-and-accumulate data output by the respective output lines of the resistive processing unit array that differ from the slope of multiply-and-accumulate data ideally output (i.e. by determining, for each output line, a synaptic weight scaling factor based on a difference between the slope of a straight line fitted to the generated set of multiply-and-accumulate distribution data for the output line, and a target slope of an ideal strait line that is fitted to a known set of multiply-and-accumulate distribution data that is obtained by performing digital analog vector-matrix multiplication operation using the known input vectors and the target synaptic weight values of the synaptic weight matrix; by scaling the target synaptic weight values of the synaptic weight matrix that correspond to the stored synaptic weight values of the given output line, based on the weight scaling factor; and by reprogramming the stored synaptic weight values of the given output line of the resistive processing unit array based on the scaled target weight values), like generally taught by Chakraborty. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array, as is evident from Chakraborty (see e.g. section 1 “Introduction”). Accordingly, Jain, Huang, Hu, Lee and Chakraborty are considered to teach, to one of ordinary skill in the art, system like that of claim 20.
Regarding claim 24, Jain generally presents “CxDNN, a hardware-software methodology that enables the realization of large-scale DNNs [deep neural networks] on crossbar systems by compensating for errors due to nonidealities, greatly mitigating the degradation in accuracy.” (Page 113:1). Like claimed, Jain particularly teaches a system comprising:
a processor and a resistive processing unit array coupled to the processor, the resistive processing unit array comprising an array of cells, the cells respectively comprising resistive memory devices which are programmable to store weight values (see e.g. section 3.1 “Resistive Crossbar System”: Jain describes a system comprising a host processor, a main memory and a resistive crossbar processing unit, which comprises a resistive crossbar array of synaptic elements that each comprise resistive memory devices that are programmable to store DNN weights:
Figure 2 (left side) depicts a representative resistive crossbar–based accelerator, viz., Resistive crossbar processing unit (XPU), connected to a host processor and main memory via a system bus. The host processor off-loads execution of compute kernels having vector-matrix multiplications (e.g., convolution and fully-connected layers in DNNs) to the XPU, which in turn realizes them using multiple Crossbar processing tiles (XPTs) and a scheduler. XPTs are composed of Resistive Crossbar Arrays (RCAs) that store weights and compute vector-matrix multiplications, local memories in the form of input and output buffers to store activations, a local bus, a controller to orchestrate various operations, and a special function unit (SFU) to execute other DNN operations such as ReLU, max pooling, and so on. Next, we describe RCAs, the fundamental compute units of resistive crossbar systems, in detail.
As shown in Figure 2 (right side), a resistive crossbar array is composed of a 2D array of synaptic elements, Analog-to-Digital Converters (ADCs), Digital-to-Analog Converters (DACs), registers, a decoder, a column MUX, a reduce unit, and write circuits. The synaptic elements are programmable resistors that store DNN weights as conductances. The number of available conductance levels determines the precision of the synaptic element. They can be realized using emerging NVMs technologies, viz., PCM, ReRAM, and spintronics [18, 28, 33]. RCA supports two main operations: (i) vector-matrix multiplication and (ii) programming. A vector-matrix multiplication is performed by driving all wordlines (WL) to analog voltages using DACs and sensing the resultant currents flowing through the columns (BL) using ADCs. The vector-matrix multiplication ideally realized in an RCA can be expressed as
I
o
u
t
i
d
e
a
l
=
V
i
n
i
d
e
a
l
*
G
i
d
e
a
l
, where
V
i
n
i
d
e
a
l
represents input voltages,
G
i
d
e
a
l
denotes programmed conductances, and
I
o
u
t
i
d
e
a
l
represents output currents. In contrast, the programming operations, i.e., write operations on synaptic elements, are performed row-wise, wherein the write circuitry applies the necessary currents and sets them to the desired conductance.
(Section 3.1 “Resistive Crossbar System”’; emphasis added).
PNG
media_image1.png
307
439
media_image1.png
Greyscale
Jain thus describes a system comprising a host processor and a resistive processing unit array, i.e. a resistive crossbar processing unit, which comprises an array of cells, i.e. synaptic elements, that each comprise resistive memory devices that are programmable to store DNN weight values.);
wherein the processor is configured to:
obtain a matrix comprising target weight values (Jain discloses that the CxDNN methodology entails quantizing the weights of an input DNN, and transforming the weights into conductances to obtain a DNN for an ideal crossbar, denoted as
D
N
N
X
I
:
Figure 5 outlines the CxDNN compensation flow, which consists of a quantization and conversion algorithm, hardware-independent re-training, and hardware compensation. CxDNN takes a readily available floating-point (FP32) DNN as an input. It first quantizes the weights and activations of the FP32 network to match the precisions of the synaptic devices and ADCs/DACs, resulting in a fixed-point (FxP) DNN. Subsequently, it transforms the weights into conductances and activations to voltages to obtain a DNN for an “ideal” crossbar (with no device and circuit non-idealities other than limited precision), denoted
D
N
N
X
I
. CxDNN then uses a fast hardware-instance-independent re-training method to recover accuracy lost during the conversion process. Next, CxDNN maps the
D
N
N
X
I
to a resistive crossbar system that suffers from crossbar non-idealities (denoted
D
N
N
X
N
I
), which results in further degradation in accuracy. CxDNN overcomes this accuracy drop using a hardware compensation scheme. Note that
D
N
N
X
I
can also be obtained using end-to-end DNN training, i.e., used for designing FP32 DNNs. However, training is computationally very expensive, requiring exa-ops of compute and hence days to weeks to complete. In contrast, CxDNN designs
D
N
N
X
I
by converting FP32 models and utilizing a fast re-training method to recover accuracy loss during conversion with very few iterations. We next describe CxDNN’s software and hardware methods in detail.
(Section 4.1 “CxDNN: Approach and Overview”; emphasis added and footnote omitted).
PNG
media_image2.png
205
423
media_image2.png
Greyscale
The host processor thus obtains a matrix that comprises target weight values, i.e. that comprises conductances/weights of a DNN for an ideal crossbar,
D
N
N
X
I
.); and
program cells of the array of cells to store weight values, in the resistive processing unit array, which correspond to respective target weight values of the matrix, and perform a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines of the resistive processing unit array (Jain discloses that the CxDNN methodology comprises using a “Compensation Factor” (CF) for each crossbar column, wherein the Compensation Factor is used to compensate for variance across crossbar columns of the resistive crossbar processing unit:
As mentioned in Section 4.1, DNNs executed on crossbar-based systems will suffer from accuracy degradation due to data-dependent and hardware-instance-specific non-idealities. CxDNN employs a hardware compensation method to overcome this degradation. Our hardware compensation is motivated by a key observation that errors in vector-matrix multiplications realized using resistive crossbars result from the cumulative effect of all non-idealities. It is not possible to isolate the effect of individual non-idealities on the executed vector-matrix multiplication, as all non-idealities kick in simultaneously (as described in Section 3.1). Therefore, we propose error compensation at the crossbar-level, where the outputs of the realized vector-matrix multiplications are compensated using Compensation Factors (CFs). CxDNN uses a separate compensation factor for each crossbar column of each crossbar instance, which allows many degrees of freedom in tuning the compensation process. The use of per-column compensation factors is motivated by the error characteristics in Figure 3(c) that show a significant variance across both crossbar instances and columns within an instance.
(Section 4.4 “Hardware Compensation”; emphasis added).
Jain further discloses that the CF for each column is identified through a calibration process that occurs after the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, are mapped to the resistive crossbar processing unit, whereby the calibration process entails: (1) the host processor sending predetermined inputs to resistive crossbar arrays (RCAs) of the resistive crossbar processing unit; (2) the RCAs performing vector-matrix multiplication using the predetermined inputs and the stored weights to obtain actual output vectors; and (3) the host processor computing the compensation factors using the actual outputs and ideal outputs:
CxDNN’s hardware compensation method consists of two phases—a calibration phase to determine CFs and a runtime phase to mitigate errors using these CFs. The host processor initiates the calibration phase after the
D
N
N
X
I
is mapped and programmed to the resistive crossbar system. The calibration phase consists of three main steps: In step 1, the host sends predetermined inputs (Inp) to each RCA in the resistive crossbar system. In step 2, RCAs perform vector-matrix multiplications using these calibration inputs (Inp) and stored weights to obtain the actual (erroneous) output vectors (Out-act) and subsequently send them to the host. In step 3, the host processor computes compensation factors using the actual outputs (Out-act) and the ideal outputs (Out-idl). The host processor could either store or generate the calibration inputs (Inp) and expected outputs (Out-idl). We generate the calibration inputs using a characterization dataset. We note that our method does not restrict the nature or size of the characterization dataset—it can be as large as required, subject to the characterization time being acceptable. In our experiments, we found that a small subset of the validation set was sufficient to compute compensation factors that led to the accuracy improvements reported in the article. Figure 10 depicts the CF computation process for an RCA using actual and ideal outputs (Out-idl). As shown in the figure, we compute a vector of CFs, one for each column. The computed CFs are then sent back to the respective crossbars to be stored locally in the compensation logic. While all RCAs execute vector-matrix multiplications in parallel, the host processor computes the compensation factor for each RCA sequentially. Therefore, the overall calibration time is dominated by the host processor. Further, the calibration time depends on the total number of RCAs utilized by the DNN. In our evaluation, the overall calibration time was 0.2 s to 1.3 s across our benchmark DNNs for a single core Intel-Atom C2350 as our host processor. We invoke the calibration phase whenever a new DNN is mapped to the XPU system and subsequently execute millions of inference operations during the runtime phase. We can also periodically invoke the calibration phase to cope with aging issues and drift in synaptic conductances. Aging and synaptic drifts are very slow phenomena, hence the calibration phase will only need to be performed rarely (e.g., once every 10 K–100 K inference operations). The execution time required for calibration is negligible when amortized over 10 K–100 K inference operations. We also note that the computed CFs are not exact but approximate factors that compensate the errors effectively with limited hardware overhead, as discussed next.
(Section 4.4 “Hardware Compensation”; emphasis added).
Accordingly, Jain is further considered to teach configuring the processor to (i) program cells of the array of cells to store weight values, i.e. the conductances/weights of the DNN for the ideal crossbar,
D
N
N
X
I
, in the resistive processing unit array, which correspond to respective target weight values of the matrix, and (ii) perform a calibration process to calibrate the resistive processing unit array using multiply-and-accumulate distribution data, i.e. actual output data, that is generated and output from respective output lines of the resistive processing unit array during the calibration process to reduce a variation between output lines, i.e. columns, of the resistive processing unit array.).
Jain thus teaches a system similar to that of claim 24. Jain, however, does not disclose or suggests that the calibration process comprises: (A) a first calibration process to analyze the multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, to iteratively adjust the target weight values of the matrix based on the determined error for each output line, and reprogram the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values, to (i) reduce a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for each respective output line of the resistive processing unit array, the target offset value corresponding to a multiply-and-accumulate value of zero, and to (ii) reduce a spread of the multiply-and-accumulate distribution data, which is generated and output from respective output lines of the resistive processing unit array during the first calibration process; and (B) a second calibration process, which is performed subsequent to the first calibration process, to scale the adjusted target weight values of the output lines, which exist at a completion of the first calibration process, by respective weight scaling factors, and reprogram the stored weight values of the output lines of the resistive processing unit array based on the scaled target weight values to reduce a slope variation between the output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data which is generated and output from the respective output lines of the resistive processing unit array, as is required by claim 24.
Similar to Jain, Huang describes a compute-in-memory architecture that comprises a processing unit array comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values:
As DNNs are generally data and compute intensive, frequent data movements between logic and memory units limit the energy efficiency on traditional Von Neumann architecture. In recent years, there are increasingly efforts on developing specific hardware accelerators to run large-scale DNN models from the cloud to the edge. For example, systolic architecture such as TPU [5] employs many digital multiply and accumulate (MAC) engines close to a large global buffer (i.e., SRAM) to reduce the cost of data movement. As a more aggressive approach, compute-in-memory (CIM) architecture [4] merges the computation directly into the memory sub-arrays that ideally addresses the memory-wall problem. The weights of a DNN model could be mapped as the conductance of the memory cells in the sub-array, while the input vector is loaded in parallel as the voltage to the rows, then the multiplication is done in analog fashion, and the current summation along columns represents weighted sum. In principle, CIM could be implemented by different device technologies. SRAM with modified bit-cell and array periphery could enable parallel access as demonstrated in recent silicon prototype chips [6]. Emerging non-volatile memory (eNVM) technologies also provide promising solutions due to a smaller cell size and potential of multi-bit per cell, yielding a higher integration density at the same technology node [7]. Besides, because of the non-volatile nature and near-zero leakage, the eNVM-based CIM is more attractive to edge devices. No matter which kind of memory technologies is used, ADC is commonly essential as an important part of periphery circuitry to convert the analog partial sum back to digital signal for further processing. In other words, CIM is essentially mixed-signal compute, thus the variations are unavoidable. As reported in prior work [8], inference accuracy measured in CIM prototypes generally is degraded from the software baseline. The primary variation sources include the cell-to-cell variation for eNVMs and the intrinsic ADC offset. Cell-to-cell variation could be minimized by iterative write-verify technique with tolerable overhead for inference engine [9]. A more critical challenge is the intrinsic ADC offset introduced by the manufacturing process variation. As a result, the ADC offset may noticeably degrade the inference accuracy and cause different chip instances having different inference results even for the same input. It is noted that when ADC offset introduces quantization error because of the process variation, these offset patterns are static once the chip is fabricated.
(Section I “Introduction”; emphasis added).
The crossbar nature of memory array is a natural substrate for implementing VMM in a highly parallel manner. As shown in Fig. 1, the crossbar array consists of perpendicular rows and columns with the memory cell located at each cross-point. Weights in the filters are mapped as the content of the cells. The VMM operation is performed as follows: read voltages representing the input feature map are applied to all the rows so that the read voltages are multiplied by the memory cells at each cross-point. The current through each device is summed up along columns. Different columns represent filters for different output channels, who should see the same input thus all the columns work at the same time in parallel. Typically, ADCs are needed at the end of the column to convert the analog current to the digital output so that the subsequent processing such as activation and pooling could be performed in the digital domain. In principle, VMM could be done in fully parallel fashion if asserting all the rows and all the columns simultaneously. In practice, multiple rows/columns could be partially turned on due to the sensing resolution of ADCs or the mismatch of column pitch to the peripheral circuitry’s dimension.
(Section II.A “Pricinple [sic] of CIM”; emphasis added).
Huang particularly teaches fine-tuning the weights of the processing unit array to reduce an offset variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array and to reduce a spread of the data that is generated and output from respective output lines of the processing unit array:
Compute-in-memory (CIM) has been proposed to accelerate the convolution neural network (CNN) computation by implementing parallel multiply and accumulation in analog domain. However, the subsequent processing is still preferred to be performed in digital domain. This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples. Our evaluation results show that by adapting the model weights to the specific ADC offset pattern to each chip, the transferability of the adversarial attack is suppressed. For a chip being attacked by the C&W method, the classification for CIFAR-10 dataset will drop to almost 0%. However, when applying the similarly generated adversarial examples to other chips, the accuracy could still maintain more than 62% and 85% accuracy for VGG-8 and DenseNet-40, respectively.
(Abstract; emphasis added).
In this work, we leverage the ADC offset pattern (which is believed to be detrimental to the inference accuracy) but finetune the model weights to take its advantage against the adversarial attack on the CIM accelerator. In our evaluation, we find that the accuracy drop could be compensated by finetuning DNN parameters that adapt to ADC offset. This finetune, while recovering the inference accuracy, makes the DNN parameters slightly different from chip to chip, which brings us a byproduct: the chip will be robust to the adversarial examples generated by attacking other chips or software baseline. Explicitly, even if the adversary attacks one chip instance by manipulating the adversarial input, he/she could not use the same adversarial input to attack all the other chip instances due to the uniqueness of the DNN model for each chip.
(Section I “Introduction”).
In this work, instead of trying to defend against adversarial examples of a certain model in software, we aim to reduce the transferability of the adversarial examples among actual chips. It works like the aforementioned software defense method of introducing randomness into the network parameters. For CIM architecture that employs ADCs, there are intrinsic process variations that will introduce quantization error. As shown in previous prototype chip measurement results, even with precisely designed ADCs, the accuracy will be low if the ADCs all share the same references [24]. To achieve high accuracy, the references of each ADC need to be adjusted independently. This requirement for ADC reference adjustment and independent reference for each ADC brings additional hardware overhead. Alternatively, we could adjust the weights with several retrain epoch, namely “finetune” process. When the model is adapted to the ADC offset pattern, the inference accuracy could be recovered. The overhead of model finetune in software is much less than the implementation of adjustable ADC references on-chip. Now we could take advantages of the model finetune to reduce the transferability of the adversarial examples from software baseline to actual chips, and from one chip to another.
(Section II.C “Adversarial Defense”).
We now discuss the on-chip/off-chip hybrid finetune procedure to mitigate adversarial attack from the hardware standpoint. During the retraining, the feedforward propagation (inference) is first performed on-chip, and then the backpropagation and weight update are done off-chip by software. The detailed process is as follows: we will run the inference on a specific chip that captures its specific ADC offset pattern, then the prediction of the inference will be compared with the ideal label for the loss function; after obtaining the estimated loss, weights are updated through backpropagation in software; finally, the memory cells will be reprogramed to the new weights possibly with write-verify. In our evaluation, the partial sum will sample the ADC offset from the estimated distribution as in Fig. 3(a), and the distorted partial sum will be used as output feature map and saved for error and gradient calculation. The backpropagation and weight update are all directly use floating-point calculation as done in software.
(Section III.B “Procedure of weights finetune”).
We evaluate the proposed hybrid finetune defense method with VGG-8 and DenseNet-40 networks for CIFAR-10 dataset. The precision setting is 8-bit activation and 2-bit weight for VGG-8, and 8-bit activation and 8-bit weight for DenseNet-40. The software baseline accuracy is ~92% for both networks. For the weight finetune process, the batch size of retrain is 200, which means there are 250 iterations to finish the finetune in one epoch. Fig. 5 (a) & (b) shows the retrain curve of one specific chip that implements VGG-8 which uses Flash-ADC and SARADC, respectively. Chip with either Flash-ADC or SAR-ADC could recover the accuracy, however Flash-ADC has less initial accuracy drop and easier to be retrained to recover the high accuracy. This is consistent with the analysis in Section Ⅲ on the possible compensation of SA offsets for Flash-ADC. It is also seen that as the W/L decreases, it will be more difficult to retrain the model to recover the accuracy under process variations. When the W/L is small, which means the sense pass rate is also low, the accuracy may could not be fully recovered. It needed to be pointed out that the W/L reported here appears high since we use the minimum length (
L
m
i
n
) as the L in our simulation. Generally, two to three times of
L
m
i
n
will be used in the analog circuit to avoid very large process variation. We did not optimize the ADC with advanced offset cancellation techniques. Here our goal is just to show that by changing W/L, we could achieve different degrees of the process variation. Fig. 5 (c) & (d) presents the accuracy distribution of several retrain tests before finetune and after finetune collected from multiple chips, respectively. From the plot, we could observe that the accuracy recovery from finetune is generally achievable.
(Section IV. “Evaluation Results”).
As suggested by the above excerpts, such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values.
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain and Huang before the effective filing date of the claimed invention, to modify the calibration process taught by Jain so as to additionally or alternatively comprise such fine-tuning like taught by Huang, which would entail an iterative process that uses output data generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error between expected output data and the generated output data, iteratively adjusting the target weight values of the matrix based on the determined error, and reprogramming the stored weight values of the matrix in the processing unit array based on the respective adjusted weight values. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array without requiring additional hardware modifications, as is evident from Huang (see e.g. the first paragraph of section III.A “ADC offset modeling”). Accordingly, Jain and Huang are considered to teach a system similar to that of claim 24, but do not explicitly disclose that the first calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is further required by claim 24. Jain and Huang also do not disclose that the first calibration process (i) reduces a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for each respective output line of the resistive processing unit array, the target offset value corresponding to a multiply-and-accumulate value of zero, and (ii) reduces a spread of the multiply-and-accumulate distribution data, which is generated and output from respective output lines of the resistive processing unit array, as is further required by claim 24. Moreover, Jain and Huang also do not teach or suggest the “second calibration process” required by claim 24.
Similar to Jain and Huang, Hu describes a compute-in-memory architecture that comprises a processing unit array (i.e. a crossbar array) comprising an array of cells, with each cell comprising memory devices that are programmable to store weight values (see e.g. paragraphs 0016 and 0022-0023). Hu particularly teaches increasing computational accuracy in the processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061). Hu further teaches that the iterative process reduces a difference between a target offset value (i.e. ideal output signal or value) and a measured offset value (i.e. actual output signal or value) associated with the multiply-and-accumulate distribution data obtained for each respective output line of the resistive processing unit array (see e.g. paragraphs 0060-0061).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang and Hu before the effective filing date of the claimed invention, to modify the first calibration process taught by Jain and Huang such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line and wherein the process reduces a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for each respective output line of the resistive processing unit array.
Lee generally describes a process for calibrating a resistive processing unit array (i.e. a MAC – an analog multiplier and accumulator) by inputting various values of calibration data to the resistive processing unit array, and by determining a gain and offset based on the output values of the resistive processing unit array (see e.g. paragraphs 0006, 0008, 0014, 0023, 0054 and 0056). Regarding the claimed invention, Lee particularly suggests that the range of calibration input data can particularly comprise an input value of zero, whereby the associated target offset value would correspond to a multiply-and-accumulate value of zero (see e.g. paragraphs 0077-0079).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu and Lee before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to employ a range of input values like taught by Lee, including an input value of zero, which would have a target offset value of zero. It would have been advantageous to one of ordinary skill to utilize such a combination because it would ensure that the processing unit array can more accurately process a plurality of different input values, as is suggested by Lee (see e.g. paragraphs 0006, 0014 and 0054).
Chakraborty generally teaches that the slopes of multiply-and-accumulate data (i.e. current) actually output by a resistive crossbar differ from the slopes of the ideal multiply-and-accumulate data output by such a crossbar (see e.g. section 3 “Analysis of NVM Non-Idealities” and particularly FIG. 2(a) therein). Chakraborty suggests that mitigation strategies (e.g. retraining of the neural network weights) can mitigate such non-idealities (see e.g. section 1 “Introduction,” which states: “To address this accuracy degradation, there have been efforts towards exploring techniques to model non-idealities and subsequently mitigating them [9-11]. The efficacy of these mitigation techniques strongly depend upon the modelling [9, 10, 12] approach to exhaustively capture the sources of the non-idealities and retraining of the neural network weights.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Chakraborty before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang and Hu so as to mitigate the slopes of multiply-and-accumulate data output by the respective output lines of the resistive processing unit array that differ from the slope of multiply-and-accumulate data ideally output (i.e. to perform a second calibration process to scale the adjusted target weight values of the output lines, which exist at a completion of the first calibration process, by respective weight scaling factors, and reprogram the stored weight values of the output lines of the resistive processing unit array based on the scaled target weight values to reduce a slope variation between the output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data which is generated and output from the respective output lines of the resistive processing unit array), as is generally taught by Chakraborty. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array, as is evident from Chakraborty (see e.g. section 1 “Introduction”). Accordingly, Jain, Huang, Hu, Lee and Chakraborty are considered to teach, to one of ordinary skill in the art, system like that of claim 24.
Claims 7, 15 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Jain, Huang, Hu and Lee, which is described above, and also over the article entitled, “Digital-to-Analog and Analog-to-Digital Conversion with Metal Oxide Memristors for Ultra-Low Power Computing” by Gao et al. “Gao”).
Regarding claims 7 and 15, Jain, Huang, Hu and Lee teach a system like that of claim 1 and a computer program product like that of claim 9, as is described above, which entail performing a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claims 7 and 15, Huang particularly teaches that the calibration process includes: (i) applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which result from performing analog multiplication operations by multiplying each of the known input vectors by the synaptic weight matrix in the processing unit array; (ii) determining, for a given output line of the processing unit array, an offset associated with the generated set of multiply-and-accumulate distribution data for the given output line; (iii) determining, for the given output line, an error between the determined offset of the generated set of multiply-and-accumulate distribution data, and a target offset associated with a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target weight values; (iv) adjusting one or more target weight values, which correspond to one or more weights of the given output line, to counteract the error between the determined offset and the target offset; and (v) reprogramming the one or more stored weights of the given output line of the resistive processing unit array, based on the adjusted target weight values (see e.g. section III.B “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Jain, Huang, Hu and Lee, however, do not explicitly teach that the adjusted target weight values are bias weight values that correspond to one or more bias weights of the given output line, as is required by claims 7 and 15.
Gao nevertheless discloses that non-ideal behaviors of neurons can be compensated by fine-tuning bias weights (see e.g. section IV “Hopefield Network ADC”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Gao before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee such that the adjusted target weight values include bias weight values that correspond to one or more bias weights of the output line like taught by Gao. It would have been advantageous to one of ordinary skill to utilize such a combination because it can compensate for non-deal behaviors of neurons, as is taught by Gao (see e.g. section IV “Hopefield Network ADC”). Accordingly, Jain, Huang, Hu, Lee and Gao are considered to teach, to one of ordinary skill in the art, system like that of claim 7 and a computer program product like that of claim 15.
Regarding claim 21, Jain, Huang, Hu and Lee teach a system like that of claim 17, as is described above, which performs a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claim 21, Huang particularly teaches that the calibration process includes: (i) applying a set of known input vectors to the processing unit array to generate a set of multiply-and-accumulate distribution data for each output line (i.e. ADC), which result from performing analog multiplication operations by multiplying each of the known input vectors by the synaptic weight matrix in the processing unit array; (ii) determining, for a given output line of the processing unit array, an offset associated with the generated set of multiply-and-accumulate distribution data for the given output line; (iii) determining, for the given output line, an error between the determined offset of the generated set of multiply-and-accumulate distribution data, and a target offset associated with a known set of multiply-and-accumulate distribution data that is obtained by performing a digital analog vector-matrix multiplication operation using the known input vectors and the target synaptic weight values of the synaptic weight matrix; (iv) adjusting one or more target weight values, which correspond to one or more weights of the given output line, to counteract the error between the determined offset and the target offset; and (v) reprogramming the one or more stored weights of the given output line of the resistive processing unit array, based on the adjusted target weight values (see e.g. section III.B “Procedure of weights finetune”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Jain, Huang, Hu and Lee, however, do not explicitly teach that the adjusted target weight values are bias weight values that correspond to one or more bias weights of the given output line, as is required by claim 21.
Gao nevertheless discloses that non-ideal behaviors of neurons can be compensated by fine-tuning bias weights (see e.g. section IV “Hopefield Network ADC”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Gao before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee such that the adjusted target weight values include bias weight values that correspond to one or more bias weights of the output line like taught by Gao. It would have been advantageous to one of ordinary skill to utilize such a combination because it can compensate for non-deal behaviors of neurons, as is taught by Gao (see e.g. section IV “Hopefield Network ADC”). Accordingly, Jain, Huang, Hu, Lee and Gao are considered to teach, to one of ordinary skill in the art, system like that of claim 21.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Jain, Huang, Hu and Lee, which is described above, over the article by Chakraborty (“GENIEx: A Generalized Approach to Emulating Non-Ideality in Memristive Xbars using Neural Networks”) described above, and also over the article by Gao (“Digital-to-Analog and Analog-to-Digital Conversion with Metal Oxide Memristors for Ultra-Low Power Computing”) described above.
Regarding claim 23, Jain, Huang, Hu and Lee teach a method like that of claim 22, as is described above, which entails performing a calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claim 23, Huang suggests that such a calibration process can converge the measured offset values of respective multiply-and-accumulate distribution data of respective output lines to the target offset value (see e.g. the abstract, which recites: “This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples.”). Hu provides a similar teaching (see e.g. paragraphs 0060-0061). Jain, Huang, Hu and Lee, however, do not explicitly teach that the calibration process comprises converging respective slopes of the multiply-and-accumulate distribution data, which are output from the respective output lines, to a target slope, as is required by claim 23. Moreover, while Huang further suggests that the calibration process comprises iteratively adjusting one or more target weight values, which correspond to one or more stored weights of the one or more output lines, and reprogramming the one or more stored weights of the one or more output lines, based on the adjusted target weight values to reduce residual line-to-line offset variation (see e.g. section III.B “Procedure of weights finetune”), Jain, Huang, Hu and Lee do explicitly teach that the adjusted target weight values are bias weight values that correspond to one or more bias weights of the given output line, as is further required by claim 23.
Chakraborty nevertheless generally teaches that the slopes of multiply-and-accumulate data (i.e. current) actually output by a resistive crossbar differ from the slopes of the ideal multiply-and-accumulate data output by such a crossbar (see e.g. section 3 “Analysis of NVM Non-Idealities” and particularly FIG. 2(a) therein). Chakraborty suggests that mitigation strategies (e.g. retraining of the neural network weights) can mitigate such non-idealities (see e.g. section 1 “Introduction,” which states: “To address this accuracy degradation, there have been efforts towards exploring techniques to model non-idealities and subsequently mitigating them [9-11]. The efficacy of these mitigation techniques strongly depend upon the modelling [9, 10, 12] approach to exhaustively capture the sources of the non-idealities and retraining of the neural network weights.”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee and Chakraborty before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu and Lee so as to mitigate the divergent slopes of multiply-and-accumulate data output by the respective output lines of the resistive processing unit array (i.e. to converge the respective slopes of multiply-and-accumulate data to a target, ideal slope), like generally taught by Chakraborty. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the DNN implemented on the processing unit array, as is evident from Chakraborty (see e.g. section 1 “Introduction”).
Gao generally teaches that non-ideal behaviors of neurons can be compensated by fine-tuning bias weights (see e.g. section IV “Hopefield Network ADC”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee, Chakraborty and Gao before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu, Lee and Chakraborty such that the adjusted target weight values include bias weight values that correspond to one or more bias weights of the output line like taught by Gao. It would have been advantageous to one of ordinary skill to utilize such a combination because it can compensate for non-deal behaviors of neurons, as is taught by Gao (see e.g. section IV “Hopefield Network ADC”). Accordingly, Jain, Huang, Hu, Lee, Chakraborty and Gao are considered to teach, to one of ordinary skill in the art, system like that of claim 23.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Jain, Huang, Hu, Lee and Chakraborty, which is described above, and also over the article by Gao (“Digital-to-Analog and Analog-to-Digital Conversion with Metal Oxide Memristors for Ultra-Low Power Computing”) described above.
Regarding claim 25, Jain, Huang, Hu, Lee and Chakraborty teach a system like that of claim 24, as is described above, which performs a first and second calibration process to calibrate a resistive processing unit array to reduce a variation between output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data that is generated and output from respective output lines of the resistive processing unit array during the calibration process. Like in claim 25, Jain further teaches that the obtained matrix comprises one of a computational matrix utilized to perform matrix computations for a linear system, and a trained synaptic weight matrix of a trained artificial neural network to perform inference processing (see e.g. section 4.1 “CxDNN: Approach and Overview:” Jain suggests that the obtained matrix can comprise a trained synaptic matrix of a trained artificial neural network, i.e. a DNN, to perform inference processing.). Jain, Huang, Hu, Lee and Chakraborty, however, do not explicitly disclose that the calibration process further comprises a third calibration process, which is performed subsequent to the second calibration process, to iteratively adjust one or more target bias weight values, which correspond to one or more stored bias weights of the one more of the output lines, and reprogram the one or more stored bias weights of the one or more output lines, based on the adjusted target bias weight values, to reduce residual offset variation between the output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data which is generated and output from respective output lines of the resistive processing unit array during the third calibration process, as is required by claim 25.
Gao nevertheless generally teaches that non-ideal behaviors of neurons can be compensated by fine-tuning bias weights (see e.g. section IV “Hopefield Network ADC”).
It would have been obvious to one of ordinary skill in the art, having the teachings of Jain, Huang, Hu, Lee, Chakraborty and Gao before the effective filing date of the claimed invention, to modify the calibration process taught by Jain, Huang, Hu, Lee and Chakraborty so as to include a third calibration process that finetunes bias weights like taught by Gao, which would entail iteratively adjusting one or more target bias weight values, which correspond to one or more stored bias weights of the one more of the output lines, and reprogramming the one or more stored bias weights of the one or more output lines, based on the adjusted target bias weight values, to reduce residual offset variation between the output lines of the resistive processing unit array with respect to multiply-and-accumulate distribution data which is generated and output from respective output lines of the resistive processing unit array during the third calibration process. It would have been advantageous to one of ordinary skill to utilize such a combination because it can compensate for non-deal behaviors of neurons, as is taught by Gao (see e.g. section IV “Hopefield Network ADC”). Accordingly, Jain, Huang, Hu, Lee, Chakraborty and Gao are considered to teach, to one of ordinary skill in the art, system like that of claim 25.
Response to Arguments
The Examiner acknowledges the Applicant’s amendments to claims 1, 2, 9, 10, 17, 18 and 22-24. In response to these amendments, the objections presented in the previous Office Action to claims 1-25 are respectfully withdrawn.
Regarding the 35 U.S.C. § 103 rejections, the Applicant argues that Hu fails to teach or suggest the limitation reciting, “wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for the respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the target weight values of the matrix based on the determined error for each output line, and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values, to reduce a variation between the output lines of the resistive processing unit array,” as in claim 1.
In response, the Examiner respectfully submits that Huang and Hu, in combination, provide such a teaching. In particular, like noted above, Huang teaches fine-tuning the weights of a resistive processing unit array to reduce a variation between output lines (i.e. columns/ADCs) of the processing unit array with respect to data that is generated and output from respective output lines of the processing unit array:
Compute-in-memory (CIM) has been proposed to accelerate the convolution neural network (CNN) computation by implementing parallel multiply and accumulation in analog domain. However, the subsequent processing is still preferred to be performed in digital domain. This makes the analog to digital converter (ADC) critical in CIM architectures. One drawback is the ADC error introduced by process variation. While research efforts are being made to improve ADC design to reduce the offset, we find that the accuracy loss introduced by the ADC error could be recovered by model weight finetune. In addition to compensate ADC offset, on-chip weight finetune could be leveraged to provide additional protection for adversarial attack that aims to fool the inference engine with manipulated input samples. Our evaluation results show that by adapting the model weights to the specific ADC offset pattern to each chip, the transferability of the adversarial attack is suppressed. For a chip being attacked by the C&W method, the classification for CIFAR-10 dataset will drop to almost 0%. However, when applying the similarly generated adversarial examples to other chips, the accuracy could still maintain more than 62% and 85% accuracy for VGG-8 and DenseNet-40, respectively.
(Abstract; emphasis added).
In this work, we leverage the ADC offset pattern (which is believed to be detrimental to the inference accuracy) but finetune the model weights to take its advantage against the adversarial attack on the CIM accelerator. In our evaluation, we find that the accuracy drop could be compensated by finetuning DNN parameters that adapt to ADC offset. This finetune, while recovering the inference accuracy, makes the DNN parameters slightly different from chip to chip, which brings us a byproduct: the chip will be robust to the adversarial examples generated by attacking other chips or software baseline. Explicitly, even if the adversary attacks one chip instance by manipulating the adversarial input, he/she could not use the same adversarial input to attack all the other chip instances due to the uniqueness of the DNN model for each chip.
(Section I “Introduction”).
In this work, instead of trying to defend against adversarial examples of a certain model in software, we aim to reduce the transferability of the adversarial examples among actual chips. It works like the aforementioned software defense method of introducing randomness into the network parameters. For CIM architecture that employs ADCs, there are intrinsic process variations that will introduce quantization error. As shown in previous prototype chip measurement results, even with precisely designed ADCs, the accuracy will be low if the ADCs all share the same references [24]. To achieve high accuracy, the references of each ADC need to be adjusted independently. This requirement for ADC reference adjustment and independent reference for each ADC brings additional hardware overhead. Alternatively, we could adjust the weights with several retrain epoch, namely “finetune” process. When the model is adapted to the ADC offset pattern, the inference accuracy could be recovered. The overhead of model finetune in software is much less than the implementation of adjustable ADC references on-chip. Now we could take advantages of the model finetune to reduce the transferability of the adversarial examples from software baseline to actual chips, and from one chip to another.
(Section II.C “Adversarial Defense”).
We now discuss the on-chip/off-chip hybrid finetune procedure to mitigate adversarial attack from the hardware standpoint. During the retraining, the feedforward propagation (inference) is first performed on-chip, and then the backpropagation and weight update are done off-chip by software. The detailed process is as follows: we will run the inference on a specific chip that captures its specific ADC offset pattern, then the prediction of the inference will be compared with the ideal label for the loss function; after obtaining the estimated loss, weights are updated through backpropagation in software; finally, the memory cells will be reprogramed to the new weights possibly with write-verify. In our evaluation, the partial sum will sample the ADC offset from the estimated distribution as in Fig. 3(a), and the distorted partial sum will be used as output feature map and saved for error and gradient calculation. The backpropagation and weight update are all directly use floating-point calculation as done in software.
(Section III.B “Procedure of weights finetune”).
We evaluate the proposed hybrid finetune defense method with VGG-8 and DenseNet-40 networks for CIFAR-10 dataset. The precision setting is 8-bit activation and 2-bit weight for VGG-8, and 8-bit activation and 8-bit weight for DenseNet-40. The software baseline accuracy is ~92% for both networks. For the weight finetune process, the batch size of retrain is 200, which means there are 250 iterations to finish the finetune in one epoch. Fig. 5 (a) & (b) shows the retrain curve of one specific chip that implements VGG-8 which uses Flash-ADC and SARADC, respectively. Chip with either Flash-ADC or SAR-ADC could recover the accuracy, however Flash-ADC has less initial accuracy drop and easier to be retrained to recover the high accuracy. This is consistent with the analysis in Section Ⅲ on the possible compensation of SA offsets for Flash-ADC. It is also seen that as the W/L decreases, it will be more difficult to retrain the model to recover the accuracy under process variations. When the W/L is small, which means the sense pass rate is also low, the accuracy may could not be fully recovered. It needed to be pointed out that the W/L reported here appears high since we use the minimum length (
L
m
i
n
) as the L in our simulation. Generally, two to three times of
L
m
i
n
will be used in the analog circuit to avoid very large process variation. We did not optimize the ADC with advanced offset cancellation techniques. Here our goal is just to show that by changing W/L, we could achieve different degrees of the process variation. Fig. 5 (c) & (d) presents the accuracy distribution of several retrain tests before finetune and after finetune collected from multiple chips, respectively. From the plot, we could observe that the accuracy recovery from finetune is generally achievable.
(Section IV. “Evaluation Results”).
As suggested by the above excerpts, such fine-tuning entails an iterative process that uses output data (e.g. an inference prediction) generated during each iteration of the process, wherein each iteration of the process comprises analyzing the output data to determine an error (i.e. a loss) between expected output data (e.g. an ideal label) and the generated output data, and iteratively adjusting the weight values based on the determined error (i.e. through backpropagation), and reprogramming the stored weight values in the processing unit array based on the respective adjusted weight values. Accordingly, Huang teaches an iterative calibration process, wherein each iteration of the process comprises analyzing the output data of the resistive processing unit array to determine an error between expected output data and the generated output data, iteratively adjusting the target weight values of the matrix based on the determined error, and reprogramming the stored weight values of the matrix in the processing unit array based on the respective adjusted weight values. Huang however does not explicitly disclose that each iteration of the calibration process particularly comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on these errors for each output line, as is claimed.
Hu nevertheless teaches increasing computational accuracy in a resistive processing unit array through an iterative process that comprises analyzing multiply-and-accumulate distribution data (i.e. output signals) obtained for respective output lines of the resistive processing unit array, to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data (i.e. ideal output signals) and the multiply-and-accumulate distribution data obtained for the respective output line, and to adjust the weight values of the resistive processing unit array based on the determined error for each output line (see e.g. paragraphs 0011 and 0060-0061).
Accordingly, like noted above, it would have been obvious to one of ordinary skill in the art, having the teachings of Huang and Hu (and Jain) before the effective filing date of the claimed invention, to modify the calibration process taught by Huang (and Jain) such that it applies to individual respective output lines like taught by Hu, i.e. wherein each iteration of the calibration process comprises analyzing multiply-and-accumulate distribution data obtained for respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, wherein the target weight values are iteratively adjusted based on the error for each output line, and wherein the error that is determined for a given output line comprises a difference between a target offset value and a measured offset value associated with the multiply-and-accumulate distribution data obtained for the given output line. It would have been advantageous to one of ordinary skill to utilize such a combination because it would improve the accuracy of the resistive processing unit array, as is suggested by Hu (see e.g. paragraph 0011). Accordingly, the Examiner respectfully submits that at least Huang and Hu teach the limitation reciting, “wherein each iteration of the calibration process comprises analyzing the multiply-and-accumulate distribution data obtained for the respective output lines of the resistive processing unit array to determine, for each respective output line, an error between an expected multiply-and-accumulate distribution data and the multiply-and-accumulate distribution data obtained for the respective output line, iteratively adjusting the target weight values of the matrix based on the determined error for each output line, and reprogramming the stored weight values of the matrix in the resistive processing unit array based on the respective adjusted target weight values, to reduce a variation between the output lines of the resistive processing unit array,” as in claim 1.
The Applicant’s remaining arguments with respect to the 35 U.S.C. § 103 rejections have been considered, but are moot in view of the new grounds of rejection (i.e. including Lee) provided above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BLAINE T BASOM whose telephone number is (571)272-4044. The examiner can normally be reached Monday-Friday, 9:00 am - 5:30 pm, EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Ell can be reached at (571)270-3264. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BTB/
5/16/2026
/TAN H TRAN/Primary Examiner, Art Unit 2141
Prosecution Insights