DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Acknowledgement is made of Applicant’s claim amendments on 02/04/2026. The claim amendments are entered. Presently, claims 1-27 remain pending. Claims 1-2, 9, 13, 19, and 24-25 have been amended and claim 27 is newly added.
Response to Arguments
Applicant’s arguments, see pages 11-16 of remarks, filed 02/04/2026, with respect to claims 1-26 have been fully considered and are persuasive. The 35 USC 101 rejection of claims 1-26 has been withdrawn.
Regarding the 35 USC 103 rejection, Applicant’s arguments with respect to claim(s) 1-26 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Allowable Subject Matter
Claims 6 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 13-15, 19-21, 24-25 and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Darvish et al. (US-20200202213-A1) in view of Tzoufras (US-20200311522-A1), Yim et al. (US-20190130255-A1), and Siracusa et al. (US 20200380301 A1).
Regarding Claim 1,
Darvish teaches a computer-implemented method comprising: for a neural network that includes a plurality of network layers (para [0112] The test neural network model used in this experiment was a ResNet50 neural network having 16 layers, and the training was performed using the ImageNet database.) and one or more target hardware devices on which the neural network is to run (para [0023] At the same time, hardware accelerators that can be used with NNs include specialized NN processing units, such as tensor processing units (TPUs) and Field Programmable Gate Arrays (FPGAs) programmed to accelerate neural network processing.),
determining at least two points within the neural network that generate numeric values during execution of the neural network, the numeric values represented as a floating point data type (para [0041] The normal-precision values can be represented in 16-, 32-, 64-bit, or another suitable floating-point format. For example, a portion of values representing the neural network can be received, including edge weights, activation values, or other suitable parameters for quantization. Weights and activation values are number values that are generated at neurons and layers (i.e., at least two point) in the neural network.), wherein one or more of the at least two points is associated with an input layer or a hidden layer of the neural network (para [0049] As shown in FIG. 2, a first set 210 of nodes (including nodes 215 and 216) form an input layer. Each node of the set 210 is connected to each node in a first hidden layer formed from a second set 220 of nodes (including nodes 225 and 226). A second hidden layer is formed from a third set 230 of nodes, including node 235. and para [0058] At process block 310, a first tensor having one or more NN parameter values represented in a normal-precision floating-point format is obtained. The first tensor can include values of one or more, or all, of the parameters of one or more layers of the NN. For example, this can include values of activation weights, edge weights, etc. The first tensor can take the form of a matrix, for example.);
quantizing the neural network at the at least two points within the neural network, wherein the at least two points are two or more of inputs to the plurality of network layers, outputs of the plurality of network layers, or intermediate values of the plurality of network layers (para [0043] In some examples, the quantized values are actually stored in memory as normal floating-point values. In other words, the quantization domain 140 quantizes the inputs, weights, and activations for a neural network model, but the underlying operations are performed in normal floating-point. Activations and weights which can be input, outputs, or intermediate values of the layers are quantized.), the quantizing including changing the floating point data type for the numeric values to the different data type, the quantizing based on one or more characteristics of the one or more target hardware devices including that the one or more target hardware devices supports the different data type (para [0024] Traditionally NNs have been trained and deployed using single-precision floating-point (32-bit floating-point or float32 format). However, it has been shown that lower precision floating-point formats, such as 16-bit floating-point (float16) or fixed-point can be used to perform inference operations with minimal loss in accuracy. On specialized hardware, such as FPGAs, reduced precision formats can greatly improve the latency and throughput of DNN processing. 16-bit floating-point (i.e., different data type.);
generating performance information for the neural network following the quantizing (para [0067] For example, the experimental results discussed in Section X below show that the accuracy of training results for quantized NNs can be improved to match, or even exceed, the accuracy achieved when training an equivalent non-quantized NN. Accuracy (i.e., performance information) generated after quantizing the neural network.);
presenting, a visualization of the performance information (para [0110]-[0111] Results are charted for the baseline float32 model (represented by the plot labeled “baseline”) as well as models with parameters in four different quantized-precision formats: an 8-bit shared exponent format with one mantissa bit and one sign bit (represented by the plots labeled “1M_1S_8Exp_bk16 (scaled)” and “1M_1S_8Exp_bk16 (no scaling)”); The accuracies (i.e. performance) is plotted on a graph which is computed based on the quantized neural network.); and
generating executable program code for the neural network following the quantizing and based on changing the floating point data type for the numeric values to the different data type (para [0038] The compiler 132 analyzes the source code and data (e.g., the weights and biases learned from training the model) provided for a neural network model and transforms the model into a format that can be accelerated in the quantization domain 140 and/or an optional neural network accelerator 180, which will be described in further detail below.).
Darvish does not explicitly disclose
analyzing the numeric values generated at the at least two points during execution of the neural network to derive statistics for the at least two points within the neural network and to determine whether the floating point data type for the numeric values of the at least two points are to be changed to a different data type that is different from the floating point data type, the statistics determined based on the analysis of the numeric values generated at the at least two points during execution of the neural network;
presenting, through a first User Interface, a visualization of the statistics derived for the at least two points based on the analysis of the numeric values generated at the at least two points during execution of the neural network;
through the first User Interface or a second User Interface,
However, Tzoufras (US 20200311522 A1) teaches
determining at least two points within the neural network that generate numeric values during execution of the neural network (para [0119] “non-quantized weights 624 stored as floating point values; and [0120] non-quantized activations 626 stored as floating point values;” weights and activations (i.e., at least two points).), the numeric values represented as a floating point data type, wherein one or more of the at least two points is associated with an input layer or a hidden layer of the neural network (para [0060] “For example, in some embodiments, on-chip memory 204 stores trained weights for a plurality of layers of the ANN, including, in some circumstances, one or more quantized (e.g., binary) layers (stored as quantized weights 208) and one or more non-quantized (e.g., floating point) layers (e.g., stored as non-quantized weights 210).”);
analyzing the numeric values generated at the at least two points during execution of the neural network to derive statistics for the at least two points within the neural network… (para [0074] “For example, the neural network includes a combination of quantized (e.g., binarized) layers and floating point (e.g., non-binarized) layers. In some implementations, each of the at least one quantized layer comprises (412) a binary layer.”), the statistics determined based on the analysis of the numeric values generated at the at least two points during execution of the neural network (para [0068] “Graph 300 illustrates the accuracy of a Network In Network (NIN) neural network as the training process proceeds.” Training (i.e., execution) accuracy (i.e., statistic).);
presenting, through a first User Interface (para [0115] “a user interface module 616 for receiving information from one or more input device 613 of user interface 609, and to convey information to a user of computer system 630 via one or more display devices 611;”), a visualization of the statistics derived for the at least two points based on the analysis of the numeric values generated at the at least two points during execution of the neural network (fig. 3; para [0068] “Graph 300 illustrates the accuracy of a Network In Network (NIN) neural network as the training process proceeds.” accuracy (i.e., statistic).);
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Darvish with the quantization of Tzoufras.
Doing so would allow for improving the speed and efficiency of an ANN inference (classification) process (Tzoufras para [0008]).
Yim (US 20190130255 A1) teaches
analyzing the numeric values generated at the at least two points during execution of the neural network (para [0041] “The processor 110 may analyze statistical distribution by neuron for parameter values of a floating-point type used in each neuron included in each of feature maps and kernels, from the pre-trained neural network data. The processor 110 may analyze the statistical distribution by obtaining statistics by neuron for parameter values of floating-point type activations, weights, and biases used in each neuron while a neural network is pre-trained.” activations (i.e., at least two points). para [0058] “Referring to FIG. 5, in operation S100, the processor 110 may obtain post-activations processed in a layer included in a floating point-based neural network. Post-activation may be post-activation of a neural network that is pre-trained on a floating point basis. The post-activation may be, for example, a value obtained by applying an activation function to the sum of activations received from a previous layer.”) to derive statistics for the at least two points within the neural network (para [0059] “Next, in operation S200, the processor 110 may derive statistical characteristics of at least some of the obtained post-activations. In example embodiments, the processor 110 may derive statistical characteristics of post-activations having a positive value among the post-activations.”) and to determine whether the floating point data type for the numeric values of the at least two points are to be changed to a different data type that is different from the floating point data type (para [0060] “Next, in operation S300, the processor 110 may calculate a step size for quantization based on the statistical characteristics (S300). The step size may be, for example, a quantization interval for floating point values. In example embodiments, the processor 110 may calculate the step size for the quantization through a closed-form equation based on a plurality of parameters derived as the statistical characteristics.” Para [0062] “Next, in operation S400, the processor 110 may determine a fraction length for the fixed-point type neural network based on the step size.” Fixed point (i.e., different data type).),
presenting, a visualization of the statistics derived for the at least two points based on the analysis of the numeric values generated at the at least two points during execution of the neural network (para [0069] “FIG. 6B shows a graph of a plurality of post-activations for each layer and generalized gamma distribution that is approximated by being curve-fitted by the plurality of post-activations. An x-axis of each graph may denote normalized values of various values that the post-activations can have, and a y-axis may denote probability density.”);
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Darvish with the method of deriving statistics from post activations to determine a quantized step size.
Doing so would allow for quantizing the neural network to a fixed point-based neural network that improves quantization error and minimizes performance degradation compared to a floating point-based neural network is provided, thereby improving a memory area, a bandwidth, and power consumption (Yim para [0048]).
Siracusa (US 20200380301 A1) teaches
presenting, through the first User Interface or a second User Interface, a second visualization of the performance information (para [0073] FIG. 5 illustrates a training graphical user interface 500 for generating a machine learning model. FIG. 5 is a follow-on to the graphical user interface 400 depicted in FIG. 4. The graphical user interface 500 lists the name 502 of the project file and the model template 504. multiple visualizations can be generated for a plurality of machine learning models. para [0075] The training graphical user interface 500 depicts a line plot graph of a training accuracy line 518 and the validation accuracy line 520 over the number of iterations for the model.);
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish with the machine learning interface of Siracusa.
Doing so would allow for providing the developer further insights into the trained model's accuracy for different classes. For example, the training graphical user interface can depict the precision and recall (Siracusa para [0075]).
Regarding Claim 3,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1. Darvish further teaches wherein the at least two points are determined automatically based on a type of the one or more target hardware devices (para [0030] As used herein, the term “quantized-precision floating-point” refers to a floating-point number format where two or more values of a tensor have been modified to emulate neural network hardware.).
Regarding Claim 4,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1. Darvish teaches further comprising: generating the statistics for the numeric values based on a running of the neural network on instrumentation data (para [0037] In addition to the source code, the memory 125 can also store training data. The training data includes a set of input data for applying to the neural network model 200 and a desired output from the neural network model for each respective dataset of the input data. Input data (i.e. instrumentation data) is used to determine accuracy. para [0111] As shown, scaling the learning rate in the disclosed manner improves the accuracy of training compared to using the same learning rate as the baseline float32 model. For example, the accuracy achieved during training of the model quantized in the 8-bit shared exponent format with four mantissa bits and one sign bit is higher than the accuracy achieved by the baseline float32 model.).
Regarding Claim 13,
Claim 13 is the computer-readable medium corresponding to the method of claim 1. Claim 13 is substantially similar to claim 1 and is rejected on the same grounds.
Regarding Claim 14,
Claim 14 is the computer-readable medium corresponding to the method of claim 3. Claim 14 is substantially similar to claim 3 and is rejected on the same grounds.
Regarding Claim 15,
Claim 15 is the computer-readable medium corresponding to the method of claim 4. Claim 15 is substantially similar to claim 4 and is rejected on the same grounds.
Regarding Claim 19,
Claim 19 is the apparatus corresponding to the method of claim 1. Claim 19 is substantially similar to claim 1 and is rejected on the same grounds.
Regarding Claim 20,
Claim 20 is the apparatus corresponding to the method of claim 3. Claim 20 is substantially similar to claim 3 and is rejected on the same grounds.
Regarding Claim 21,
Claim 21 is the apparatus corresponding to the method of claim 4. Claim 21 is substantially similar to claim 4 and is rejected on the same grounds.
Regarding Claim 24,
Darvish teaches a computer-implemented method comprising:
for a neural network that includes a plurality of network layers (para [0112] The test neural network model used in this experiment was a ResNet50 neural network having 16 layers, and the training was performed using the ImageNet database.), determining a plurality of points within the neural network that generate numeric values during execution of the neural network, the numeric values represented as a floating point data type by the neural network (para [0041] The normal-precision values can be represented in 16-, 32-, 64-bit, or another suitable floating-point format. For example, a portion of values representing the neural network can be received, including edge weights, activation values, or other suitable parameters for quantization. Weights and activation values are number values that are generated at neurons and layers (i.e., at least two point) in the neural network.);
executing, by one or more processors, the neural network, the executing utilizing instrumentation data (para [0037] In addition to the source code, the memory 125 can also store training data. The training data includes a set of input data for applying to the neural network model 200 and a desired output from the neural network model for each respective dataset of the input data.);
quantizing, by the one or more processors, at least two of the plurality points within the neural network (para [0043] In some examples, the quantized values are actually stored in memory as normal floating-point values. In other words, the quantization domain 140 quantizes the inputs, weights, and activations for a neural network model, but the underlying operations are performed in normal floating-point. Activations and weights which can be input, outputs, or intermediate values of the layers are quantized.), the quantizing including changing the floating point data type for the numeric values to the different data type, the quantizing based on a quantization scheme and being constrained by a limitation of a target hardware device on which the neural network is to run (para [0024] Traditionally NNs have been trained and deployed using single-precision floating-point (32-bit floating-point or float32 format). However, it has been shown that lower precision floating-point formats, such as 16-bit floating-point (float16) or fixed-point can be used to perform inference operations with minimal loss in accuracy. On specialized hardware, such as FPGAs, reduced precision formats can greatly improve the latency and throughput of DNN processing. 16-bit floating-point (i.e., different data type.);
generating, by the one or more processors, performance information for the neural network following the quantizing (para [0067] For example, the experimental results discussed in Section X below show that the accuracy of training results for quantized NNs can be improved to match, or even exceed, the accuracy achieved when training an equivalent non-quantized NN.);
presenting the performance information on the display (para [0110]-[0111] Results are charted for the baseline float32 model (represented by the plot labeled “baseline”) as well as models with parameters in four different quantized-precision formats: an 8-bit shared exponent format with one mantissa bit and one sign bit (represented by the plots labeled “1M_1S_8Exp_bk16 (scaled)” and “1M_1S_8Exp_bk16 (no scaling)”); The accuracies (i.e. performance) is plotted on a graph which is computed based on the quantized neural network.); and
Darvish does not explicitly disclose
analyzing, by the one or more processors, the numeric values generated at the at least two points during execution of the neural network to derive statistics for the at least two points within the neural network and to determine whether the floating point datatype for the numeric values of the at least two points are to be changed to a different datatype that is different from the floating point data type, the statistics determined based on the analysis of the numeric values generated at the at least two points during the executing;
presenting … through the first User Interface or a second User Interface on the display.
presenting the statistics derived for the at least two points through a first User Interface on a display;
changing the quantization scheme and repeating the quantizing step, the generating step, and the presenting the performance information step.
However, Tzoufras (US 20200311522 A1) teaches
for a neural network that includes a plurality of network layers (para [0119] “non-quantized weights 624 stored as floating point values; and [0120] non-quantized activations 626 stored as floating point values;” weights and activations (i.e., at least two points).), determining a plurality of points within the neural network that generate numeric values during execution of the neural network, the numeric values represented as a floating point data type by the neural network (para [0060] “For example, in some embodiments, on-chip memory 204 stores trained weights for a plurality of layers of the ANN, including, in some circumstances, one or more quantized (e.g., binary) layers (stored as quantized weights 208) and one or more non-quantized (e.g., floating point) layers (e.g., stored as non-quantized weights 210).”);
analyzing, by the one or more processors, the numeric values generated at the at least two points during execution of the neural network to derive statistics for the at least two points within the neural network… (para [0074] “For example, the neural network includes a combination of quantized (e.g., binarized) layers and floating point (e.g., non-binarized) layers. In some implementations, each of the at least one quantized layer comprises (412) a binary layer.”), the statistics determined based on the analysis of the numeric values generated at the at least two points during the executing (para [0068] “Graph 300 illustrates the accuracy of a Network In Network (NIN) neural network as the training process proceeds.” Training (i.e., execution) accuracy (i.e., statistic).);
presenting the statistics derived for the at least two points through a first User Interface on a display (para [0115] “a user interface module 616 for receiving information from one or more input device 613 of user interface 609, and to convey information to a user of computer system 630 via one or more display devices 611;” fig. 3; para [0068] “Graph 300 illustrates the accuracy of a Network In Network (NIN) neural network as the training process proceeds.” accuracy (i.e., statistic).);
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Darvish with the quantization of Tzoufras.
Doing so would allow for improving the speed and efficiency of an ANN inference (classification) process (Tzoufras para [0008]).
Yim (US 20190130255 A1) teaches
analyzing, by the one or more processors, the numeric values generated at the at least two points during execution of the neural network (para [0041] “The processor 110 may analyze statistical distribution by neuron for parameter values of a floating-point type used in each neuron included in each of feature maps and kernels, from the pre-trained neural network data. The processor 110 may analyze the statistical distribution by obtaining statistics by neuron for parameter values of floating-point type activations, weights, and biases used in each neuron while a neural network is pre-trained.” activations (i.e., at least two points). para [0058] “Referring to FIG. 5, in operation S100, the processor 110 may obtain post-activations processed in a layer included in a floating point-based neural network. Post-activation may be post-activation of a neural network that is pre-trained on a floating point basis. The post-activation may be, for example, a value obtained by applying an activation function to the sum of activations received from a previous layer.”) to derive statistics for the at least two points within the neural network (para [0059] “Next, in operation S200, the processor 110 may derive statistical characteristics of at least some of the obtained post-activations. In example embodiments, the processor 110 may derive statistical characteristics of post-activations having a positive value among the post-activations.”) and to determine whether the floating point datatype for the numeric values of the at least two points are to be changed to a different datatype that is different from the floating point data type (para [0060] “Next, in operation S300, the processor 110 may calculate a step size for quantization based on the statistical characteristics (S300). The step size may be, for example, a quantization interval for floating point values. In example embodiments, the processor 110 may calculate the step size for the quantization through a closed-form equation based on a plurality of parameters derived as the statistical characteristics.” Para [0062] “Next, in operation S400, the processor 110 may determine a fraction length for the fixed-point type neural network based on the step size.” Fixed point (i.e., different data type).),
presenting the statistics derived for the at least two points through a first User Interface on a display (para [0069] “FIG. 6B shows a graph of a plurality of post-activations for each layer and generalized gamma distribution that is approximated by being curve-fitted by the plurality of post-activations. An x-axis of each graph may denote normalized values of various values that the post-activations can have, and a y-axis may denote probability density.”);
changing the quantization scheme and repeating the quantizing step, the generating step, and the presenting the performance information step (para [0040] “The pre-trained neural network data may be data repeatedly trained with floating-point type parameters. Neural network training may be repeatedly performed by receiving training set data as an input, and then repeatedly performed with test set data again, but is not limited thereto.” The steps of quantizing, generating statistics, and displaying can be repeated.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Darvish with the method of deriving statistics from post activations to determine a quantized step size.
Doing so would allow for quantizing the neural network to a fixed point-based neural network that improves quantization error and minimizes performance degradation compared to a floating point-based neural network is provided, thereby improving a memory area, a bandwidth, and power consumption (Yim para [0048]).
Siracusa (US 20200380301 A1) teaches
presenting the performance information through the first User Interface or a second User Interface on the display (para [0073] FIG. 5 illustrates a training graphical user interface 500 for generating a machine learning model. FIG. 5 is a follow-on to the graphical user interface 400 depicted in FIG. 4. The graphical user interface 500 lists the name 502 of the project file and the model template 504. multiple visualizations can be generated for a plurality of machine learning models. para [0075] The training graphical user interface 500 depicts a line plot graph of a training accuracy line 518 and the validation accuracy line 520 over the number of iterations for the model.);
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish with the machine learning interface of Siracusa.
Doing so would allow for providing the developer further insights into the trained model's accuracy for different classes. For example, the training graphical user interface can depict the precision and recall (Siracusa para [0075]).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish with the machine learning interface of Siracusa.
Doing so would allow for providing the developer further insights into the trained model's accuracy for different classes. For example, the training graphical user interface can depict the precision and recall (Siracusa para [0075]).
Regarding Claim 25,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 24. Yim further teaches wherein the quantization scheme indicates the different data type (para [0042] “The processor 110 may determine a fixed-point representation of statistical distribution by neuron that statistically covers a distribution range of parameter values based on the analyzed statistical distribution by neuron. Thus, a floating-point type neural network may be converted to a fixed-point type neural network.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the neural network of Darvish with the method of deriving statistics from post activations to determine a quantized step size.
Doing so would allow for quantizing the neural network to a fixed point-based neural network that improves quantization error and minimizes performance degradation compared to a floating point-based neural network is provided, thereby improving a memory area, a bandwidth, and power consumption (Yim para [0048]).
Regarding Claim 27,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1. Yim further teaches wherein the different data type is one of an integer data type, a fixed point data type, or a Boolean data type (para [0042] “The processor 110 may determine a fixed-point representation of statistical distribution by neuron that statistically covers a distribution range of parameter values based on the analyzed statistical distribution by neuron. Thus, a floating-point type neural network may be converted to a fixed-point type neural network.”), or the floating point data type is a double precision floating point or a single precision floating point and the different data type is a half precision floating point data type.
Claims 2, 7, 9, 18, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Hasselgren et al. (US-20200272162-A1).
Regarding Claim 2,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the executable program code generated for the neural network is executable on the target hardware device
However, Hasselgren further teaches wherein the executable program code generated for the neural network is executable on the target hardware device (para [0067] In an embodiment, the computer code depicted in example 800 can be executed by any suitable system, such as the system described in connection with FIG. 1. In an embodiment, the computer code depicted in example 800 can be executed by a system such as a computer system and/or graphics system.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the quantization method of Hasselgren.
Doing so would allow for improved inference performance of the neural network due to a reduced total number of quantization operations (Hasselgren para [0020]).
Regarding Claim 7,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 4.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the statistics are generated for the numeric values at each of the at least two points and the statistics include at least one of: a minimum range value, a maximum range value, a number of times the numeric values are zero, or an indication whether the numeric values are always an integer
However, Hasselgren further teaches wherein the statistics are generated for the numeric values at each of the at least two points and the statistics include at least one of: a minimum range value, a maximum range value, a number of times the numeric values are zero, or an indication whether the numeric values are always an integer (para [0054] In an embodiment, the data range 510 comprises a value of 52 mV that represents the difference between the smallest minimum value of the data signal 506 and data signal 508 and the largest maximum value of the data signal 506 and data signal 508.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the quantization method of Hasselgren.
Doing so would allow for improved inference performance of the neural network due to a reduced total number of quantization operations (Hasselgren para [0020]).
Regarding Claim 9,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the quantizing includes applying a quantization scheme that specifies allowable formats of the different data type.
However, Hasselgren further teaches wherein the quantizing includes applying a quantization scheme that specifies allowable formats of the different data type. (para [0052] In an embodiment, the determined resolution for the data signal 506 and data signal 508 utilizes an 8-bit fixed point representation. And para [0060] In an embodiment, the system performing the process 600 establishes 612 a base value for a quantization format based on the combined minimum. In an embodiment, the base value for the quantization format determines the base value of the quantization format. In an embodiment, the base value is an approximate or exact value of the combined minimum. In an embodiment, the system performing the process 600 establishes 614 a range for the quantization format based on the difference between the combined minimum and maximum.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the quantization method of Hasselgren.
Doing so would allow for improved inference performance of the neural network due to a reduced total number of quantization operations (Hasselgren para [0020]).
Regarding Claim 18,
Claim 18 is the computer-readable medium corresponding to the method of claim 7. Claim 18 is substantially similar to claim 7 and is rejected on the same grounds.
Regarding Claim 23,
Claim 23 is the apparatus corresponding to the method of claim 7. Claim 23 is substantially similar to claim 7 and is rejected on the same grounds.
Claims 5, 16, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Chai et al. (US-20200134461-A1).
Regarding Claim 5,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 4.
Darvish, Tzoufras, Yim, and Siracusa does not explicitly disclose
wherein the generating the statistics includes assigning the numeric values to power of two bins representing at least one of range or precision of the numeric values.
However, Chai (US 20200134461 A1) teaches
wherein the generating the statistics includes assigning the numeric values to power of two bins representing at least one of range or precision of the numeric values (para [0061] The selection of values that are integer powers-of-two represents a quantization of high-precision weights 114.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, Siracusa with the dynamic precision weights of Chai.
However, because low-precision weights may contain fewer bits than high-precision weights, fewer operations may be required to read low-precision weights from memory than may be required to read high-precision weights from memory (Chai para [0033]).
Regarding Claim 16,
Claim 16 is the computer-readable medium corresponding to the method of claim 5. Claim 16 is substantially similar to claim 5 and is rejected on the same grounds.
Regarding Claim 22,
Claim 22 is the apparatus corresponding to the method of claim 5. Claim 22 is substantially similar to claim 5 and is rejected on the same grounds.
Claim(s) 8 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Bonnell et al. (US-20200110580-A1).
Regarding Claim 8,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the visualization of the statistics derived for the at least two points includes histogram heat map elements that present range information and precision information for the numeric values at the at least two points within the neural network.
However, Bonnell (US 20200110580 A1) teaches
wherein the visualization of the statistics derived for the at least two points includes histogram heat map elements that present range information and precision information for the numeric values at the at least two points within the neural network (para [0002] Histograms have been employed to approximate data distributions in application performance data. A high dynamic range (HDR) histogram supports recording and analyzing of sampled data value counts across a configurable integer value range with configurable value precision within the range. Value precision is expressed as the number of significant digits in the value recording and provides control over value quantization behavior across the value range and the subsequent value resolution.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the method of storing floating point values in integer representations for histogram recording.
Doing so would allow for providing control over value quantization behavior across the value range and the subsequent value resolution (Bonnell para [0002])
Claims 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Li et al. (US-20210232890-A1).
Regarding Claim 10,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the performance information includes at least one of inference accuracy, inference time, or memory usage.
However, Li (US 20210232890 A1) teaches
wherein the performance information includes at least one of inference accuracy, inference time, or memory usage (para [0074] The obtained mixed bits in ResNet20 for each layer are shown in Table 2. It is interesting to note that most of the bits in the final quantization scheme are 1, contributing much to the impressive compression ratio (25.6). This also shows that there is a lot of redundancy among the neural network layers. In addition, the compressed model achieves a prediction accuracy of 92.18% on the test set, which is better than the original full precision model (92.06%) based on our own implementation.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the quantization scheme of Li.
Doing so would allow for utilizing quantization schemes to further improve the compression ratio and prediction accuracy (Li para [0038])
Regarding Claim 11,
Darvish, Tzoufras, Yim, Siracusa, and Li teach the computer-implemented method of claim 10. Li further teaches wherein the inference accuracy is determined based on a user selected metric function (para [0056] Given the output of the neural network from the training data batch, a training data loss for the neural network may be determined (115) given the training data output and a loss function that comprises a loss component related to prediction accuracy of the neural network and a compression component related to memory size of parameter values of the neural network after quantization. In one or more embodiments, the loss function may be a function such as Eq. (2). Given the loss, at least some of the parameter values of the neural network may be updated (120) using the training data loss. For example, in embodiments, parameter values may be updated using gradient descent. Loss function (i.e. selected metric function).).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Yim et al. (US-20190180177-A1).
Regarding Claim 12,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 1.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the quantizing is based on at least one of the following user adjustable options: selected layers from the plurality of network layers of the neural network, an outlier threshold, an inclusion threshold, or a rounding mode.
However, Yim2 (US 20190180177 A1) teaches
wherein the quantizing is based on at least one of the following user adjustable options: selected layers from the plurality of network layers of the neural network, an outlier threshold, an inclusion threshold, or a rounding mode (para [0049] The parameter adjusting module 120 may select an object layer, a quantization parameter of which is to be updated, from among the plurality of layers of the fixed point neural network, in operation S120. According to an example embodiment, when the fixed point neural network includes N sequentially processed layers, the parameter adjusting module 120 may select the last layer of the plurality of layers, that is, an N.sup.th layer, as the object layer.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the selection of a layer for quantization updates of Yim2.
Doing so would allow for adjusting quantization parameters to improve the performance of the neural network while not increasing the complexity (Yim2 para [0046])
Claim(s) 26 is rejected under 35 U.S.C. 103 as being unpatentable over the combination of Darvish/Tzoufras/Yim/Siracusa, as applied above, and further in view of Gudovskiy et al. (US-20200097802-A1).
Regarding Claim 26,
Darvish, Tzoufras, Yim, and Siracusa teach the computer-implemented method of claim 24.
Darvish, Tzoufras, Yim, and Siracusa do not explicitly disclose
wherein the floating point data type is at least one of double precision floating point or single precision floating point and the quantization scheme constrains the changing the floating point data type to an 8-bit integer data type or a half precision floating point data type
However, Gudovskiy (US 20200097802 A1) teaches
wherein the floating point data type is at least one of double precision floating point or single precision floating point and the quantization scheme constrains the changing the floating point data type to an 8-bit integer data type or a half precision floating point data type (fig. 4; para [0069] FIG. 4 shows the results of inference accuracies of 50,000 images from an ImageNet validation dataset. The leftmost column indicates model type (fp32: number of single-precision floating points, uint8: 8-bit unsigned integer array, etc.), and the remaining columns indicate, from left to right: weight data size; activation, i.e., feature map size; top-most answer (top-1) accuracy; and top five answer (top-5) accuracy.).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify the teachings of Darvish, Tzoufras, Yim, and Siracusa with the quantization scheme of Gudovskiy.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HENRY K NGUYEN whose telephone number is (571)272-0217. The examiner can normally be reached Mon - Fri 7:00am-4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Li B Zhen can be reached at 5712723768. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HENRY NGUYEN/Examiner, Art Unit 2121
Prosecution Insights