Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/28/2026 has been entered.
Remarks
This Office Action is responsive to Applicants' Amendment filed on April 28, 2026, in which claims 1, 4, 7, 8, 11, 14, 15, 16, 17, and 20 are currently amended. Claim 3 is cancelled. Claims 1, 2, and 4-20 are currently pending.
Response to Arguments
The rejections to claim 3 under 35 U.S.C. § 112(b) is hereby withdrawn, as necessitated by applicant's amendments and remarks made to the rejections.
Applicant’s arguments with respect to rejection of claims 1, 2, and 4-20 under 35 U.S.C. 102/103 based on amendment have been considered and are persuasive. The argument is moot in view of a new ground of rejection set forth below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 2, 4, 5, 9, 10, and 12-15 are rejected under U.S.C. §103 as being unpatentable over the combination of Xu (“ReCU: Reviving the Dead Weights in Binary Neural Networks”, 2021) and Jain (“TRAINED QUANTIZATION THRESHOLDS FOR ACCURATE AND EFFICIENT FIXED-POINT INFERENCE OF DEEP NEURAL NETWORKS”, 2020).
Regarding claim 1, Xu teaches A method of training a binarized neural network (BNN), the method comprising: generating a binarized weight set by applying a clipping function to a weight set;([p. 1 §1] "a binary neural network (BNN) restricts the weights and activations to only two possible values, i.e.,-1 and +1" [p. 4] "ReCU(w)=max minw,Q(τ) ,Q(1−τ)" [p. 6] "binarize the inputs and the revived weights using the sign function of Eq. (2)" ReCU clamp unit interpreted as clipping function.)
wherein the clipping function includes a first threshold value as a lower limit of a range of the clipping function and a second threshold value as an upper limit of the range of the clipping function;([p. 4] "Q(τ) and Q(1−τ) respectively denote the τ quantile and 1−τ quantile [55] of W […] ReCU relocates w to Q(1−τ) if it is smaller than Q(1−τ), and to Q(τ) if it is larger than Q(τ)" Xu's ReCU function is a clamp: values below Q(1-τ) (first/lower threshold) are moved up to Q(1-τ) and values above Q(τ) (second/upper threshold) are moved down to Q(τ))
generating output data by sequentially performing a forward computation on the binarized neural network based on input data and the binarized weight set;([p. 6 §4.3] "in the forward propagation, we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively. Then, we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2). Finally, we complete the binary convolution using Eq. (3) for the forward propagation")
generating a gradient of the weight set, ([p. 3] "we use the simple STE [4] in this paper to compute the approximate gradient of the loss w.r.t. wr ∈ Wr")
by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data; and([p. 6 §4.3] "During backpropagation, we derive the gradients w.r.t. W and A using Eq. (5) and Eq. (6), respectively, and update W using the stochastic gradient descent (SGD) described in Sec. 5.1.")
training the binarized neural network, comprising:([p. 2 §1] "In this paper, we present a novel perspective to improve the effectiveness and training efficiency of BNN" [p. 6 §5.1] "Training Details. Our network is trained from scratch without depending on a pre-trained model. For all experiments, we use SGD for optimization with a momentum of 0.9 and the weight-decay is set to 5e-4")
updating the weight set based on the gradient of the weight set; and([p. 6 §4.3] "During backpropagation, we derive the gradients w.r.t. W and A using Eq. (5) and Eq. (6), respectively, and update W using the stochastic gradient descent (SGD) described in Sec. 5.1")
changing the range of the clipping function.([p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tao is the parameter that controls the clipping range.).
However, Xu does not explicitly teach a gradient of the first threshold value, and a gradient of the second threshold value
wherein changing the range of the clipping function comprises updating the first threshold value using the gradient of the first threshold value of the clipping function and/or updating the second threshold value using the gradient of the second threshold value of the clipping function.
Jain, in the same field of endeavor, teaches a gradient of the first threshold value, and a gradient of the second threshold value ([p. 1 §1] "uniform quantization between two clipping thresholds" [p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons" [p. 3] "the exact clipping thresholds of x in the real domain are xn=s·(n−0.5) and xp=s·(p+0.5). [...] Role of threshold gradients: As seen from the plots of ∇(log2t)Lvs. x in Figure2, threshold gradients are positive for x within clipping thresholds (xn,xp) and negative otherwise [...] the update rule log2t:=log2t−α∇(log2t)L where α is the learning rate" [p. 5] "Quantization thresholds and weights are simultaneously trained on a global loss")
wherein changing the range of the clipping function comprises updating the first threshold value using the gradient of the first threshold value of the clipping function and/or updating the second threshold value using the gradient of the second threshold value of the clipping function([p. 1 §1] "uniform quantization between two clipping thresholds" [p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons" [p. 3] "the exact clipping thresholds of x in the real domain are xn=s·(n−0.5) and xp=s·(p+0.5). [...] Role of threshold gradients: As seen from the plots of ∇(log2t)Lvs. x in Figure2, threshold gradients are positive for x within clipping thresholds (xn,xp) and negative otherwise [...] the update rule log2t:=log2t−α∇(log2t)L where α is the learning rate" [p. 5] "Quantization thresholds and weights are simultaneously trained on a global loss").
Xu as well as Jain are directed towards neural network quantization using adaptive clipping thresholds. Therefore, Xu as well as Jain are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Xu with the teachings of Jain by learning the threshold with the weight parameters through gradient updates. Jain provides as additional motivation for combination ([p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 2, the combination of Xu and Jain teaches The method of claim 1, wherein the range of the clipping function is adaptively changed based on a magnitude of the weight set.(Xu [p. 5 §4.3] "Accordingly, the information entropy of W after applying ReCU can be computed by [See Eqn. 17] which is a function of τ by substituting ˆb in Eq. (11) for b. [...] the mean of the absolute values of the weights after standardization is [See Eqn. 20]" Eqn. 20 explicitly computes a magnitude of the weight set, where the information entropy of weights in Eqn. 17 varies with b and tau which are adaptively changed based on said magnitude.).
Regarding claim 4, the combination of Xu and Jain teaches The method of claim 1, wherein changing the range of the clipping function comprises changing at least one of a first threshold value (Xu [p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tao is the parameter that controls the clipping range.)
and a second threshold value of the clipping function.(Xu [p. 4 4.2] "Obviously, α is a function of τ after replacing b with the estimation ˆb in Eq" alpha interpreted as second threshold value which is explicitly varied with the first threshold value tau.).
Regarding claim 5, the combination of Xu and Jain teaches The method of claim 1, wherein generating the binarized weight set comprises: obtaining a clipped weight set by applying the clipping function to a plurality of weight elements included in the weight set; and(Xu [p. 6 §4.3] "we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively")
obtaining the binarized weight set by applying a scaled sign function to a plurality of clipped weight elements included in the clipped weight set.(Xu [p. 6 §4.3] "we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2)").
Regarding claim 9, the combination of Xu and Jain teaches The method of claim 1, further comprising: iteratively performing the operations of generating the binarized weight set, generating the output data, generating the gradient of the weight set, updating the weight set, and changing the range of the clipping function.(Xu [p. 7 §5.2.1] "Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows [See Eqn. 24] where I denotes the total number of training epochs" epoch interpreted as synonymous with iteration).
Regarding claim 10, the combination of Xu and Jain teaches The method of claim 9, further comprising: storing a result of the training operation after iteratively performing the operations of generating the binarized weight set, generating the output data, generating the gradient of the weight set, updating the weight set, and changing the range of the clipping function a predetermined number of iterations.(Xu See Table 6 where results of the training operation after iteratively performing the operations of generating the binarized weight set, generating the output data, generating the gradient of the weight set, updating the weight set, and changing the range of the clipping function a predetermined number of iterations are stored/logged.).
Regarding claim 12, the combination of Xu and Jain teaches The method of claim 10, wherein storing the result of the training operation comprises: storing the binarized weight set.(Xu [pp. 15-16] "Quantized model link" provides links to stored binarized weight sets produced by the method.).
Regarding claim 13, the combination of Xu and Jain teaches The method of claim 1, wherein the binarized neural network is a binarized convolutional neural network (BCNN).(Xu [p. 6 §5.1] "Network Structures. On CIFAR-10, we evaluate ReCU with ResNet-18/20 [21] and VGG-Small [51]. Following the compared methods, we binarize all convolutional and fully-connected layers except the first and the last ones").
Regarding claim 14, the combination of Xu and Jain teaches The method of claim 13, wherein the binarized convolutional neural network includes a plurality of convolutional layers, and wherein the operations of generating the binarized weight set, generating the output data, generating the gradient of the weight set, updating the weight set, and changing the range of the clipping function are performed on at least one remaining convolutional layer other than a first convolutional layer among the plurality of convolutional layers.(Xu [p. 6 §5.1] "Network Structures. On CIFAR-10, we evaluate ReCU with ResNet-18/20 [21] and VGG-Small [51]. Following the compared methods, we binarize all convolutional and fully-connected layers except the first and the last ones").
Regarding claim 15, the combination of Xu and Jain teaches The method of claim 13, wherein the binarized convolutional neural network includes a plurality of layers, wherein the plurality of layers include at least one fully connected layer, and wherein the operations of generating the binarized weight set, generating the output data, generating the gradient of the weight set, updating the weight set, and changing the range of the clipping function are performed on at least one remaining layer other than the fully connected layer among the plurality of layers.(Xu [p. 6 §5.1] "Network Structures. On CIFAR-10, we evaluate ReCU with ResNet-18/20 [21] and VGG-Small [51]. Following the compared methods, we binarize all convolutional and fully-connected layers except the first and the last ones").
Claims 6, 7, and 20 are rejected under U.S.C. §103 as being unpatentable over the combination of Xu and Jain as evidenced by He (“Deep Residual Learning for Image Recognition”, 2016).
Regarding claim 6, the combination of Xu and Jain teaches The method of claim 1, wherein generating the output data comprises: performing a convolution operation on the input data using the binarized weight set;(Xu [p. 6 §5.1] "we binarize all convolutional and fully-connected layers" See also performance comparison tables 5 and 6 which are output accuracies compared to expected results when using the binarization method.).
While both Xu and Jain explicitly use a ResNet model, the combination of Xu and Jain doesn't explicitly teach performing a pooling operation on a result of the convolution operation;
performing a batch normalization on a result of the pooling operation; and
obtaining the output data by applying an activation function to a result of the batch normalization.
He, in the same field of endeavor, teaches performing a pooling operation on a result of the convolution operation;([p. 776 §4.2] "The network ends with a global average pooling" See also ResNet-34 architecture in FIG. 3 which shows pooling applied to the first and last convolutions)
performing a batch normalization on a result of the pooling operation; and ([p. 4 §3.4] "We adopt batch normalization (BN) [16] right after each convolution and before activation")
obtaining the output data by applying an activation function to a result of the batch normalization.(See also ResNet-34 architecture in FIG. 3 which shows pooling applied to the first and last convolutions).
He is merely introduced as a teaching reference to explicitly reinforce the model structure used by Xu. It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the ResNet model was first described by He and the ResNet model explicitly used by Xu and Jain has a known structure.
Regarding claim 7, the combination of Xu and Jain teaches The method of claim 1, wherein generating the gradient of the weight set comprises: reversely applying an activation function to loss input data calculated from the output data;(Xu [p. 6 §5.1] "we binarize all convolutional and fully-connected layers" [p. 6 §4.3] "update W using the stochastic gradient descent (SGD)" See also performance comparison tables 5 and 6 which are output accuracies compared to expected results when using the binarization method. Backpropagation with stochastic gradient descent (SGD) applies the loss calculated from the input data to the network in reverse by definition.)
obtaining a gradient of a clipped weight set by reversely applying a scaled sign function to the gradient of the binarized weight set; and(Xu [p. 2] "Since the quantization function in the forward propagation of BNNs has zero gradient almost everywhere, an approximate gradient function is required to enable the network to update. A typical example is the straight through estimator (STE) [...] The experimental results show that ReCU achieves state-of-the-art performance, as well as faster training convergence even with the simple STE [4] as our weight gradient approximation" Xu explicitly uses STE to reversely apply a gradient to the sign function.)
obtaining the gradient of the weight set, the gradient of the first threshold value of the clipping function, (Xu [p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tau is the parameter that controls the clipping range, tau in the first epoch is interpreted as the first threshold value, the gradient computed this epoch is interpreted as the gradient of the first threshold (tau). Tau is changed from first value to second value to third value.)
and the gradient of the second threshold value of the clipping function by reversely applying the clipping function to the gradient of the clipped weight set.(Xu [p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tau is the parameter that controls the clipping range, tau in the second epoch is interpreted as the second threshold value, the gradient computed this epoch is interpreted as the gradient of the second threshold (tau).).
However, the combination of Xu and Jain doesn't explicitly teach reversely performing a batch normalization on a result of reversely applying the activation function;
reversely performing a pooling operation on a result of reversely performing the batch normalization;
obtaining loss output data and a gradient of the binarized weight set by reversely performing a convolution operation on a result of reversely performing the pooling operation;.
He, in the same field of endeavor, teaches reversely performing a batch normalization on a result of reversely applying the activation function;([p. 776 §4.2] "The network ends with a global average pooling" See also ResNet-34 architecture in FIG. 3 which shows pooling applied to the first and last convolutions)
reversely performing a pooling operation on a result of reversely performing the batch normalization;([p. 4 §3.4] "We adopt batch normalization (BN) [16] right after each convolution and before activation")
obtaining loss output data and a gradient of the binarized weight set by reversely performing a convolution operation on a result of reversely performing the pooling operation;(See also ResNet-34 architecture in FIG. 3 which shows pooling applied to the first and last convolutions).
He is merely introduced as a teaching reference to explicitly reinforce the model structure used by Xu. It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the ResNet model was first described by He and the ResNet model explicitly used by Xu and Jain has a known structure.
Regarding claim 20, Xu teaches A method of training a binarized neural network (BNN), the method comprising: obtaining a clipped weight set by applying a clipping function to a weight set,([Abstract] "In this paper, for the first time, we explore the influence of “dead weights” which refer to a group of weights that are barely updated during the training of BNNs, and then introduce rectified clamp unit (ReCU) to revive the “dead weights” for updating" clamp unit interpreted as clipping function.)
wherein the clipping function includes a first threshold value as a lower limit of a range of the clipping function and a second threshold value as an upper limit of the range of the clipping function;([p. 4] "Q(τ) and Q(1−τ) respectively denote the τ quantile and 1−τ quantile [55] of W […] ReCU relocates w to Q(1−τ) if it is smaller than Q(1−τ), and to Q(τ) if it is larger than Q(τ)" Xu's ReCU function is a clamp: values below Q(1-τ) (first/lower threshold) are moved up to Q(1-τ) and values above Q(τ) (second/upper threshold) are moved down to Q(τ))
obtaining a binarized weight set by applying a scaled sign function to the clipped weight set;([p. 6 §4.3] "we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2)")
generating output data by sequentially performing a forward computation on the binarized neural network based on input data and the binarized weight set;([p. 6 §4.3] "in the forward propagation, we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively. Then, we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2). Finally, we complete the binary convolution using Eq. (3) for the forward propagation")
updating the weight set based on the gradient of the weight set; and([p. 6 §4.3] "During backpropagation, we derive the gradients w.r.t. W and A using Eq. (5) and Eq. (6), respectively, and update W using the stochastic gradient descent (SGD) described in Sec. 5.1")
wherein: the binarized neural network is a binarized convolutional neural network (BCNN),([p. 6 §5.1] "Network Structures. On CIFAR-10, we evaluate ReCU with ResNet-18/20 [21] and VGG-Small [51]. Following the compared methods, we binarize all convolutional and fully-connected layers except the first and the last ones")
the scaled sign function, and the clipping function,([p. 2] "Since the quantization function in the forward propagation of BNNs has zero gradient almost everywhere, an approximate gradient function is required to enable the network to update. A typical example is the straight through estimator (STE) [...] The experimental results show that ReCU achieves state-of-the-art performance, as well as faster training convergence even with the simple STE [4] as our weight gradient approximation" Xu explicitly uses STE to reversely apply a gradient to the sign function.)
a range of the clipping function is adaptively changed based on a magnitude of the weight set, and([p. 5 §4.3] "Accordingly, the information entropy of W after applying ReCU can be computed by [See Eqn. 17] which is a function of τ by substituting ˆb in Eq. (11) for b. [...] the mean of the absolute values of the weights after standardization is [See Eqn. 20]" Eqn. 20 explicitly computes a magnitude of the weight set, where the information entropy of weights in Eqn. 17 varies with b and tau which are adaptively changed based on said magnitude.)
the range of the clipping function is changed using the gradient of the first threshold value of the clipping function ([p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tau is the parameter that controls the clipping range, tau in the first epoch is interpreted as the first threshold value, the gradient computed this epoch is interpreted as the gradient of the first threshold (tau). Tau is changed from first value to second value to third value.)
and the gradient of the second threshold value of the clipping function.([p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tau is the parameter that controls the clipping range, tau in the second epoch is interpreted as the second threshold value, the gradient computed this epoch is interpreted as the gradient of the second threshold (tau).).
However, Xu does not explicitly teach generating a gradient of the weight set, a gradient of a first threshold value of the clipping function, and a gradient of a second threshold value of the clipping function by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data;
changing the range of the clipping function, wherein changing the range of the clipping function comprises updating the first threshold value of the clipping function using the gradient of the first threshold value of the clipping function; and/or updating the second threshold value of the clipping function using the gradient of the second threshold value of the clipping function,
the forward computation is performed in an order of a convolution operation, a pooling operation, a batch normalization, and an activation function,
the backward computation is performed in an order of the activation function, the batch normalization, the pooling operation, the convolution operation,.
Jain, in the same field of endeavor, teaches generating a gradient of the weight set, a gradient of a first threshold value of the clipping function, and a gradient of a second threshold value of the clipping function by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data;([p. 1 §1] "uniform quantization between two clipping thresholds" [p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons" [p. 3] "the exact clipping thresholds of x in the real domain are xn=s·(n−0.5) and xp=s·(p+0.5). [...] Role of threshold gradients: As seen from the plots of ∇(log2t)Lvs. x in Figure2, threshold gradients are positive for x within clipping thresholds (xn,xp) and negative otherwise [...] the update rule log2t:=log2t−α∇(log2t)L where α is the learning rate" [p. 5] "Quantization thresholds and weights are simultaneously trained on a global loss")
changing the range of the clipping function, wherein changing the range of the clipping function comprises updating the first threshold value of the clipping function using the gradient of the first threshold value of the clipping function; and/or updating the second threshold value of the clipping function using the gradient of the second threshold value of the clipping function,([p. 1 §1] "uniform quantization between two clipping thresholds" [p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons" [p. 3] "the exact clipping thresholds of x in the real domain are xn=s·(n−0.5) and xp=s·(p+0.5). [...] Role of threshold gradients: As seen from the plots of ∇(log2t)Lvs. x in Figure2, threshold gradients are positive for x within clipping thresholds (xn,xp) and negative otherwise [...] the update rule log2t:=log2t−α∇(log2t)L where α is the learning rate" [p. 5] "Quantization thresholds and weights are simultaneously trained on a global loss").
Xu as well as Jain are directed towards neural network quantization using adaptive clipping thresholds. Therefore, Xu as well as Jain are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Xu with the teachings of Jain by learning the threshold with the weight parameters through gradient updates. Jain provides as additional motivation for combination ([p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons"). This motivation for combination also applies to the remaining claims which depend on this combination.
However, the combination of Xu and Jain does not explicitly teach the forward computation is performed in an order of a convolution operation, a pooling operation, a batch normalization, and an activation function,
the backward computation is performed in an order of the activation function, the batch normalization, the pooling operation, the convolution operation.
He, in the same field of endeavor, teaches the forward computation is performed in an order of a convolution operation, a pooling operation, a batch normalization, and an activation function,([p. 4 3.4] "We adopt batch normalization (BN) [16] right after each convolution and before activation" See also FIG. 3)
the backward computation is performed in an order of the activation function, the batch normalization, the pooling operation, the convolution operation,([p. 4 3.4] "We adopt batch normalization (BN) [16] right after each convolution and before activation" See also FIG. 3).
He is merely introduced as a teaching reference to explicitly reinforce the model structure used by Xu. It would be obvious to one of ordinary skill in the art before the effective filing date of the claimed invention that the ResNet model was first described by He and the ResNet model explicitly used by Xu and Jain has a known structure.
Claim 8 is rejected under U.S.C. §103 as being unpatentable over the combination of Xu and Jain as evidenced by Stanford (“CS231n Convolutional Neural Networks for Visual Recognition”, 2021).
Regarding claim 8, the combination of Xu and Jain teaches The method of claim 1.
However, the combination of Xu and Jain doesn't explicitly teach updating the weight set comprises: updating the weight set by subtracting the gradient from the weight set.
Stanford, in the same field of endeavor, teaches updating the weight set comprises: updating the weight set by subtracting the gradient from the weight set; ([p. 9] "W_new = W - step_size * df [...] Update in negative gradient direction. In the code above, notice that to compute W_new we are making an update in the negative direction of the gradient df since we wish our loss function to decrease, not increase." stochastic gradient descent by definition updates the weights by subtracting the gradient from the weight set.).
Stanford is introduced merely to reinforce the obviousness of the claim and to show that the training in Xu which is explicitly backpropagation with stochastic gradient descent by definition requires updating the weight set by subtracting the gradient from the weight set.
Claim 11 is rejected under U.S.C. §103 as being unpatentable over the combination of Xu and Jain and Guo (US20200082264A1).
Regarding claim 11, the combination of Xu and Jain teaches The method of claim 10, wherein storing the result of the training operation comprises: storing the weight set, (Xu [pp. 15-16] "Quantized model link" provides links to stored binarized weight sets produced by the method.).
However, the combination of Xu and Jain doesn't explicitly teach the first threshold value of the clipping function, and the second threshold value of the clipping function. .
Guo, in the same field of endeavor, teaches the first threshold value of the clipping function, and the second threshold value of the clipping function. ([¶0212] "a first goal is to find a binary expansion of W that approximates it well (as illustrated in FIG. 16, which means W ≈ 〈 B , a 〉 = ∑ j = 0 m - 1 a j B j , in which and B∈{+1,−1}c×w×h×m and a∈Rm are the concatenations of m binary tensors{B0, . . . , Bm-1} and the same number of scale factors {a0, . . . , am-1}, respectively. The appropriate choice of B and a with a fixed m can be investigated. In particular, FIG. 16 shows approximating the real-valued weight tensor with a sum of binary scaled tensors" [¶0388] "apparatus comprises a memory to store input initial, intermediate, and final results" Guo explicitly teaches that the weight tensors comprise the binary tensors as well as scale factors (interpreted as threshold values of the clipping function), which are explicitly stored in memory.).
The combination of Xu and Jain as well as Guo are directed towards range based neural network quantization. Therefore, the combination of Xu and Jain as well as Guo are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Xu and Jain with the teachings of Guo by applying the binarization it on a memory system and storing the weights (variables) in memory. While it would be obvious to one of ordinary skill in the art to store variables in memory on a computer, this is reinforced by Guo who provides as additional motivation for combination ([¶0204] "The disclosed examples and embodiments introduce network sketching as a new way of pursuing binary-weight CNNs [...] the disclosed examples and embodiments propose two theoretical grounded algorithms, making it possible to regulate the precision of sketching for more accurate inference. Moreover, to further improve the efficiency of generated models (a.k.a., sketches), the disclosed examples and embodiments propose an algorithm to associatively implement binary tensor convolutions, with which the required number of floating-point additions and subtractions (FADDs) is likewise reduced.").
Claims 16, 17, 18, and 19 are rejected under U.S.C. §103 as being unpatentable over the combination of Xu, Guo, and Jain.
Regarding claim 16, Xu teaches generating a binarized weight set by applying a clipping function to a weight set;([Abstract] "In this paper, for the first time, we explore the influence of “dead weights” which refer to a group of weights that are barely updated during the training of BNNs, and then introduce rectified clamp unit (ReCU) to revive the “dead weights” for updating" clamp unit interpreted as clipping function.)
wherein the clipping function includes a first threshold value as a lower limit of a range of the clipping function and a second threshold value as an upper limit of the range of the clipping function;([p. 4] "Q(τ) and Q(1−τ) respectively denote the τ quantile and 1−τ quantile [55] of W […] ReCU relocates w to Q(1−τ) if it is smaller than Q(1−τ), and to Q(τ) if it is larger than Q(τ)" Xu's ReCU function is a clamp: values below Q(1-τ) (first/lower threshold) are moved up to Q(1-τ) and values above Q(τ) (second/upper threshold) are moved down to Q(τ))
generating output data by sequentially performing a forward computation on a binarized neural network based on input data and the binarized weight set;([p. 6 §4.3] "in the forward propagation, we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively. Then, we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2). Finally, we complete the binary convolution using Eq. (3) for the forward propagation")
training the binarized neural network, comprising:([p. 2 §1] "In this paper, we present a novel perspective to improve the effectiveness and training efficiency of BNN" [p. 6 §5.1] "Training Details. Our network is trained from scratch without depending on a pre-trained model. For all experiments, we use SGD for optimization with a momentum of 0.9 and the weight-decay is set to 5e-4")
updating the weight set based on the gradient of the weight set; and([p. 6 §4.3] "During backpropagation, we derive the gradients w.r.t. W and A using Eq. (5) and Eq. (6), respectively, and update W using the stochastic gradient descent (SGD) described in Sec. 5.1")
changing the range of the clipping function.([p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tao is the parameter that controls the clipping range.)
wherein changing the range of the clipping function comprises updating the first threshold value using the gradient of the first threshold value of the clipping function and/or updating the second threshold value using the gradient of the second threshold value of the clipping function ([p. 4 4.1] "in Sec. 5, we introduce an adaptive exponential scheduler to identify the range of the “dead weights” in order to seek a balance between the quantization error and the information entropy" See Eqn. 8. [p. 7 §5.2.1] "we further propose an exponential scheduler for adapting τ along the network training. Our motivation lies in that τ should start with a value falling within [0.85, 0.94] to pursue a good accuracy, and then gradually go to the interval [0.96, 1.00] to stabilize the variance of performance. Based on this, given an initial τs and an end threshold τe, τi at the i-th training epoch is calculated as follows" See Eqn. 24. Tau is the parameter that controls the clipping range, tau in the first epoch is interpreted as the first threshold value, the gradient computed this epoch is interpreted as the gradient of the first threshold (tau). Tau is changed from first value to second value to third value.).
However, Xu does not explicitly teach A memory device comprising: processing logic; and
a memory core including a plurality of memory cells and comprising data embodied in the memory cells that is executable by the processing logic to perform operations comprising:
generating a gradient of the weight set, a gradient of the first threshold value, and a gradient of the second threshold value by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data; and.
Guo, in the same field of endeavor, teaches A memory device comprising: processing logic; and ([¶0062] "a local instance of the parallel processor memory 222 may be excluded in favor of a unified memory design that utilizes system memory in conjunction with local cache memory.")
a memory core including a plurality of memory cells and comprising data embodied in the memory cells that is executable by the processing logic to perform operations comprising: ([¶0062] "the memory units 224A-224N can include various types of memory devices, including dynamic random access memory (DRAM) or graphics random access memory, such as synchronous graphics random access memory (SGRAM), including graphics double data rate (GDDR) memory. In one embodiment, the memory units 224A-224N may also include 3D stacked memory, including but not limited to high bandwidth memory (HBM). Persons skilled in the art will appreciate that the specific implementation of the memory units 224A-224N can vary, and can be selected from one of various conventional designs").
Xu as well as Guo are directed towards range based neural network quantization. Therefore, Xu as well as Guo are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of Xu with the teachings of Guo by applying the binarization it on a memory system and storing the weights (variables) in memory. While it would be obvious to one of ordinary skill in the art to store variables in memory on a computer, this is reinforced by Guo who provides as additional motivation for combination ([¶0204] "The disclosed examples and embodiments introduce network sketching as a new way of pursuing binary-weight CNNs [...] the disclosed examples and embodiments propose two theoretical grounded algorithms, making it possible to regulate the precision of sketching for more accurate inference. Moreover, to further improve the efficiency of generated models (a.k.a., sketches), the disclosed examples and embodiments propose an algorithm to associatively implement binary tensor convolutions, with which the required number of floating-point additions and subtractions (FADDs) is likewise reduced.").
However, the combination of Xu, and Guo does not explicitly teach generating a gradient of the weight set, a gradient of the first threshold value, and a gradient of the second threshold value by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data.
Jain, in the same field of endeavor, teaches generating a gradient of the weight set, a gradient of the first threshold value, and a gradient of the second threshold value by sequentially performing a backward computation on the binarized neural network based on loss calculated from the output data; and ([p. 1 §1] "uniform quantization between two clipping thresholds" [p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons" [p. 3] "the exact clipping thresholds of x in the real domain are xn=s·(n−0.5) and xp=s·(p+0.5). [...] Role of threshold gradients: As seen from the plots of ∇(log2t)Lvs. x in Figure2, threshold gradients are positive for x within clipping thresholds (xn,xp) and negative otherwise [...] the update rule log2t:=log2t−α∇(log2t)L where α is the learning rate" [p. 5] "Quantization thresholds and weights are simultaneously trained on a global loss").
The combination of Xu and Guo as well as Jain are directed towards neural network quantization using adaptive clipping thresholds. Therefore, the combination of Xu and Guo as well as Jain are analogous art in the same field of endeavor. It would have been obvious before the effective filing date of the claimed invention to combine the teachings of the combination of Xu and Guo with the teachings of Jain by learning the threshold with the weight parameters through gradient updates. Jain provides as additional motivation for combination ([p. 3] "the choice to train thresholds in the log-domain is simple yet effective for various stability reasons"). This motivation for combination also applies to the remaining claims which depend on this combination.
Regarding claim 17, the combination of Xu, Guo, and Jain teaches The memory device of claim 16, wherein the operations further comprise: storing the weight set (Xu [pp. 15-16] "Quantized model link" provides links to stored binarized weight sets produced by the method.)
the first threshold value of the clipping function, and the second threshold value of the clipping function as a result of training the binarized neural network; (Guo [¶0212] "a first goal is to find a binary expansion of W that approximates it well (as illustrated in FIG. 16, which means W ≈ 〈 B , a 〉 = ∑ j = 0 m - 1 a j B j , in which and B∈{+1,−1}c×w×h×m and a∈Rm are the concatenations of m binary tensors{B0, . . . , Bm-1} and the same number of scale factors {a0, . . . , am-1}, respectively. The appropriate choice of B and a with a fixed m can be investigated. In particular, FIG. 16 shows approximating the real-valued weight tensor with a sum of binary scaled tensors" [¶0388] "apparatus comprises a memory to store input initial, intermediate, and final results" Guo explicitly teaches that the weight tensors comprise the binary tensors as well as scale factors (interpreted as threshold values of the clipping function), which are explicitly stored in memory.)
generating the binarized weight set based on the stored weight set, (Xu [p. 6 §5.1] "Network Structures. On CIFAR-10, we evaluate ReCU with ResNet-18/20 [21] and VGG-Small [51]. Following the compared methods, we binarize all convolutional and fully-connected layers except the first and the last ones" [pp. 15-16] "Quantized model link" provides links to stored binarized weight sets produced by the method.)
the stored first threshold value of the clipping function, and the stored second threshold value of the clipping function; and (Guo [¶0212] "As described above, a first goal is to find a binary expansion of W that approximates it well (as illustrated in FIG. 16, which means W ≈ 〈 B , a 〉 = ∑ j = 0 m - 1 a j B j , in which and B∈{+1,−1}c×w×h×m and a∈Rm are the concatenations of m binary tensors{B0, . . . , Bm-1} and the same number of scale factors {a0, . . . , am-1}, respectively. The appropriate choice of B and a with a fixed m can be investigated. In particular, FIG. 16 shows approximating the real-valued weight tensor with a sum of binary scaled tensors.")
operating the binarized neural network in inference mode using the generated binarized weight set. (Xu [p. 6 §4.3] "in the forward propagation, we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively. Then, we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2). Finally, we complete the binary convolution using Eq. (3) for the forward propagation").
Regarding claim 18, the combination of Xu, Guo, and Jain teaches The memory device of claim 16, wherein the operations further comprise: storing the binarized weight set as a result of training the binarized neural network; and(Guo [¶0377] " approximating trained filters of the trained CNN by determining a basis of binary tensors and a series of scale factors." [¶0212] "a first goal is to find a binary expansion of W that approximates it well (as illustrated in FIG. 16, which means W ≈ 〈 B , a 〉 = ∑ j = 0 m - 1 a j B j , in which and B∈{+1,−1}c×w×h×m and a∈Rm are the concatenations of m binary tensors{B0, . . . , Bm-1} and the same number of scale factors {a0, . . . , am-1}, respectively. The appropriate choice of B and a with a fixed m can be investigated. In particular, FIG. 16 shows approximating the real-valued weight tensor with a sum of binary scaled tensors" [¶0388] "apparatus comprises a memory to store input initial, intermediate, and final results")
operating the binarized neural network in inference mode using the stored binarized weight set. (Xu [p. 6 §4.3] "in the forward propagation, we first standardize W and revive the “dead weights” using Eq. (22) and ReCU of Eq. (8), respectively. Then, we compute the scaling factor α using Eq. (13), and binarize the inputs and the revived weights using the sign function of Eq. (2). Finally, we complete the binary convolution using Eq. (3) for the forward propagation").
Regarding claim 19, the combination of Xu, Guo, and Jain teaches The memory device of claim 16, wherein the binarized neural network is trained using the memory device or a binarized neural network training device located outside the memory device. (Guo [¶0018] "FIG. 11 illustrates exemplary embodiment of training and deployment of a deep neural network." [¶0168] "The training framework 604 can hook into an untrained neural network 1106 and enable the untrained neural net to be trained using the parallel processing resources described herein to generate a trained neural net 1108.").
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Gholami (“A Survey of Quantization Methods for Efficient Neural Network Inference”, 2022) describes all uniform quantization as having symmetric or asymmetric clipping range with an upper and lower threshold, with exemplary methods including LQ-Nets (Zhang, “LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks”, 2018).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SIDNEY VINCENT BOSTWICK whose telephone number is (571)272-4720. The examiner can normally be reached M-F 7:30am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang can be reached on (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SIDNEY VINCENT BOSTWICK/Examiner, Art Unit 2124
Prosecution Insights