DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1, 2, 3, 4, 21, 26, 27-31, 33, and 36-41 have been amended by Applicant. Claims 6-20, 22, 25, 32, 35, 42, and 45-48 are cancelled and new claims 49-51 have been added. Claims 1-5, 21, 23-24, 26-31, 33-34, 36-41, 43-44, and 49-51 are currently pending.
Response to Arguments
Claim Rejections under 35 U.S.C. 101
In view of the broadening amendments to independent claims 1, 26, and 36, a new grounds of rejection under 35 U.S.C. 101 has been made as to claims 1-5, 21, 23-24, 16-31, 33-34, 36-41, 43-44, and new claims 49-50.
Claim Rejections under 35 U.S.C. 103
The rejection of claims 1, 3-4, 21, 23, 26, 28-29, 31, 33, 36, 38-39, 41, and 43 (as amended) under 35 U.S.C. 103 has been maintained.
The rejection of claims 5, 30, and 40 under 35 U.S.C. 103 has been maintained.
The rejection of claims 24, 34, and 44 under 35 U.SC. 103 has been maintained.
Applicant’s arguments (filed 01/16/2026) have been fully considered but have been found unpersuasive.
Applicant argues (in pg. 12 of Applicant’s remarks) that the combination of Wu in view of Yao does not teach layer-wise scaling. In support, Applicant’s contend that Wu is limited to a global scaling factor applied to the entire network’s loss and that Wu does not teach a scale factor for a set of gradient values in each of the plurality of layers.
Examiner respectfully disagrees with Applicant’s argument above. As set forth in the Non-final Office Action dated 09/15/2025 and the instant office action, Wu was cited as teaching the limitation determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers. To this effect, (Wu, Paragraph [0119] teaches The system obtains the scale factor S from the hyperparameter (block 622); Wu, Paragraph [0129] teaches a number of schemes can be used for picking the starting S as well as for ways to reduce/increase S as required during training. As an example, if the automatic back-off technique is used in combination with automatically selecting the scale factor S for the next iteration based on the results from the previous iteration, there may be enough assurance that the automatic selection will be successful that the amount of back-off of the S value can be relatively small.; Wu, Paragraph [0113] teaches before back propagation to compute gradients (402(x)), the loss value resulting from forward propagation is scaled by S.sub.x and the gradient computations are performed at lower precision. After the gradient computations are completed, they are compensated for the scaling and the compensated values are used for additional training procedures (which may for example comprise weight adjustments at higher precision) (400(x)). Similar processes can be repeated for training iteration Q.sub.Y—potentially with a different scale factor S.sub.y. (402(y), 400(y)).; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; Wu, Abstract, teaches one technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision; Wu [0089] teaches Example Methods for Selecting the Scaling Factor S [0090] Constant selected by user as another hyperparameter (see FIGS. 9, 11) [0091] Automatically-select the factor for each iteration (see FIG. 10) [0092] Find the weight gradient with the largest magnitude, x [0093] Compute upper bound on the factor: μ=log.sub.2(2.sup.15−x)).
Examiner believes that Wu at [claim 29], in particular teaches layer-wise scaling in view of the rest of the cited paragraphs of Wu.
As stated in the Non-Final Office Action, Examiner believes that Wu teaches, or at least suggest the limitation determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers (see mapping of the limitation above). However Yao, more clearly and concurrently teaches the limitation as provided below.
Yao teaches:
determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers (Yao, Paragraphs 0064, 0077, 0090, 0101, “[0077] On the other hand, the updated weights are quantized at a feed-forward stage. From this perspective, custom-character is the final variable to be optimized. After the weight sets are updated at iteration t, the respective low-bit ternary or binary equivalents can be obtained when the corresponding optimal scaling factor set {α.sub.l} can be determined. [0064] By representing DNN models with very low-bit parameter values such as {−1, 0, 1} and {−1, 1} multiplied with layer-wise scaling factors, certain examples provide great benefits to applications of DNN solutions, especially on specialized deep learning hardware where originally time intensive multiplication operations can be replaced by simple bit-shift and accumulation operations, for example.”, i.e., scale factors that are based off the weights, which may be based off the loss, are calculated/determined for each layer of the neural network [the term “layer-wise” teaching for each layer], which are used to update the layers’ weights, or parameters, of the neural network with the loss calculated. This is done via the neural network processor 710, as mentioned above; Yao, Paragraph [0104] teaches FIG. 10 illustrates a flow diagram representing example machine readable instructions to train a neural network (e.g., block 906 of the example method of FIG. 9). At block 1002, each layer of the input neural network is processed and trained. For example, to quantize the weights of the full-precision DNN according to ELQ, the example network training manager 730 traverses each layer of the DNN and processes the weights associated with that layer.; Yao, Paragraph [0124] teaches Example 3 includes example 2, wherein the network initializer, the network weight partitioner, the loss calculator, the weight quantizer, the loss calculator, and the weight updater are to process each layer of the first deep neural network model to generate the low-bit second network weights for each layer to enable the model deployer to deploy the low-bit deep neural network model including a plurality of layers; Yao, Paragraph [0087] As shown in Table 1, the ELQ algorithm can be used to train a low-precision DNN model using training data and a corresponding full-precision DNN model. For each network layer l, weights W.sub.a and W.sub.b and binary matrix T1 are initialized (e.g., W.sub.a←Ø, W.sub.b←W.sub.l, and T.sub.l←1), and a scaling factor is calculated based on W.sub.l as defined in Equation (10) above. Interval bound factors are set at successive partition steps or intervals (e.g., σ.sub.1=a, σ.sub.2=b, σ.sub.N=0). Then, a quantization loop executes to determine network weights for the low-bit DNN model.).
Because the combination of Wu in view of Yao teaches the argued limitation, Applicant’s argument has been found unpersuasive.
Applicant further argues (in pgs. 13-14 of Applicant’s remarks) that Yao fails to teach rescaling for gradient precision.
Examiner respectfully disagrees with Applicant’s argument above. As set forth in the instant office action, the combination of Wu in view of Yao was shown to teach the limitation rescaling, by one or more processors, the scaled gradient values based on the scaled factor. To this effect Wu, Paragraph [0113] was shown to teach after the gradient computations are completed, they are compensated for the scaling and the compensated values are used for additional training procedures (which may for example comprise weight adjustments at higher precision) (400(x)). Similar processes can be repeated for training iteration Q.sub.Y—potentially with a different scale factor S.sub.y. (402(y), 400(y)).; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; See also Fig. 6A and Fig. 7.
Examiner has understood the repetition of the training iteration with a different scale factor to read on the argue limitation as claimed. Examiner has further understood that Wu at claim 29 further teaches “a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient”. Further reading on the argued limitation as claimed.
Applicant further argues (in pg. 14 of Applicant’s remarks) that there is no motivation to combine Wu with Yao.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case,
it would have been obvious to one of ordinary skill in the art to combine the teachings of training a deep neural network using mixed precision and loss scale factors of Wu with the teachings of a deep neural network that is trained using layer-wise values and factors of Yao to improve the speed of the training/learning operations and to suppress accuracy loss (Yao, Paragraphs 0018-0019)
Examiner notes that Wu further teaches in paragraph [0045] because of the potentially vast number of activation and weight gradient computations that may be needed to train DNN 200, it may be attractive or efficient to use reduced precision to perform those computations. For example, in some platforms such as mixed precision hardware accelerators or distributed computing architectures featuring arithmetic computation units having different computation precisions, it may be possible to perform some multiple number of half-precision (e.g., FP16) computations as compared with full-precision (e.g., FP32) computations in a given time period. As an example, certain NVIDIA Pascal based architectures can perform two FP16 computations for every FP32 computation using the same width data paths and increased efficiency of memory bandwidth utilization. Such speed and efficiency performance gains can be an attractive solution when it is desirable to reduce the amount of training time, reduce memory bandwidth requirements, increase the size of the DNN or for other reasons. In other computing platforms such as cloud-based architectures having both higher precision (but more expensive) computation hardware or capabilities and lower precision (but less expensive) computation hardware or capabilities, it may be desirable to use the cheaper or more efficient lower precision computation hardware/capabilities for parts of the learning process that require a plethora of computations, and save the more expensive higher precision computation hardware/capabilities for other parts of the learning process that require more precise computations or comparisons. There are also instances in which it is desirable to run legacy software and processes written for a higher precision computation platform on a mixed precision platform to take advantage of the efficiency and speed increases that may be gained by performing some computations using reduced precision.
In view of the above, Applicant’s arguments have been found unpersuasive and the rejection of claims 1, 3-4, 21, 23, 26, 28-29, 31, 33, 36, 38-39, 41, 43 (as amended) under 35 U.S.C. 103 over the combination of Wu in view of Yao has been maintained.
For at least the same reasons provided for independent claim 1 (as amended) the rejection of dependent claims 5, 30, and 40 and 24, 34, and 44 under 35 U.SC. 103 has been maintained.
New claims 49-51 have also been rejected under 35 U.S.C. 103 over the combination of Wu in view of Yao. (See detailed mapping of the rejection of said claims in the section provided below).
101
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-5, 21, 23-24, 16-31, 33-34, 36-41, 43-44, and 49-50 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (abstract idea) without significantly more.
Regarding claim 1,
Step 1: Claim 1 is directed towards a method.
Step 2A, Prong 1: Claim 1 recites the following limitations:
determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers; [i.e., the highlighted limitations recite mere mathematical calculations in view of Applicant’s specification at paragraphs 0048-0054, 0057, 0064, 0067]
calculating, by the one or more processors, using the scale factor of the set of gradient values, scaled gradients values for the set of gradient values; [i.e., the highlighted limitations recite mere mathematical calculations in view of Applicant’s specification at paragraphs 0039-0047, 0064-0066, 0102]
rescaling, by the one or more processors, the scaled gradient values based on the scale factor; [i.e., the highlighted limitations recite mere mathematical calculations in view of Applicant’s specification at paragraphs 0048-0050, 0103]
updating, by the one or more processors, parameters using the rescaled gradient values. [i.e., the highlighted limitations recited mere mathematical calculations in view of Applicant’s specification at paragraphs 0051-0053]
Hence the claim recites an abstract idea.
Step 2A, Prong 2: The additional element of “a method of training a neural network including a plurality of layers” consists of mere instructions to apply the judicial exception using generic computer components”. (see MPEP 2106.05(f)). The additional element of “by one or more processors” is recited at a high-level of generality such that it amounts to no more than a generic computer component used as a tool to perform an abstract idea. (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “a method of training a neural network including a plurality of layers” consists of mere instructions to apply the judicial exception using generic computer components”. (see MPEP 2106.05(f)). Furthermore, the additional element of “by one or more processors” is recited at a high-level of generality such that it amounts to no more than a generic computer component used as a tool to perform an abstract idea. (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 2,
Step 2A, Prong 1: Claim 2 recites an abstract idea as inherited from claim 1 above.
Step 2A, Prong 2: The additional element of “wherein the one or more processors support an 8-bit floating point format” merely generally links the use of the judicial exception to a particular technological environment or field of use. (See MPEP 2106.05(h)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 2 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the one or more processors support an 8-bit floating point format” merely generally links the use of the judicial exception to a particular technological environment or field of use. (See MPEP 2106.05(h)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 3,
Step 2A, Prong 1: Claim 3 recites an abstract idea as inherited from claim 1. Claim 3 further recites the following limitations:
wherein the scale factor is dynamically updated during training. [i.e., the highlighted limitation is a mere mathematical calculation in view of Applicant’s specification at paragraphs 0098-0100]
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: The additional element of “during training” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 3 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “during training” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 4,
Step 2A, Prong 1: Claim 4 recites an abstract idea as inherited from claim 1. Claim 4 further recites the following limitations:
wherein the determining comprises determining the scale factor based on statistics [i.e., the highlighted limitation merely recites mathematical calculations in view of Applicant’s specification at paragraphs 0067-0080]
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 4 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 5,
Step 2A, Prong 1: Claim 5 recites an abstract idea as inherited from claims 1 and 4. Claim 5 further recites the following limitations:
wherein the determining comprises determining the scale factor to be larger than a lower bound, and the lower bound is determined based on the statistics, a predetermined value and a Gaussian error function value of a hyperparameter. [i.e., the highlighted limitation consists of mere mathematical calculations in view of Applicant’s specification at paragraphs 0067-0080]
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 5 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 21,
Step 2A, Prong 1: Claim 21 recites an abstract idea as inherited form claims 1 and 4.
Step 2A, Prong 2: The additional elements of “wherein the statistics include the parameters of the set of gradient values” merely generally links the use of the judicial exception to a particular technological environment or field of use”. (see MPEP 2106.05(h)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 21 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the statistics include the parameters of the set of gradient values” merely generally links the use of the judicial exception to a particular technological environment or field of use”. (see MPEP 2106.05(h)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 23,
Step 2A, Prong 1: Claim 23 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: The additional element of “wherein at least one scale factor among the scale factors that are determined by the determining is larger than 1” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 23 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein at least one scale factor among the scale factors that are determined by the determining is larger than 1” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 24,
Step 2A, Prong 1: Claim 24 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: The additional element of “wherein inputs of the neural network are related to pixel intensities” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 24 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein inputs of the neural network are related to pixel intensities” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 26,
Step 1: Claim 26 is directed towards an apparatus.
Step 2A, Prong 1: Claim 26 recites the same or similar limitations as claim 1 and, as such, is rejected under the same rationale as claim 1.
Step 2A, Prong 2: Claim 26 recites the additional element of “a training apparatus”, “one or more memories”, and “one or more processors configured to”. These additional elements to perform the limitations in the claim consists of mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 26 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “a training apparatus”, “one or more memories”, and “one or more processors configured to”. These additional elements to perform the limitations in the claim consists of mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 27,
Claim 27 recites the same or similar limitations as claim 2 and, as such, is rejected under the same rationale as claim 2.
Regarding claim 28,
Claim 28 recites the same or similar limitations as claim 3 and, as such, is rejected under the same rationale as claim 3.
Regarding claim 29,
Claim 29 recites the same or similar limitations as claim 4 and, as such, is rejected under the same rationale as claim 4.
Regarding claim 30,
Claim 30 recites the same or similar limitations as claim 5 and, as such, is rejected under the same rationale as claim 5.
Regarding claim 31,
Claim 31 recites the same or similar limitations as claim 21 and, as such, is rejected under the same rationale as claim 21.
Regarding claim 33,
Claim 33 recites the same or similar limitations as claim 23 and, as such, is rejected under the same rationale as claim 23.
Regarding claim 34,
Claim 34 recites the same or similar limitations as claim 24 and, as such, is rejected under the same rationale as claim 24.
Regarding claim 36,
Step 1: Claim 36 is directed towards a non-transitory computer-readable storage medium for storing a program.
Step 2A, Prong 1: Claim 36 recites the same or similar limitations as claim 1 and, as such, is rejected under the same rationale as claim 1.
Step 2A, Prong 2: Claim 36 recites the additional element of “a non-transitory computer-readable storage medium for storing a program, that when executed by one or more processors of one or more computers, cause the one or more computers to”. This additional element to carry out the limitations in the claim is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 36 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “a non-transitory computer-readable storage medium for storing a program, that when executed by one or more processors of one or more computers, cause the one or more computers to”. This additional element to carry out the limitations in the claim is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 37,
Claim 37 recites the same or similar limitations as claims 2 and 27 and, as such, is rejected under the same rational as claims 2 and 27.
Regarding claim 38,
Claim 38 recites the same or similar limitations as claims 3 and 28 and, as such, is rejected under the same rationale as claims 3 and 28.
Regarding claim 39,
Claim 39 recites the same or similar limitations as claims 4 and 29 and, as such, is rejected under the same rationale as claims 4 and 29.
Regarding claim 40,
Claim 40 recites the same or similar limitations as claim 5, as such, is rejected under the same rationale as claim 5.
Regarding claim 43,
Claim 43 recites the same or similar limitations as claims 23 and 33 and, as such, is rejected under the same rationale as claims 23 and 33.
Regarding claim 44,
Claim 44 recites the same or similar limitations as claims 24 and 34 and, as such, is rejected under the same rationale as claims 24 and 34.
Regarding claim 49,
Step 2A, Prong 1: Claim 49 recites an abstract idea as inherited from claim 1. Claim 49 further recites the following limitation:
wherein the scale factor is determined based on a value related to a maximum value of the set of gradient values. [i.e., the highlighted limitation consists of mere mathematical calculations in view of Applicant’s specification at paragraphs 0066, 0078, 0079]
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 49 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 50,
Step 2A, Prong 1: Claim 50 recites an abstract idea as inherited from claim 1. Claim 50 further recites the following limitation:
wherein the scale factor is determined based on a value related to a maximum value of weight values. [i.e., the highlighted limitation consists of mere mathematical calculations in view of Applicant’s specification at paragraphs 0078, 0106]
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 50 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Hence the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 51,
Step 2A, Prong 1: Claim 51 recites an abstract idea as inherited from claim 1.
Step 2A, Prong 2: The additional element of “wherein the scale factor is a power of 2” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 51 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the scale factor is a power of 2” merely generally links the use of the judicial exception to a particular technological environment or field of use. (see MPEP 2106.05(h)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3-4, 21, 23, 26, 28-29, 31, 33, 36, 38-39, 41, 43, and 49-51 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (US 20180322391 A1, filed May 4, 2018 and published Nov. 8, 2018) in view of Yao et al. (US 20210019630 A1, filed Jul. 26, 2018 and published Jan. 21, 2021).
Regarding claim 1 (as amended), Wu teaches a method of training a neural network including a plurality of layers (Wu, Abstract, teaches training a deep neural network and Paragraph [0037] teaches an example DNN comprising layers; Wu, Paragraph [0041] further teaches training the DNN…), comprising:
determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers (Wu, Paragraph [0119 teaches The system obtains the scale factor S from the hyperparameter (block 622); Wu, Paragraph [0129] teaches a number of schemes can be used for picking the starting S as well as for ways to reduce/increase S as required during training. As an example, if the automatic back-off technique is used in combination with automatically selecting the scale factor S for the next iteration based on the results from the previous iteration, there may be enough assurance that the automatic selection will be successful that the amount of back-off of the S value can be relatively small.; Wu, Paragraph [0113] teaches before back propagation to compute gradients (402(x)), the loss value resulting from forward propagation is scaled by S.sub.x and the gradient computations are performed at lower precision. After the gradient computations are completed, they are compensated for the scaling and the compensated values are used for additional training procedures (which may for example comprise weight adjustments at higher precision) (400(x)). Similar processes can be repeated for training iteration Q.sub.Y—potentially with a different scale factor S.sub.y. (402(y), 400(y)).; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; Wu, Abstract, teaches One technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision; Wu [0089] teaches Example Methods for Selecting the Scaling Factor S [0090] Constant selected by user as another hyperparameter (see FIGS. 9, 11) [0091] Automatically-select the factor for each iteration (see FIG. 10) [0092] Find the weight gradient with the largest magnitude, x [0093] Compute upper bound on the factor: μ=log.sub.2(2.sup.15−x))
calculating, by the one or more processors, using the scale factor of the set of gradient values, scaled gradients values for the set of gradient values (Wu, Paragraph [0038] teaches each layer of the deep neural network has its own weights…the goal of training the DNN is to learn weights for all layers of the DNN; Wu, Paragraph [0041] further teaches to train DNN 200, training data is applied to the DNN as input. In a forward pass, the DNN 200 propagates the input through all the layers 100 to compute the output and the loss. Then, in a backward pass, the network propagates the loss backwards, computing the gradients (derivatives) for all weights w.; Wu, Paragraph [0069] further teaches scaling applied to compute partial derivatives (gradients).; Wu, Paragraph [0119] teaches “the system obtains the scale factor S from the hyperparameter (block 622). Or as shown in FIG. 11, the system can read a default S which a user can optionally modify with a hyperparameter (650, 652, 654). The system then applies the training data to the network 200 (624) and performs a forward propagation through the network to compute a loss value(s) (blocks 626, 656). The system scales the loss value(s) by S (blocks 628, 658) and uses the scaled loss value in a back propagation to compute gradients using reduced precision computations (blocks 630, 660). If the results are okay (decision block 631), the computed gradient additions to network weights are compensated (block 632) and (assuming no further problems as tested by decision block 634) the network weights are adjusted and stored (block 636)”; [Note: Examiner is interpreting the adjustment of the weights as an adjustment of the layers]; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; Wu, Abstract, teaches One technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision, and reduces the magnitude of the computed gradients to compensate for scaling of the loss. In one example non-limiting arrangement, the training forward pass scales a loss value by some factor S and the weight update reduces the weight gradient contribution by 1/S.; Wu, Paragraph [0071] teaches “even though the gradient values have been scaled upward for purposes of gradient computation using reduced precision computation hardware.”);
rescaling, by one or more processors, the scaled gradient values based on the scaled factor (Wu, Paragraph [0113] teaches after the gradient computations are completed, they are compensated for the scaling and the compensated values are used for additional training procedures (which may for example comprise weight adjustments at higher precision) (400(x)). Similar processes can be repeated for training iteration Q.sub.Y—potentially with a different scale factor S.sub.y. (402(y), 400(y)).; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; See also Fig. 6A and Fig. 7); and
updating, by the one or more processors, parameters using the rescaled gradient values (Wu, Paragraph [0127] teaches “the algorithm inspects the values of the weight gradients after the iteration is performed to determine whether such unacceptable results occurred. If such values are found to have occurred, the example non-limiting system infers that the scaling factor was too high, discards the results of the particular iteration, does not perform the weight update and repeats the iteration [i.e. updates] with a lower value of S. This process can occur iteratively with a succession of back-off values of S until no problem is observed, in which case the results of the successful iteration are used to update the weights. This automated trial-and-error approach can be used to relatively rapidly determine an optimally or at least reasonably large value for S for a particular iteration.”; [Note: Examiner is interpreting the updating of the weights, as described in Wu [0127], as updating a parameter of the layers as claimed]; Wu [claim 29] teaches “A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith, the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision”; Wu, Abstract, teaches one technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision, and reduces the magnitude of the computed gradients to compensate for scaling of the loss. In one example non-limiting arrangement, the training forward pass scales a loss value by some factor S and the weight update reduces the weight gradient contribution by 1/S.; Wu, Paragraph [0071] teaches “even though the gradient values have been scaled upward for purposes of gradient computation using reduced precision computation hardware.”).
Examiner believes that Wu teaches, or at least suggest the limitation determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers (see mapping of the limitation above). However Yao, more clearly and concurrently teaches the limitation as provided below.
Yao teaches:
determining, by one or more processors, a scale factor for a set of gradient values in each of the plurality of layers (Yao, Paragraphs 0064, 0077, 0090, 0101, “[0077] On the other hand, the updated weights are quantized at a feed-forward stage. From this perspective, custom-character is the final variable to be optimized. After the weight sets are updated at iteration t, the respective low-bit ternary or binary equivalents can be obtained when the corresponding optimal scaling factor set {α.sub.l} can be determined. [0064] By representing DNN models with very low-bit parameter values such as {−1, 0, 1} and {−1, 1} multiplied with layer-wise scaling factors, certain examples provide great benefits to applications of DNN solutions, especially on specialized deep learning hardware where originally time intensive multiplication operations can be replaced by simple bit-shift and accumulation operations, for example.”, i.e., scale factors that are based off the weights, which may be based off the loss, are calculated/determined for each layer of the neural network [the term “layer-wise” teaching for each layer], which are used to update the layers’ weights, or parameters, of the neural network with the loss calculated. This is done via the neural network processor 710, as mentioned above; Yao, Paragraph [0104] teaches FIG. 10 illustrates a flow diagram representing example machine readable instructions to train a neural network (e.g., block 906 of the example method of FIG. 9). At block 1002, each layer of the input neural network is processed and trained. For example, to quantize the weights of the full-precision DNN according to ELQ, the example network training manager 730 traverses each layer of the DNN and processes the weights associated with that layer.; Yao, Paragraph [0124] teaches Example 3 includes example 2, wherein the network initializer, the network weight partitioner, the loss calculator, the weight quantizer, the loss calculator, and the weight updater are to process each layer of the first deep neural network model to generate the low-bit second network weights for each layer to enable the model deployer to deploy the low-bit deep neural network model including a plurality of layers; Yao, Paragraph [0087] As shown in Table 1, the ELQ algorithm can be used to train a low-precision DNN model using training data and a corresponding full-precision DNN model. For each network layer l, weights W.sub.a and W.sub.b and binary matrix T1 are initialized (e.g., W.sub.a←Ø, W.sub.b←W.sub.l, and T.sub.l←1), and a scaling factor is calculated based on W.sub.l as defined in Equation (10) above. Interval bound factors are set at successive partition steps or intervals (e.g., σ.sub.1=a, σ.sub.2=b, σ.sub.N=0). Then, a quantization loop executes to determine network weights for the low-bit DNN model.);
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to combine the teachings of training a deep neural network using mixed precision and loss scale factors of Wu with the teachings of a deep neural network that is trained using layer-wise values and factors of Yao to improve the speed of the training/learning operations and to suppress accuracy loss (Yao, Paragraphs 0018-0019).
Regarding Claim 3 (as amended), the combination of Wu in view of Yao teaches all of the limitations of claim 1, and Wu further teaches wherein the scale factor is dynamically updated during training (Wu, Paragraph [0023], teaches training procedures using automatic selection of a scaling factor S.; Wu, Paragraph [0120] teaches the system can automatically select the scaling S factor for each iteration.; Wu, Paragraph [0143] further teaches “a larger modification to the code can be used to implement dynamic scaling procedure shown in FIG. 12. After starting with some scaling factor S (702), the S value is evaluated in each training iteration by examining the gradient values (704, 706). If an overflow or NaN is detected (“yes” exit to decision block 706) then weight update (710) is skipped and S is reduced (708). Conversely, if no overflow or NaN is detected for some number of iterations (“no” exit to decision block 706), S is increased (714). An alternative approach is to adjust the scaling factor S by examining the gradient value statistics and computing their distribution—S is selected to ensure that probability of overflowing in any iteration is under a chosen threshold.”).
Regarding claim 4 (as amended), the combination of Wu in view of Yao teaches all of the limitations of claim 1, and Wu further teaches wherein the determining comprises determining the scale factor based on statistics (Wu, Paragraph [0143] teaches “adjust the scaling factor S by examining the gradient value statistics”).
Regarding claim 21 (as amended), the combination of Wu in view of Yao teaches all of the limitations of claim 1, and Wu teaches wherein the statistics include statistics of the parameters and the set of gradient values (Wu, Paragraph [0023], teaches training procedures using automatic selection of a scaling factor S.; Wu, Paragraph [0120] teaches the system can automatically select the scaling S factor for each iteration.; Wu, Paragraph [0143] further teaches “a larger modification to the code can be used to implement dynamic scaling procedure shown in FIG. 12. After starting with some scaling factor S (702), the S value is evaluated in each training iteration by examining the gradient values (704, 706). If an overflow or NaN is detected (“yes” exit to decision block 706) then weight update (710) is skipped and S is reduced (708). Conversely, if no overflow or NaN is detected for some number of iterations (“no” exit to decision block 706), S is increased (714). An alternative approach is to adjust the scaling factor S by examining the gradient value statistics and computing their distribution—S is selected to ensure that probability of overflowing in any iteration is under a chosen threshold.” [i.e., statistics of gradients]; Wu, Fig. 5B teaches statistics of weight values – Wu, Paragraph [0122] teaches referring back to FIGS. 5A, 5B, from observations one can see that the weight gradients tend to have the larger values. In the particular example shown, the largest activation gradient value is 1/64—that is, 0×1p−6, i.e., 2.sup.−6—whereas the largest weight gradient is 1/8—that is, 0×1p−3, i.e. 2.sup.−3. Because weight gradients tend to have larger values than activation gradients, weight gradients typically cannot be shifted as far to the right as activation gradients. On the other hand, one can see from these diagrams that there is a substantial amount of range on the upper part of the graphs into which gradient values can be shifted and still avoid overflow.; Wu, Paragraph [0054] FIGS. 5A-5B show example non-limiting histograms of Resnet50 gradients. These histograms are a graphical representation of the distribution of numerical data , and provide an estimate of the probability distribution of a continuous or quantitative variable—in this case activation and weight gradients calculated during DNN training. [i.e., statistics of weight values – wherein the weights are understood as a type of parameter]; Wu [0103] teaches further teaches adjust other parameters (or in some cases adjust both weight gradients and other parameters) to compensate for scaled gradient computations (see FIGS. 13B-13D) [0106] Parameters can include: [0107] Learning rate, gradient clipping threshold, weight decay parameter, etc. [0108] The potential benefit of adjusting parameters other than the computed weight gradients is that often fewer values are being modified)
Regarding claim 23, the combination of Wu in view of Yao teaches all of the limitations of claim 1, and Wu further teaches wherein at least one scale factor among the scale factors that are determined by the determining is larger than 1 (Wu, Paragraph [0132] teaches if S=1000, then the weight gradients will be one thousand times larger than they would have otherwise have been and using such weight gradients to update the weights will result in the weight update that is one thousand times larger than it should be. It is useful to address this in order to prevent the neural training from failing or becoming inaccurate. [Note: S is the scaling factor and is larger than 1).
Regarding claim 26 (as amended),
Claim 26 (as amended) recites the same or similar limitations as claim 1 (as amended) and, as such, is rejected under the same rationale and motivation as claim 1.
Yao further teaches a training apparatus, comprising:
one or more memories that store a neural network including a plurality of layers (Yao, Paragraphs 0090, 0097, 0101, Fig. 7 “[0090] For example, a full-precision DNN model is retrieved and/or otherwise acquired along with a training data set to be stored in the data storage 720… The DNN model can be stored in the data storage 720, for example.”, i.e., the data storage 720, or memory, is able to store a deep neural network, which would have a plurality of layers);
and one or more processors (Yao, Paragraphs 0090, 0112, Figs. 7 and 11, “[0090] In operation, the example neural network processor 710 acquires and/or otherwise retrieves a neural network, such as the example network(s) 100, 300, 500-530 of FIGS. 1-6, and initializes the acquired neural network. [0112] The processor platform 1100 of the illustrated example includes a processor 1112. The processor 1112 of the illustrated example is hardware.”)
Regarding claim 28 (as amended),
Claim 28 recites the same or similar limitations as claim 3 and, as such, is rejected under the same rationale as claim 3.
Regarding claim 29 (as amended),
Claim 29 recites the same or similar limitations as claim 4 and, as such, is rejected under the same rationale as claim 4.
Regarding claim 31 (as amended),
Claim 31 recites the same or similar limitations as claim 21 and, as such, is rejected under the same rationale as claim 21.
Regarding claim 33 (as amended),
Claim 33 recites the same or similar limitations as claim 23 and, as such, is rejected under the same rationale as claim 23.
Regarding claim 36 (as amended),
Claim 36 (as amended) recites the same or similar limitations as claim 1 (as amended), as such, is rejected under the same rationale and motivation as claim 1.
Yao further teaches a non-transitory computer-readable storage medium for storing a program (Yao, Paragraph 0098, Fig. 11 “In this example, the machine-readable instructions include a program for execution by a processor such as a processor 1112 shown in the example processor platform 1100 discussed below in connection with FIG. 11. The program can be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor 1112…”, i.e., a storage medium is capable of storing a program related to the training of a neural network.)
Regarding claim 38 (as amended),
Claim 38 recites the same or similar limitations as claims 3 and 28 and, as such, is rejected under the same rationale as claims 3 and 28.
Regarding claim 39 (as amended),
Claim 39 recites the same or similar limitations as claims 4 and 29 and, as such, is rejected under the same rationale as claims 4 and 29.
Regarding claim 41 (as amended),
Claim 41 recites the same or similar limitations as claims 21 and 31 and, as such, is rejected under the same rationale as claims 21 and 31.
Regarding claim 43,
Claim 43 recites the same or similar limitations as claims 23 and 33 and, as such, is rejected under the same rationale as claims 23 and 33.
Regarding claim 49,
The method as claimed in claim 1, wherein the scale factor is determined based on a value related to a maximum value of the set of gradient values (Wu [0089] teaches Example Methods for Selecting the Scaling Factor S [0090] Constant selected by user as another hyperparameter (see FIGS. 9, 11); [0091] further teaches Automatically-select the factor for each iteration (see FIG. 10) [0092] Find the weight gradient with the largest magnitude, x; [0093] Compute upper bound on the factor: μ=log.sub.2(2.sup.15−x)).
Regarding claim 50,
The method as claimed in claim 1, wherein the scale factor is determined based on a value related to a maximum value of weight values (Wu [0089] teaches Example Methods for Selecting the Scaling Factor S [0090] Constant selected by user as another hyperparameter (see FIGS. 9, 11); [0091] further teaches Automatically-select the factor for each iteration (see FIG. 10) [0092] Find the weight gradient with the largest magnitude, x;).
Regarding claim 51,
The method as claimed in claim 1, wherein the scale factor is a power of 2 (Wu [0093] teaches Compute upper bound on the factor: μ=log.sub.2(2.sup.15−x); Wu [0121] further teaches In one example, this upper bound is: μ=log.sub.2(2.sup.15−x) where x is the magnitude of the largest gradient value seen in the previous iteration. In this non-limiting implementation, for each back propagation the system performs, it examines all of the weight gradients at that point and from that determines the scaling factor S for the next iteration as S=μ−k where k is a constant (block 644). With such examination, it is straightforward to determine the largest computed gradient values. An [Note: the upper bound of the scaling factor being a factor based on Log2 being understood as the scale is a power of 2].
Claims 2, 27, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Yao, as applied in claim 1, and further in view of Lee et al. (US 20190340504 A1, filed Jan. 10, 2019 and published Nov. 7, 2019)
Regarding claim 2 (as amended), the combination of Wu in view of Yao teaches all of the limitations of claim 1, however the combination does not distinctly disclose wherein the one or more processors support an 8-bit floating point format.
Nevertheless, Lee teaches:
wherein the one or more processors support an 8-bit floating point format (In an embodiment, unlike such typical neural networks that use 32-bit or 64-bit floating-point or fixed-point data, the processor 110 may use 8-bit or 16-bit floating-point or fixed-point data as the parameters according to the low precision.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the training a deep neural network using mixed precision and loss scale factors of Wu with the processor to support 8-bit floating point operations, as taught by Lee, in order to improve processing operations (e.g. helping to improve speed and helping to minimize any corresponding accuracy loss) of a processor, reduce memory requirements, improve memory access speeds, and/or improve the speed of classification determinations. Further, with one or more embodiments, more complex and sophisticated trained neural networks may be performed on processing systems that have lesser capabilities, such as in mobile examples, where such trained neural networks may not have been available for implementation or may not have been able to be performed with sufficient speed to operate in real-time during operation of such mobile apparatuses, as non-limiting examples. (Lee, Paragraph [0048]).
Regarding claim 27 (as amended),
Claim 27 recites the same or similar limitations as claim 2 and, as such, is rejected under the same rationale as claim 2.
Regarding claim 37 (as amended),
Claim 37 recites the same or similar limitations as claims 2 and 27 and, as such, is rejected under the same rational as claims 2 and 27.
Claim 5, 30, and 40 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Yao, as applied in claims 1, 26, and 36, and further in view of Harrer et al. (US 20180107451 A1, filed Oct. 14, 2016 and published Apr. 19, 2018) and Tripathy et al. “Deep UQ: Learning Deep Neural Network Surrogate Models for High Dimensional Uncertainty Quantification.” (published Feb. 2, 2018)
Regarding claim 5, the combination of Wu in view of Yao teaches all of the limitations of claim 4, however the combination does not distinctly disclose wherein the determining comprises determining the scale factor to be larger than a lower bound, and the lower bound is determined based on the statistics, a predetermined value and a Gaussian error function value of a hyperparameter.
Nevertheless, Harrer teaches wherein the determining comprises determining the scale factors to be larger than a lower bound (Harrer, Paragraphs 0049 and 0051, “[0051] In block 325, the scaling factor, Sf.sub.n.sub._.sub.new, is saturated within [lower bound, upper bound]. That is, the scaling factor will be set to the lower bound if the scaling factor is less than the lower bound, or will be set to the upper bound if the scaling factor is greater than the upper bound. The bounds are typically predefined parameters, such as in an exemplary embodiment the bound can be specified for each layer.”), and the lower bound is determined based on the statistics, a predetermined value and a Gaussian error function value of a hyperparameter.
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the teachings of training a deep neural network using mixed precision and loss scale factors of Wu with the teachings of training a deep neural network that uses layer-wise scaling factors that are bounded, as taught by Harrer, for stable accuracy performance and improve fixed point performance (Harrer, Paragraphs [0021-0023]).
Furthermore, Tripathy teaches, wherein the determining comprises determining the scale factors to be larger than a lower bound, and the lower bound is determined based on the statistics, a predetermined value and a Gaussian error function value of a hyperparameter (Tripathy, Pages 16, 20-22, Equations 22-24, Algorithm 1, “[Pg. 16] While typically the L1 and L2 parts in the elastic net are assigned different scaling factors, we share the scaling parameter λ (called the regularization constant). [Pgs. 21-22] Eq. (23) is a stochastic global optimization problem characterized by a noisy objective function. BGO sequentially seeks out the global optimum of the objective function, R, by iteratively updating a Gaussian process (GP) surrogate response surface for R(λ; S). During each iteration of BGO, a new pair of input-output observations is generated by maximizing an information acquisition function (IAF). The most popular choice of IAF is the expected improvement (EI) function…where, φ and Φ are the probability density function and the cumulative distribution function of the standard normal distribution. Z = µ(λ)−R(λ∗;S) σ(λ) where, µ(λ) is the predictive mean of the GP surrogate at λ, and σ(λ)2 =σGP(λ) 2−σnoise(λ) 2 , where σGP(λ)2 is the predictive variance of the GP surrogate which captures the epistemic uncertainty induced due to the limited set of observations and σnoise(λ)2 is GP estimate of the observational noise induced by D34random initializations of the DNN weights and random splitting of the dataset into Dtrain, Dtest and Dval. σ(λ)2 is thus a filtered version of the predictive variance which is robust to observational noise.”; [Note: the scaling factor, or regularization function, is used with the gradient calculation when updating the parameters of a DNN, or deep neural network. When updating/selecting the parameter, a lower bound is used as a way to update the factor as well as a minimum function. This lower bound is based on statistics of gradients, a gaussian process, and a predetermined value, which would be a structure parameter, S. The gaussian process, or GP, is able to use hyperparameters L and h, which defines the structure of the network.]).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the teachings of training a deep neural network using mixed precision and loss scale factors of Wu with the teachings of using scaled Gradient descent with the training and learning of a deep neural network of Tripathy for reduced complexity and effectively predict solutions (Tripathy, Pg. 16, Paragraph 1, Pgs. 34-35, Section 4).
Regarding claim 30 (as amended),
Claim 30 recites the same or similar limitations as claim 5 and, as such, is rejected under the same rationale as claim 5.
Regarding claim 40,
Claim 40 recites the same or similar limitations as claim 5, as such, is rejected under the same rationale as claim 5.
Claim 24, 34, and 44 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Yao, as applied in claims 1, 26, and 36, and further in view of Yu et al. (US 20210100468 A1, file Oct. 8, 2019 and published Apr. 8, 2021)
Regarding claim 24, the combination of Wu in view of Yao teaches all of the limitations of claim 1, however the combination does not distinctly disclose wherein inputs of the neural network are related to pixel intensities.
Nevertheless, Yu teaches wherein inputs of the neural network are related to pixel intensities (Yu, Paragraph [0090] teaches In some embodiments, each neuron of the input layer of the convolutional neural network may be configured to receive a single value from the ECG data, wherein the single value may correspond to a pixel intensity value of an image of an ECG waveform.).
Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to modify the teachings of training a deep neural network using mixed precision and loss scale factors of Wu with the pixel intensities, as taught by Yu, as the input layer of a convolutional neural network may comprise an equal number of input nodes/neurons as there are pixels in the one or more 2D images, thereby enabling a 1-to-1 input of pixel intensity values into input nodes/neurons of an input layer of a convolutional neural network. (Yu, Paragraph [0049]).
Regarding claim 34,
Claim 34 recites the same or similar limitations as claim 24 and, as such, is rejected under the same rationale as claim 24.
Regarding claim 44,
Claim 44 recites the same or similar limitations as claims 24 and 34 and, as such, is rejected under the same rationale as claim 24.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEATRIZ RAMIREZ BRAVO whose telephone number is 571-272-2156. The examiner can normally be reached Mon. - Fri. 7:30a.m.-5:00p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, USMAAN SAEED can be reached at 571-272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.R.B./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights