Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
In the previous Office Action issued December 16, 2025 (hereinafter “the previous Office Action”), claims 1-17 were pending.
This action is in response to the amendment and remarks filed April 15, 2026. In the amendment, claims 1, 14, and 17 were amended, no claims were canceled, and no claims were added. Thus, claims 1-17 are pending.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on March 12, 2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 101
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-13 are directed to a method [process].
Claims 14-16 are directed to a computer system [machine].
Claim 17 is directed to a non-transitory computer-readable storage medium [machine].
Regarding Claim 1:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
(a) processing the training data in respective forward and backward passes - In the context of the claim limitation, this encompasses a mathematical concept of computing forward and backward passes.
computing gradients of a pre-determined loss function with respect to the network weights; and/or computing gradients of the pre-determined loss function with respect to the computed activations of the network - In the context of the claim limitation, this encompasses a mathematical concept of calculating a gradient of a loss function.
(b) wherein an adjustment parameter is applied to at least a subset of values…the values comprising at least one of: the network weights, the activations computed in the forward pass, the gradients with respect to activations computed in the backward pass, and the gradients with respect to weights computed in the backward pass - In the context of the claim limitation, this encompasses a mental processing of evaluating to parameter to at least a subset of values.
updating the network weights in dependence on the computed gradients with respect to the weights - In the context of the claim limitation, this encompasses a mental processing of evaluating weight based on calculated gradients.
computing a proportion of the subset of values falling above a predefined threshold - In the context of the claim limitation, this encompasses a mathematical concept of calculating values.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion - In the context of the claim limitation, this encompasses a mental processing of evaluating parameter in dependence on the computed proportion.
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
A computer-implemented method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
(a) through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising:
(b) …in the neural network…
wherein the values are in an eight-bit or a sixteen-bit floating-point format
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
A computer-implemented method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
(a) through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising:
(b) …in the neural network…
wherein the values are in an eight-bit or a sixteen-bit floating-point format
Regarding Claim 2:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 1. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
wherein the adjustment parameter is a scale factor, and wherein the scale factor is applied on the backward pass to at least a subset of the gradients with respect to at least one of the activations and the gradients with respect to the network weights, wherein the scale factor is updated in dependence on the proportion of the gradients of that subset that have a value falling above a pre-defined threshold - In the context of the claim limitation, this encompasses a mental processing of evaluating parameter in dependence on the computed proportion.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 3:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 2. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
applying the scale factor to at least one of gradients with respect to weights and gradients with respect to activations of all layers of the network by multiplying the loss function by the scale factor - In the context of the claim limitation, this encompasses a mathematical concept of multiplying the loss function by a scale factor.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 4:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 2. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
constructing a histogram of gradients, the histogram comprising a plurality of bins, wherein the scale factor is updated based on a proportion of gradients occupying bins above a threshold value - In the context of the claim limitation, this encompasses a mathematical concept of constructing a histogram of a gradient.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 5:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 4. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
constructing a respective histogram of gradients…wherein the proportion of gradients occupying each of a set of bins for each histogram is input to an accumulator to obtain an aggregated proportion for each bin, the scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold - In the context of the claim limitation, this encompasses a mathematical concept of construction a histogram of gradient.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
…for each layer of the neural network…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
…for each layer of the neural network…
Regarding Claim 6:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 4. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
constructing a respective histogram of gradients for each layer, wherein for each layer a respective layer-wise scale factor is applied during the backward pass, the layer-wise scale factor being updated based on a proportion of gradients in the histogram for the corresponding layer occupying bins above a corresponding layer-wise threshold value - In the context of the claim limitation, this encompasses a mathematical concept of constructing a respective histogram gradient.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
Regarding Claim 7:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 4. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
(a) …processes a respective subset of the training data in each of the forward and backward passes, and computes a respective histogram of gradients for the corresponding subset of the training data, each histogram having defined a common set of bins, wherein the proportion of gradients occupying each bin of the set of bins defined for each histogram is aggregated to obtain an aggregated proportion for each bin, with a scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold - In the context of the claim limitation, this encompasses a mathematical concept of computing a histogram of gradients for the training data.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
(a) when implemented on a plurality of processors, wherein each processor…
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
(a) when implemented on a plurality of processors, wherein each processor…
Regarding Claim 8:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 1.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
storing at least a subset of the network weights, gradients and activations in computer memory in floating-point format
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
storing at least a subset of the network weights, gradients and activations in computer memory in floating-point format
Regarding Claim 9:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 8.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
storing at least a subset of the network weights, gradients and activations in computer memory in eight-bit floating-point format
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
storing at least a subset of the network weights, gradients and activations in computer memory in eight-bit floating-point format
Regarding Claim 10:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 8.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
storing at least a subset of the network weights, gradients and activations in computer memory in sixteen-bit floating-point format
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
storing at least a subset of the network weights, gradients and activations in computer memory in sixteen-bit floating-point format
Regarding Claim 11:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 8.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
storing the subset of values in a floating-point format, and wherein the adjustment parameter is an exponent bias applied to the floating-point representations of the subset of weights, gradients and activation
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
storing the subset of values in a floating-point format, and wherein the adjustment parameter is an exponent bias applied to the floating-point representations of the subset of weights, gradients and activation
Regarding Claim 12:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
wherein the subset of values in the neural network is a subset of network weights and activations and the adjustment parameter is an exponent bias applied to the subset of values of the network weights and activations in the forward pass
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
wherein the subset of values in the neural network is a subset of network weights and activations and the adjustment parameter is an exponent bias applied to the subset of values of the network weights and activations in the forward pass
Regarding Claim 13:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 11.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
wherein a subset of network weights, activations and gradients which are inputs to compute operations in at least one of the forward and backward passes are stored in eight-bit floating-point format, the compute operations comprising at least one of a matrix operation and a convolution operation
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
wherein a subset of network weights, activations and gradients which are inputs to compute operations in at least one of the forward and backward passes are stored in eight-bit floating-point format, the compute operations comprising at least one of a matrix operation and a convolution operation
Regarding Claim 14:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network, wherein an adjustment parameter is applied to at least a subset of values in the neural network, wherein the values on the forward pass comprise at least one of the network weights or computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations or gradients with respect to weights - In the context of the claim limitation, this encompasses a mathematical concept of computing a activation based on weights and training data.
updating the network weights in dependence on the computed gradients with respect to the weights - In the context of the claim limitation, this encompasses a mental processing of evaluating weight based on calculated gradients.
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold - In the context of the claim limitation, this encompasses a mathematical concept of calculating values.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion - In the context of the claim limitation, this encompasses a mental processing of evaluating parameter in dependence on the computed proportion.
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
A computer system comprising one or more processors configured to train a multi- layer neural network comprising a set of network weights, and memory holding the network weights, the processor configured to train the neural network by:
wherein the values are in an eight-bit or sixteen-bit floating-point format
The following additional elements are directed to insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(g)].
receiving a set of training data
storing the values to memory
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
A computer system comprising one or more processors configured to train a multi- layer neural network comprising a set of network weights, and memory holding the network weights, the processor configured to train the neural network by:
wherein the values are in an eight-bit or sixteen-bit floating-point format
The following additional elements are directed to receiving or transmitting data over a network. The courts (as per Intellectual Ventures v. Symantec, 838 F.3d 1307, 1321; 120 USPQ2d 1353, 1362 (Fed. Cir. 2016)) have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
receiving a set of training data
The following additional elements are directed to storing an receiving data. The courts (as per Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93 have recognized receiving or transmitting data over a network as well-understood, routine, and conventional functions when they are claimed in a merely generic manner (e.g., at a high level of generality) or as insignificant extra-solution activity to the judicial exception [see MPEP 2106.05(d) II.].
storing the values to memory
Regarding Claim 15:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 14.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
comprising a plurality of processors, wherein each processor is configured to process a respective subset of the training data
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
comprising a plurality of processors, wherein each processor is configured to process a respective subset of the training data
Regarding Claim 16:
Step 2A, Prong 1: This claim recites the same abstract ideas as in claim 15. Additionally,
The following limitations are/remain directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind (including an observation, evaluation, judgement, or opinion).
(a) wherein the adjustment parameter is updated in dependence on an aggregated proportion of values…falling above a predefined threshold, the aggregated proportion computed by aggregating a computed proportion of the subset of values falling above the predefined threshold - In the context of the claim limitation, this encompasses a mental processing of adjustment parameter of values.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
(a) all processors… for each of the plurality of processors
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
(a) all processors… for each of the plurality of processors
Regarding Claim 17:
Step 2A, Prong 1: The following limitations are directed to the abstract idea of a mental process [see MPEP 2106.04(a)(2) III. C.]. In particular, the claim recites mental processes that are concepts performed in the human mind or with pen and paper (including an observation, evaluation, judgement, or opinion).
processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network, wherein an adjustment parameter is applied to at least a subset of values in the neural network, and wherein the values on the forward pass comprise at least one of the network weights or computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations or gradients with respect to weights - In the context of the claim limitation, this encompasses a mathematical concept of computing an activation based on weights and training data.
updating the network weights in dependence on the computed gradients with respect to the weights - In the context of the claim limitation, this encompasses a mental processing of evaluating weight based on calculated gradients.
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold - In the context of the claim limitation, this encompasses a mathematical concept of calculating values.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion - In the context of the claim limitation, this encompasses a mental processing of evaluating parameter in dependence on the computed proportion.
As drafted, under their broadest reasonable interpretation (BRI), in view of the specification, the above limitations cover concepts performed in the human mind (observation, evaluation, judgement, or opinion). Given a sufficiently small set of data, nothing in the claim prohibits this process from being performed mentally or with pen and paper.
Step 2A, Prong 2: There are no additional elements in this claim that integrate the judicial exception into a practical application.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to integrate the judicial exception into a practical application.
A non-transitory computer-readable storage medium storing computer program instructions which when executed perform a method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
wherein the values are in an eight-bit or a sixteen-bit floating point format
Step 2B: There are no additional elements in this claim that amount to significantly more than the judicial exception.
The following additional elements are adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea [see MPEP 2106.05(f)] and therefore fails to amount to significantly more than the judicial exception.
A non-transitory computer-readable storage medium storing computer program instructions which when executed perform a method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
wherein the values are in an eight-bit or a sixteen-bit floating point format
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-3, 8, 10-12, and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Ginsburg et al. (US 12299577), hereinafter Ginsburg, in view of Tomioka et al. (US 20180336458), hereinafter Tomioka.
Regarding Claim 1:
Ginsburg discloses:
A computer-implemented method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
Ginsburg, [15], “ FIG. 1 depicts an exemplary computer-implemented method for training an artificial neural network. Steps 101-109 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 1, training of an artificial neural network typically begins at step 101 with receiving a training set of data as input. At step 103, the data is fed (typically as one or more matrices of values) into a corresponding number of neurons. At step 105, the data at each neuron is manipulated according to pre-determined parameters (weights)”
[29], “forward propagation can be performed for any layer of a neural network (e.g., inner product layers) or convolutional neural network (e.g., convolutional layers+ inner product layers”
Ginsburg teaches a method of training based on set of training data and weight of neural network.
processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data, and the backward pass comprising:
Ginsburg, [29], “forward propagation can be performed for any layer of a neural network (e.g., inner product layers) or convolutional neural network (e.g., convolutional layers+ inner product layers). To avoid the issue of vanishing or exploding activations (due to underflow and overflow, respectively), the rescaling operations described above can be performed”
[41], “the forward and backward propagation are performed in a neural network, the gradients for the weights that are used to adjust the influence of the data output from each neuron are calculated and subsequently readjusted”
Ginsburg teaches training data processing using forward and backward pass which comprising process of activation function of weight and training data.
computing gradients of a pre-determined loss function with respect to the network weights; and/or computing gradients of the pre-determined loss function with respect to the computed activations of the network
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described”
[3], “The output is then compared to the target output using a loss function, and an error value is calculated for each of the elements in the output layer. During back prop phase the gradients of error function are computed and then propagated backwards through the layers to determine gradients corresponding to each neuron”
Ginsburg teaches computing gradients of a loss with respect to activations.
the values comprising at least one of: the network weights, the activations computed in the forward pass, the gradients with respect to activations computed in the backward pass, or the gradients with respect to weights computed in the backward pass
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 2, backward propagation begins at step 201, wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, W T)”
Ginsburg teaches network weights, update parameter of the neural network, and compute gradient respect to forward and backward pass.
wherein the values are in an eight-bit or a sixteen-bit floating-point format
Ginsburg, [54], “According to such embodiments, one copy of weights is stored in memory as float32, and a second as float16 for forward and backward propagation.”
[4], “these solutions typically use a 16 bit floating-point (float16) representation.”
Ginsburg teaches weights [values] in a float16 format [sixteen-bit floating-point format]. Para. 4 specifies float16 is the float-point format.
updating the network weights in dependence on the computed gradients with respect to the weights
Ginsburg, [36], “wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, WT)”
Ginsburg teaches updating the weights of the network based on gradient.
Ginsburg does not explicitly disclose:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
computing a proportion of the subset of values falling above a predefined threshold
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
However, in the same field, analogous art Tomioka teaches:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
Tomioka, [0026], “A neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model”
[0050] “The worker node computes 802 one or more gradients of a loss function with respect to (1) the message it received during the forward pass and (2) the parameters of the local subgraph… If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node”
[0022] “The training data instance is labeled and so the ground truth output of the neural network is known and the difference or error between the observed output and the ground truth output is found and provides information about a loss function which is passed back through the neural network layers in a backward propagation or backwards pass. A search is made to try find a minimum of the loss function which is a set of weights of the neural network that enable the output of the neural network to match the ground truth data”
Tomioka teaches a subset of values in the neural network.
computing a proportion of the subset of values falling above a predefined threshold
Tomioka, [0050], “The worker node checks the number of gradients in the accumulator. If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node. It then clears 810 the accumulator. As mentioned above the threshold at operation 804 is either set globally for the pipeline as a whole or is set on a per-worker node basis”
Tomioka teaches computing the number of gradients that exceeds a threshold at worker node and threshold is set corresponding to predefined threshold.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
As cited above in para. 50, Tomioka teaches parameters [values] are adjusted based on in depended on the number of gradients are exceeds a threshold.
Ginsburg, Tomioka, and the instant application are analogous art because they are all directed to systems involving backpropagation for data processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg with Tomioka because of the following, “[a] neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model. Where the neural network is represented as a graph the subsets are referred to as subgraphs…In various examples described herein model parallelism is combined with asynchronous updates of the neural network subgraph parameters at the individual worker nodes. This scheme is found to give extremely good efficiency (as explained with reference to FIG. 9 below) and is found empirically to work well in practice despite the fact that the conventional theoretical convergence guarantee for stochastic gradient descent would not apply to the asynchronous updates of the subgraph parameters” (Tomioka, [0026]-[0027]).
Regarding Claim 2:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 1, and Ginsburg further teaches:
wherein the adjustment parameter is a scale factor, and wherein the scale factor is applied on the backward pass to at least a subset of the gradients with respect to at least one of the activations and the gradients with respect to the network weights, wherein the scale factor is updated in dependence on the proportion of the gradients of that subset that have a value falling above a pre-defined threshold
Ginsburg, [38], “the calculated gradients are also propagated backwards. In one or more embodiments, the gradients for matrix X (dX) may be calculated by the convolution of the gradient of Y (dY) using the matrix of weights W as a filter. In order to ensure that the absolute value of dX[i] is less than an upper threshold, (e.g., |dx[i]|<U, the following conditions are imposed”
[39], “However, if any one of the conditions is not met, gradient of Y, dY, and the matrix W are both rescaled such that dY=(α/k1, dy′.sub.16), where dy′[i]=dy[i]*k1; and W=(β/k2, w′.sub.16), where w′[i]=w[i]*k2. To ensure that no overflow can occur during rescaling, scale values k1 and k2 conform to the following conditions: k1*amax(dy)<U; and k2*amax(w)<U. Likewise, to ensure that no underflow can occur during rescaling, scale values k1 and k2 conform to the following conditions: k1*amean(dy)>L; and k2*amean(w)>L”
Ginsburg teaches wherein the parameter k1 and k2 is scale factor, wherein the scale factor is applied for forward and backward pass with respect to the network weights, wherein the scale factor is updated based on the value is above the threshold.
Regarding Claim 3:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 2, and Ginsburg further teaches:
comprising applying the scale factor to at least one of gradients with respect to weights and gradients with respect to activations of all layers of the network by multiplying the loss function by the scale factor
Ginsburg, [52], “Initially gradients are very small, so λ*ΔW(t) is much smaller than W, and well below normal float16 range, therefore, using traditional float16 format may cause the gradients to vanish. As a solution, the modified float16 data format described herein can be extended to prevent the loss of gradient data (precision). At later stages in training, gradients can be high, but λ becomes small, so λ*ΔW(t) becomes much smaller than W, and the weight update will disappear in traditional float16 formats due to rounding”
Ginsburg teaches loss of gradient data ΔW(t) multiply by scaler λ.
Regarding Claim 8:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 1, and Ginsburg further teaches:
comprising storing at least a subset of the network weights, gradients and activations in computer memory in floating-point format
Ginsburg, [54], “One possible solution is to use an extra copy of weights in float (32 bits) format. According to such embodiments, one copy of weights is stored in memory as float32, and a second as float16 for forward and backward propagation”
[52], “Initially gradients are very small, so λ*ΔW(t) is much smaller than W, and well below normal float16 range, therefore, using traditional float16 format may cause the gradients to vanish”
[8], “wherein a matrix is represented by the tuple X, where X=(a, v[.]), wherein a is a float scale factor and v[.] are scaled values stored in the float16 format”
Ginsburg teaches gradient, network weight and value store in floating format.
Regarding Claim 10:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 8, and Ginsburg further teaches:
comprising storing at least a subset of the network weights, gradients and activations in computer memory in sixteen-bit floating-point format
Ginsburg, [8], “wherein a matrix is represented by the tuple X, where X=(a, v[.]), wherein a is a float scale factor and v[.] are scaled values stored in the float16 format”
Ginsburg teaches storing in the float 16-bit format.
Regarding Claim 11:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 8, and Ginsburg further teaches:
comprising storing the subset of values in a floating-point format, and wherein the adjustment parameter is an exponent bias applied to the floating-point representations of the subset of weights, gradients and activations
Ginsburg, [11], “the novel data representation described herein can be used to convert single precision float into a (half-precision) float16 format that uses scalars for exponent extension”
Ginsburg teaches uses an exponent bias applied to floating format.
Regarding Claim 12:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 11, and Ginsburg further teaches:
wherein the subset of values in the neural network is a subset of network weights and activations and the adjustment parameter is an exponent bias applied to the subset of values of the network weights and activations in the forward pass
Ginsburg, [11], “the novel data representation described herein can be used to convert single precision float into a (half-precision) float16 format that uses scalars for exponent extension”
[15], “the next neuron in the next layer in sequence. The neuron in each layer receives the weighted output from the previous neuron as input, and the process is propagated forward at step 109 for each intervening layer between the input and output layers”
Ginsburg teaches an exponent bias applied to floating format.
Regarding Claim 14:
Ginsburg discloses:
A computer system comprising one or more processors configured to train a multi-layer neural network comprising a set of network weights, and memory holding the network weights, the processor configured to train the neural network by:
Ginsburg, [15], “ FIG. 1 depicts an exemplary computer-implemented method for training an artificial neural network. Steps 101-109 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 1, training of an artificial neural network typically begins at step 101 with receiving a training set of data as input. At step 103, the data is fed (typically as one or more matrices of values) into a corresponding number of neurons. At step 105, the data at each neuron is manipulated according to pre-determined parameters (weights)”
Ginsburg teaches a method of training based on set of training data and weight of neural network.
Ginsburg, [60], “In one embodiment, the processes 200 and 300 may be performed, in whole or in part, by graphics subsystem 405 in conjunction with the processor 401 and memory 402, with any resulting output displayed in attached display device 410”
Ginsburg teaches using a processor to train the multi-layer.
receiving a set of training data
Ginsburg, [15], “training of an artificial neural network typically begins at step 101 with receiving a training set of data as input”
Ginsburg teaches receiving a training data.
processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data
Ginsburg, [29], “forward propagation can be performed for any layer of a neural network (e.g., inner product layers) or convolutional neural network (e.g., convolutional layers+ inner product layers). To avoid the issue of vanishing or exploding activations (due to underflow and overflow, respectively), the rescaling operations described above can be performed”
[41], “the forward and backward propagation are performed in a neural network, the gradients for the weights that are used to adjust the influence of the data output from each neuron are calculated and subsequently readjusted”
Ginsburg teaches training data processing using forward and backward pass which comprising process of activation function of weight and training data.
and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described”
[3], “The output is then compared to the target output using a loss function, and an error value is calculated for each of the elements in the output layer. During back prop phase the gradients of error function are computed and then propagated backwards through the layers to determine gradients corresponding to each neuron”
Ginsburg teaches computing gradients of a loss with respect to activations.
wherein the values on the forward pass comprise at least one of the network weights or computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations or gradients with respect to weights
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 2, backward propagation begins at step 201, wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, WT)”
Ginsburg teaches network weights, update parameter of the neural network and compute gradient respect to forward and backward pass.
wherein the values are in an eight-bit or a sixteen-bit floating-point format
Ginsburg, [54], “According to such embodiments, one copy of weights is stored in memory as float32, and a second as float16 for forward and backward propagation.”
[4], “these solutions typically use a 16 bit floating-point (float16) representation.”
Ginsburg teaches weights [values] in a float16 format [sixteen-bit floating-point format]. Para. 4 specifies float16 is the float-point format.
storing the values to memory
Ginsburg, [54], “One possible solution is to use an extra copy of weights in float (32 bits) format. According to such embodiments, one copy of weights is stored in memory as float32”
Ginsburg teaches storing the values to memory.
updating the network weights in dependence on the computed gradients with respect to the weights
Ginsburg, [36], “wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, WT)”
Ginsburg teaches updating the weights of the network based on gradient.
Ginsburg does not explicitly disclose:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
However, in the same field, analogous art Tomioka teaches:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
Tomioka, [0026], “A neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model”
[0050] “The worker node computes 802 one or more gradients of a loss function with respect to (1) the message it received during the forward pass and (2) the parameters of the local subgraph… If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node”
[0022] “The training data instance is labeled and so the ground truth output of the neural network is known and the difference or error between the observed output and the ground truth output is found and provides information about a loss function which is passed back through the neural network layers in a backward propagation or backwards pass. A search is made to try find a minimum of the loss function which is a set of weights of the neural network that enable the output of the neural network to match the ground truth data”
Tomioka teaches a subset of values in the neural network.
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold
Tomioka, [0050], “The worker node checks the number of gradients in the accumulator. If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node. It then clears 810 the accumulator. As mentioned above the threshold at operation 804 is either set globally for the pipeline as a whole or is set on a per-worker node basis”
Tomioka teaches computing number of gradients exceeds a threshold at worker node and threshold is set corresponding to predefined threshold.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
As cited above in para. 50, Tomioka teaches parameters [values] are adjusted based on in depended on the number of gradients are exceeds a threshold.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg with Tomioka because of the following, “[a] neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model. Where the neural network is represented as a graph the subsets are referred to as subgraphs…In various examples described herein model parallelism is combined with asynchronous updates of the neural network subgraph parameters at the individual worker nodes. This scheme is found to give extremely good efficiency (as explained with reference to FIG. 9 below) and is found empirically to work well in practice despite the fact that the conventional theoretical convergence guarantee for stochastic gradient descent would not apply to the asynchronous updates of the subgraph parameters” (Tomioka, [0026]-[0027]).
Regarding Claim 15:
As discussed above Ginsburg in view of Tomioka teach [the] computer system of claim 14, and Tomioka further teaches:
comprising a plurality of processors, wherein each processor is configured to process a respective subset of the training data
Tomioka, [0026], “A neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model”
[0050] “The worker node computes 802 one or more gradients of a loss function with respect to (1) the message it received during the forward pass and (2) the parameters of the local subgraph… If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node”
[0022] “The training data instance is labeled and so the ground truth output of the neural network is known and the difference or error between the observed output and the ground truth output is found and provides information about a loss function which is passed back through the neural network layers in a backward propagation or backwards pass. A search is made to try find a minimum of the loss function which is a set of weights of the neural network that enable the output of the neural network to match the ground truth data”
Tomioka teaches a subset of values in the neural network.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg with Tomioka because of the following, “[a] neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model. Where the neural network is represented as a graph the subsets are referred to as subgraphs…In various examples described herein model parallelism is combined with asynchronous updates of the neural network subgraph parameters at the individual worker nodes. This scheme is found to give extremely good efficiency (as explained with reference to FIG. 9 below) and is found empirically to work well in practice despite the fact that the conventional theoretical convergence guarantee for stochastic gradient descent would not apply to the asynchronous updates of the subgraph parameters” (Tomioka, [0026]-[0027]).
Regarding Claim 16:
As discussed above Ginsburg in view of Tomioka teach [the] computer system of claim 15, and Ginsburg further teaches:
wherein the adjustment parameter is updated in dependence on an aggregated proportion of values for all processors falling above a predefined threshold, the aggregated proportion computed by aggregating a computed proportion of the subset of values falling above the predefined threshold for each of the plurality of processors
Ginsburg, [38], “the calculated gradients are also propagated backwards. In one or more embodiments, the gradients for matrix X (dX) may be calculated by the convolution of the gradient of Y (dY) using the matrix of weights W as a filter. In order to ensure that the absolute value of dX[i] is less than an upper threshold, (e.g., |dx[i]|<U, the following conditions are imposed” and Column 11 , lines 5-21 “However, if any one of the conditions is not met, gradient of Y, dY, and the matrix W are both rescaled such that dY=(α/k1, dy′.sub.16), where dy′[i]=dy[i]*k1; and W=(β/k2, w′.sub.16), where w′[i]=w[i]*k2. To ensure that no overflow can occur during rescaling, scale values k1 and k2 conform to the following conditions: k1*amax(dy)<U; and k2*amax(w)<U. Likewise, to ensure that no underflow can occur during rescaling, scale values k1 and k2 conform to the following conditions: k1*amean(dy)>L; and k2*amean(w)>L”
Ginsburg teaches wherein the parameter k1 and k2 is scale factor, wherein the scale factor is applied for forward and backward pass with respect to the network weights, wherein the scale factor is updated based on the value is above the threshold.
[60], “In one embodiment, the processes 200 and 300 may be performed, in whole or in part, by graphics subsystem 405 in conjunction with the processor 401 and memory 402, with any resulting output displayed in attached display device 410”
Ginsburg teaches processor to train multi-layer.
Regarding Claim 17:
Ginsburg discloses:
A non-transitory computer-readable storage medium storing computer program instructions which when executed perform a method of training, based on a set of training data, a multi-layer neural network comprising a set of network weights, the method comprising:
Ginsburg, [15], “ FIG. 1 depicts an exemplary computer-implemented method for training an artificial neural network. Steps 101-109 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 1, training of an artificial neural network typically begins at step 101 with receiving a training set of data as input. At step 103, the data is fed (typically as one or more matrices of values) into a corresponding number of neurons. At step 105, the data at each neuron is manipulated according to pre-determined parameters (weights)”
Ginsburg teaches a method of training based on set of training data and weight of neural network
[60], “In one embodiment, the processes 200 and 300 may be performed, in whole or in part, by graphics subsystem 405 in conjunction with the processor 401 and memory 402, with any resulting output displayed in attached display device 410”
Ginsburg teaches processor to train multi-layer.
processing the training data in respective forward and backward passes through a sequence of layers of the network, the forward pass comprising computing a set of activations by applying an activation function in dependence on the network weights and training data
Ginsburg, [29], “forward propagation can be performed for any layer of a neural network (e.g., inner product layers) or convolutional neural network (e.g., convolutional layers+ inner product layers). To avoid the issue of vanishing or exploding activations (due to underflow and overflow, respectively), the rescaling operations described above can be performed”
[41], “the forward and backward propagation are performed in a neural network, the gradients for the weights that are used to adjust the influence of the data output from each neuron are calculated and subsequently readjusted”
Ginsburg teaches training data processing using forward and backward pass which comprising process of activation function of weight and training data.
and the backward pass comprising determining a set of gradients of a pre-determined loss function with respect to the weights and/or activations of the network
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described”
[3], “The output is then compared to the target output using a loss function, and an error value is calculated for each of the elements in the output layer. During back prop phase the gradients of error function are computed and then propagated backwards through the layers to determine gradients corresponding to each neuron”
Ginsburg teaches computing gradients of a loss with respect to activations.
wherein the values on the forward pass comprise at least one of the network weights or computed activations, and the values on the backwards pass comprise the computed gradients with respect to activations or gradients with respect to weights
Ginsburg, [36], “three main operations are performed during backward propagation in a convolutional layer. FIG. 2 depicts the three main computer-implemented operations performed during backward propagation. Steps 201-205 describe exemplary steps of the flowchart 100 in accordance with the various embodiments herein described. As depicted in FIG. 2, backward propagation begins at step 201, wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, WT)”
Ginsburg teaches network weights, update parameter of the neural network and compute gradient respect to forward and backward pass.
wherein the values are in an eight-bit or a sixteen-bit floating- point format
Ginsburg, [54], “According to such embodiments, one copy of weights is stored in memory as float32, and a second as float16 for forward and backward propagation.”
[4], “these solutions typically use a 16 bit floating-point (float16) representation.”
Ginsburg teaches weights [values] in a float16 format [sixteen-bit floating-point format]. Para. 4 specifies float16 is the float-point format.
updating the network weights in dependence on the computed gradients with respect to the weights
Ginsburg, [36], “wherein gradients are propagated backward. In one or more embodiments, the gradient for input matrix X can be calculated as the convolution of the gradient of Y and the values for weights W: e.g., dX=conv(dY, WT)”
Ginsburg teaches updating the weights of the network based on gradient.
Ginsburg does not explicitly disclose:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
However, in the same field, analogous art Tomioka teaches:
wherein an adjustment parameter is applied to at least a subset of values in the neural network
Tomioka, [0026], “A neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model”
[0050] “The worker node computes 802 one or more gradients of a loss function with respect to (1) the message it received during the forward pass and (2) the parameters of the local subgraph… If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node”
[0022] “The training data instance is labeled and so the ground truth output of the neural network is known and the difference or error between the observed output and the ground truth output is found and provides information about a loss function which is passed back through the neural network layers in a backward propagation or backwards pass. A search is made to try find a minimum of the loss function which is a set of weights of the neural network that enable the output of the neural network to match the ground truth data”
Tomioka teaches a subset of values in the neural network.
on at least one of the forward and backward pass, computing a proportion of the subset of values falling above a predefined threshold
Tomioka, [0050], “The worker node checks the number of gradients in the accumulator. If the number of gradients in the accumulator meets or exceeds a threshold at check 804 the worker node asynchronously updates 806 the parameters of the neural network subgraph at the worker node. It then clears 810 the accumulator. As mentioned above the threshold at operation 804 is either set globally for the pipeline as a whole or is set on a per-worker node basis”
Tomioka teaches computing number of gradients exceeds a threshold at worker node and threshold is set corresponding to predefined threshold.
updating the adjustment parameter applied to the subset of values in dependence on the computed proportion
As cited above in para. 50, Tomioka teaches parameters [values] are adjusted based on in depended on the number of gradients are exceeds a threshold.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg with Tomioka because of the following, “[a] neural network is distributed over the worker nodes so that model parallelism is implemented whereby individual ones of the worker nodes hold subsets of the neural network model. Where the neural network is represented as a graph the subsets are referred to as subgraphs…In various examples described herein model parallelism is combined with asynchronous updates of the neural network subgraph parameters at the individual worker nodes. This scheme is found to give extremely good efficiency (as explained with reference to FIG. 9 below) and is found empirically to work well in practice despite the fact that the conventional theoretical convergence guarantee for stochastic gradient descent would not apply to the asynchronous updates of the subgraph parameters” (Tomioka, [0026]-[0027]).
Claims 4-7 are rejected under 35 U.S.C. 103 as being unpatentable over Ginsburg in view of Tomioka as applied to claim 2 above, and further in view of Zeng et al. (“Multi-Stage Contextual Deep Learning for Pedestrian Detection”), hereinafter Zeng.
Regarding Claim 4:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 2, but do not explicitly disclose:
comprising constructing a histogram of gradients, the histogram comprising a plurality of bins, wherein the scale factor is updated based on a proportion of gradients occupying bins above a threshold value
However, in the same field, analogous art Zeng teaches:
comprising constructing a histogram of gradients, the histogram comprising a plurality of bins, wherein the scale factor is updated based on a proportion of gradients occupying bins above a threshold value
Zeng, 3.1. Feature preparation, pp. 122-123, “Our basic classification model consists of 15 × 5 blocks of HOG and CSS features with 36 dimensions per block. The widely used version of HOG feature in [24] contains 31-dimensional feature vector, where the feature set is augmented to include both contrast sensitive and contrast insensitive features. In this implementation, 9 bins of unsigned gradient orientations, 18 bins of signed gradient orientations and 4 bins of overall gradient energy in four nearby cells are used. Denote the within-class variance of the kth feature in the (i, j)th block by
PNG
media_image1.png
23
220
media_image1.png
Greyscale
. Denote the between-class variance of the kth feature in the (i, j)th block by
PNG
media_image2.png
21
34
media_image2.png
Greyscale
”
Zeng teaches a histogram of gradient comprising plurality of bins, wherein the entries of kth feature are updated by
PNG
media_image2.png
21
34
media_image2.png
Greyscale
.
Ginsburg, Tomioka, Zeng, and the instant application are analogous art because they are all directed to systems involving backpropagation for data processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg and Tomioka with Zeng because of the following, “The discriminative power of the 31 bins is shown in Fig. 2. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load” (Zeng, p. 123).
Regarding Claim 5:
As discussed above Ginsburg in view of Tomioka, and further in view of Zeng teach [the] method of claim 4, and Zeng further teaches:
comprising constructing a respective histogram of gradients for each layer of the neural network, wherein the proportion of gradients occupying each of a set of bins for each histogram is input to an accumulator to obtain an aggregated proportion for each bin, the scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold
Zeng, 3.1. Feature preparation, pp. 122-123 “Our basic classification model consists of 15 × 5 blocks of HOG and CSS features with 36 dimensions per block. The widely used version of HOG feature in [24] contains 31-dimensional feature vector, where the feature set is augmented to include both contrast sensitive and contrast insensitive features. In this implementation, 9 bins of unsigned gradient orientations, 18 bins of signed gradient orientations and 4 bins of overall gradient energy in four nearby cells are used. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load”
Zeng teaches constructing a histogram of gradients by combining each bin based upon kth value.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg, Tomioka, and Zeng further with Zeng because of the following, “The discriminative power of the 31 bins is shown in Fig. 2. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load” (Zeng, p. 123).
Regarding Claim 6:
As discussed above Ginsburg in view of Tomioka, and further in view of Zeng teach [the] method of claim 4, and Zeng further teaches:
comprising constructing a respective histogram of gradients for each layer, wherein for each layer a respective layer-wise scale factor is applied during the backward pass, the layer-wise scale factor being updated based on a proportion of gradients in the histogram for the corresponding layer occupying bins above a corresponding layer-wise threshold value
Zeng, 3.3. Stage-by-stage training of the deep model, p. 124 “Step 1.1 (1 and 2 in Algorithm 1): the layer-by-layer unsupervised pre-training approach in [26] is used to train the hidden-to-hidden transfer matrices…Step 1.2 (3 in Algorithm 1): BP is used for fine-tuning all the Wh,i+1 together, with Ws,i+1 = 0 and Fi+1 = 0”
Experiments, p. 125, “At both training and testing stages, we use the HOG and CSS features described in Section 3.1 and a linear SVM classifier to generate score maps as the input of the bottom layer. A conservative threshold is used to prune samples and to reduce the computational load. The score map in each layer is generated in a 3×3 window and we combine 11 pyramids with the maximum score aligned to be the center of score map”
3.4.2 Analysis on Step 2, p. 124, “At the BP stage, the gradient for updatingWh,i+1 andWs,i+1”
Zeng teaches layer by layer for the scale factor and scale factor being updated based upon gradient.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg, Tomioka, and Zeng further with Zeng because of the following, “The discriminative power of the 31 bins is shown in Fig. 2. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load” (Zeng, p. 123).
Regarding Claim 7:
As discussed above Ginsburg in view of Tomioka, and further in view of Zeng teach [the] method of claim 4, and Zeng further teaches:
when implemented on a plurality of processors, wherein each processor processes a respective subset of the training data in each of the forward and backward passes, and computes a respective histogram of gradients for the corresponding subset of the training data, each histogram having defined a common set of bins, wherein the proportion of gradients occupying each bin of the set of bins defined for each histogram is aggregated to obtain an aggregated proportion for each bin, with a scale factor being derived by computing an aggregated proportion occupying bins above an overall threshold
Zeng, 4.1. Overall Performance, p. 125, “We focus on the reasonable subset, i.e. images with 50-pixel or taller, unoccluded or partially occluded pedestrians” and 3.1. Feature preparation , Page 122-123 “9 bins of unsigned gradient orientations, 18 bins of signed gradient orientations and 4 bins of overall gradient energy in four nearby cells are used…The discriminative power of the 31 bins is shown in Fig. 2. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load”
3.3. Stage-by-stage training of the deep model, p. 124, “Step 1.2 (3 in Algorithm 1): BP is used for fine-tuning all the Wh,i+1 together, with Ws,i+1 = 0 and Fi+1 = 0”
Zeng teaches subset of the data, 18 bins assigned of for the gradient orientations.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg, Tomioka, and Zeng further with Zeng because of the following, “The discriminative power of the 31 bins is shown in Fig. 2. We discard 6 bins with the least discriminative power. Therefore, the HOG features with 25-dimensions per block are used to reduce computational load” (Zeng, p. 123).
Claims 9 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Ginsburg in view of Tomioka as applied to claim 8 and 11 above, respectively, and further in view of Mellempudi et al. (“Mixed Precision Training with 8-bit Floating Point”), hereinafter Mellempudi.
Regarding Claim 9:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 8, but do not explicitly disclose:
comprising storing at least a subset of the network weights, gradients and activations in computer memory in eight-bit floating-point format
However, in the same field, analogous art Mellempudi teaches:
comprising storing at least a subset of the network weights, gradients and activations in computer memory in eight-bit floating-point format
Mellempudi, 3 Training Method, p. 3, “To perform weight update operation, first the 8-bit weight gradients need to be scaled back by dividing the weight gradients with ’loss scale’ parameter”
Mellempudi teaches 8-bit format of the gradients.
Ginsburg, Tomioka, Mellempudi, and the instant application are analogous art because they are all directed to systems involving backpropagation for data processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg and Tomioka with Mellempudi because of the following, “We propose easy to implement and scalable solution for building FP8 compute primitives, eliminating the need for stochastic rounding hardware in the critical compute path, as proposed in [24], thereby reducing the cost and complexity of the MAC unit” (Mellempudi, p. 8).
Regarding Claim 13:
As discussed above Ginsburg in view of Tomioka teach [the] method of claim 11, but do not explicitly disclose:
wherein a subset of network weights, activations and gradients which are inputs to compute operations in at least one of the forward and backward passes are stored in eight-bit floating-point format, the compute operations comprising at least one of a matrix operation and a convolution operation
However, in the same field, analogous art Mellempudi teaches:
wherein a subset of network weights, activations and gradients which are inputs to compute operations in at least one of the forward and backward passes are stored in eight-bit floating-point format, the compute operations comprising at least one of a matrix operation and a convolution operation
Mellempudi, 3 Training Method, p. 3, “To perform weight update operation, first the 8-bit weight gradients need to be scaled back by dividing the weight gradients with ’loss scale’ parameter”
Mellempudi teaches 8-bit format of the gradients.
Ginsburg, Tomioka, Mellempudi, and the instant application are analogous art because they are all directed to systems involving backpropagation for data processing.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Ginsburg and Tomioka with Mellempudi because of the following, “We propose easy to implement and scalable solution for building FP8 compute primitives, eliminating the need for stochastic rounding hardware in the critical compute path, as proposed in [24], thereby reducing the cost and complexity of the MAC unit” (Mellempudi, p. 8).
Response to Arguments
Applicant's arguments filed April 15, 2026 (“Remarks”) have been fully considered but they are not persuasive.
35 U.S.C. § 101:
Remarks, pp. 7-9. Applicant argues with respect to amended claim 1 that the claim integrates the judicial exception into a practical application by improving the functioning of a computer, citing paras. [0038], [0053], [0055], [0059]-[0062], and [0091]-[0092] of the published application (US 20230186095). Examiner respectfully disagrees. The improvements cited in the specification are not fully recited and reflected within the claim. For example, para. [0092] discusses maintaining a histogram of weights and activations. However, claim 1 does not recite nor reflect maintaining a histogram of weights and activations. For at least this reason, amended claim 1 does not integrate the judicial exception into a practical application, and remains rejected under 35 U.S.C. § 101.
35 U.S.C. § 103:
Remarks, pp. 9-10. Applicant argues Ginsburg cannot teach the feature of claim 1. Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986). It is the combination of Ginsburg in view of Tomioka that teaches the limitation of claim 1.
Remarks, pp. 11-12. Applicant further argues Tomioka does not teach the claimed limitation, specifically “the number of gradients” does not correspond to “the gradient values”. Examiner respectfully disagrees. It is unclear how “the number of gradients” is not referring to “gradient values” when both explicitly are referring to gradients. When considered gradient values are considered with the claim language “a proportion of the subset of values”, it becomes more explicit that “the number of gradients” corresponds to “a proportion of the subset of values”.
Remarks, pp. 11-12. Applicant further argues Tomioka does not teach a “computed proportion” of a subset of values falling above a predefined threshold. Examiner respectfully disagrees. As cited and discussed in above in section 103, Tomioka teaches adjusting parameters based on the number of gradients exceeding a threshold.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to STEVEN PHUNG whose telephone number is (703) 756-1499. The examiner can normally be reached Monday-Thursday: 9:00AM-4:00PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KAMRAN AFSHAR can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.H.P./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights