Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This action is in response to the amendments filed 02/27/2026. Claims 1, 10, and 18 have been amended, claims 1-19 are currently pending.
Response to Arguments
Applicant’s arguments regarding the 101 rejection have been fully considered but they are not persuasive. Applicant argues that the claim limitations directed to generating differential activation data and output data during forward propagation of a neural network could not practically be performed in the human mind. Examiner notes that as amended, these the broadest reasonable interpretation of these limitations includes mathematical calculations in light of Applicant’s specification.
Applicant also argues that any claimed judicial exceptions are integrated into a practical application, as the claims recite an “improvement wherein a memory size for the generating of differential data is determined based on a number of elements of differential input data and a maximum value of dimensions of an output of performing forward propagation for each of the plurality of layers such that the memory usage is reduced and execution speed is enhanced”. Examiner respectfully disagrees and notes that determining a size of a memory for generating differential data based on elements of input data and dimensions of output data was interpreted as a mental step, as the claims do not describe any particular technical operations required for determining this memory size. As per Section 2106.05(a) of the MPEP, “the judicial exception alone cannot provide the improvement”. Applicant has not shown how this mental step is combined with additional elements to provide the alleged improvement. The 101 rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Applicant’s arguments regarding the prior art rejection have been fully considered but they are not persuasive. Applicant argues that the Bai reference does not teach performing forward propagation operations without performing backpropagation. Examiner respectfully disagrees and notes that at least paragraph [0022] of Bai recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium”. One of ordinary skill would recognize that in this embodiment of Bai, “replacing” backpropagation would not require performing backpropagation.
Applicant also argues that Bai does not teach generating a memory size for the differential data based on a number of elements of input data and a maximum value of dimensions of output data from a forward propagation operation. Examiner respectfully disagrees and notes the Bai teaches that amount of required memory is determined using at least the size of the data generated during the forward passes, which includes the input data described in at least paragraph [0009] and the output of the corresponding Jacobian matrix for each layer calculated during forward propagation as described in at least paragraph [0022]. Examiner also notes that “such that memory usage is reduced and execution speed is enhanced” is interpreted as the intended use or necessary outcome of determining the size of the memory and does not provide additional patentable weight to this limitation (see MPEP 2103). The prior art rejections have been updated to include the amended limitations and to clarify the reasoning given for the limitations that were not amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-19 are rejected under 35 U.S.C. 101. Claims 1-8 are directed to a method, claim 9 is directed to a non-transitory computer readable medium, claims 10-17 are directed to a system, and claims 18-19 are directed to a separate method; therefore, claims 1-19 fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter). However, claims 1-19 fall within the judicial exception of an abstract idea, specifically the abstract ideas of “Mental Processes” (including observation, evaluation, and opinion) and “Mathematical Concepts (including mathematical calculations and relationships)”.
Claim 1:
Claim 1 is directed to a method; therefore, the claim does fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Claim 1 recites the following abstract ideas:
generate differential data of the activation data of the corresponding layer with respect to input data by performing forward propagation of the corresponding layer without performing backpropagation (Examiner notes that the broadest reasonable interpretation of “generating” differential data by performing forward propagation without performing backpropagation includes a mathematical calculation as supported by at least paragraph [0092] of Applicant’s specification);
and generate differential data of output data of the neural network with respect to the input data, based on the generated differential data of each layer, during the forward propagation of each layer of the neural network (Examiner notes that the broadest reasonable interpretation of “generating” differential data by performing forward propagation also includes a mathematical calculation as supported by at least paragraph [0093] of Applicant’s specification);
wherein a memory size for the generating of the differential data is determined based on a number of elements of differential input data and a maximum value of dimension of an output of performing forward propagation for each of the plurality of layers such that memory usage is reduced and execution speed is enhanced (mental step directed to observation, evaluation – a person could determine a required or optimal memory size for inference of a neural network based on observed or determined elements of differential input data and an observed or determined maximum value of dimensions of an output from a forward propagation calculation. Examiner notes that the broadest reasonable interpretation of determining this memory size includes a mathematical calculation as supported by at least paragraph [0096] of Applicant’s specification. Examiner also notes that “such that memory usage is reduced and execution speed is enhanced” is interpreted as the intended use or necessary outcome of determining the size of the memory and does not provide additional patentable weight to this limitation (see MPEP 2103)).
Claim 1 recites the following additional elements:
for each layer of a plurality of layers of a neural network for an input data provided to the neural network: obtain activation data of a corresponding layer of the plurality of layers, resulting from an inference operation of the corresponding layer. Obtaining activation data resulting from an inference operation for each layer of a plurality of layers is interpreted as well-understood, routine, conventional activity directed to receiving data over a network and as aspects of the technological environment in which the claimed abstract ideas are performed. These additional elements do not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea (see MPEP 2106.05(d)(II) and MPEP 2106.05(h)).
Claim 9 is a non-transitory computer readable medium claim which performs the inference method of claim 1. The only difference is that claim 9 requires a non-transitory computer readable medium, which is interpreted as a generic computer component merely used to apply, or perform, the abstract ideas as identified in the analysis of claim 1 (see MPEP 2106.05(f)). Therefore, claim 9 is rejected for the same reasons as claim 1.
Claim 10 is a system claim and its limitation is included in claim 1. The only difference is that claim 10 requires a system, which is interpreted as a generic computer component merely used to apply the abstract ideas as identified in the analysis of claim 1 (see MPEP 2106.05(f)). Therefore, claim 10 is rejected for the same reasons as claim 1.
Claim 18:
Claim 18 is directed to a method; therefore, the claim does fall within one of the four statutory categories (i.e., process, machine, manufacture, or composition of matter).
Claim 18 recites the following abstract ideas:
generating differential data of output data of a neural network based on respective differential data of each layer of the neural network, generated during corresponding forward propagation operations of the neural network without performing backpropagation (mental step directed to observation, evaluation – a person could generate differential output data in their mind based on observed or determined differential data generated during forward propagation operations for a neural network without performing a backpropagation calculation. Examiner notes that the broadest reasonable interpretation of “generating” differential data by performing forward propagation also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification);
wherein a memory size for the generating of the differential data is determined based on a number of elements of differential input data and a maximum value of dimension of an output of performing forward propagation for each of the plurality of layers such that memory usage is reduced and execution speed is enhanced (mental step directed to observation, evaluation – a person could determine a required or optimal memory size for inference of a neural network based on observed or determined elements of differential input data and an observed or determined maximum value of dimensions of an output from a forward propagation calculation. Examiner notes that the broadest reasonable interpretation of determining this memory size includes a mathematical calculation as supported by at least paragraph [0096] of Applicant’s specification. Examiner also notes that “such that memory usage is reduced and execution speed is enhanced” is interpreted as the intended use or necessary outcome of determining the size of the memory and does not provide additional patentable weight to this limitation (see MPEP 2103)).
Claim 18 recites the following additional elements:
wherein the differential data of output data is obtained based on a Jacobian matrix for input data of a layer of the plurality of layers. Obtaining differential output data based on a Jacobian matrix for input data of a given layer is interpreted as well-understood, routine, conventional activity directed to receiving data over a network and as aspects of the technological environment in which the claimed abstract ideas are performed. These additional elements do not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea (see MPEP 2106.05(d)(II) and MPEP 2106.05(h)).
The independent claims are not patent eligible.
Dependent claims 2-8, 11-17, and 19 when analyzed as a whole are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitations fail to establish that the claims are not directed to an abstract idea, as they recite further embellishment of the judicial exception.
Claim 2 recites wherein the generating of the differential data comprises: for each layer of the layers, calculating a Jacobian matrix with respect to the input data (mental step directed to observation, evaluation – a person could generate differential data in their mind by calculating a Jacobian matrix based on observed input data for each layer. Examiner notes that the broadest reasonable interpretation of “generating” differential data and calculating a Jacobian matrix also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification and the mathematical calculations associated with the Jacobian matrix as supported by at least [0098]-[0099] of Applicant’s specification).
Claim 3 recites wherein the generating of the differential data comprises: calculating a Jacobian matrix of the corresponding layer with respect to the input data by performing the inference operation of the corresponding layer (mental step directed to observation, evaluation – a person could generate differential data in their mind by calculating a Jacobian matrix as part of an inference operation based on observed input data. Examiner notes that the broadest reasonable interpretation of “generating” differential data and calculating a Jacobian matrix also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification and the mathematical calculations associated with the Jacobian matrix as supported by at least [0098]-[0099] of Applicant’s specification).
Claim 4 recites for each layer, calculating a Jacobian matrix of the corresponding layer with respect to the input data without performing backpropagation (mental step directed to observation, evaluation – a person could generate differential output data in their mind by calculating a Jacobian matrix based on observed input data without performing backpropagation operations. Examiner notes that the broadest reasonable interpretation of “generating” differential data and calculating a Jacobian matrix also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification and the mathematical calculations associated with the Jacobian matrix as supported by at least [0098]-[0099] of Applicant’s specification).
Claim 5 recites for each layer, performing the inference operation of the corresponding layer to generate the activation data of the corresponding layer; and generating output data of the neural network based on the generated activation data of each of the layers (mental step directed to observation, evaluation – a person could perform inference operations to generate activation data corresponding to a layer of a neural network in their mind and generate output data in their mind based on the generated activation data. Examiner notes that the broadest reasonable interpretation of “generating” differential data also includes the mathematical calculations associated with forward propagation, or inference operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification).
Claim 6 recites generating differential input data comprising one or more elements for a differential value among a plurality of elements of the input data (mental step directed to observation, evaluation – a person could generate differential input data comprising elements for differential values of observed input data in their mind. Examiner notes that the broadest reasonable interpretation of “generating” differential data also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification).
Claim 7 recites wherein the generating of the differential data comprises: for each layer, calculating a Jacobian matrix of the corresponding layer with respect to the differential input data (mental step directed to observation, evaluation – a person could generate differential data in their mind by calculating a Jacobian matrix based on observed differential input data. Examiner notes that the broadest reasonable interpretation of “generating” differential data and calculating a Jacobian matrix also includes the mathematical calculations associated with forward propagation operations as supported by at least paragraphs [0092]-[0095] of Applicant’s specification and the mathematical calculations associated with the Jacobian matrix as supported by at least [0098]-[0099] of Applicant’s specification).
Claim 8 recites wherein a memory size for inference of the neural network is determined based on a number of elements of the differential input data and a maximum value of dimensions of each Jacobian matrix of the plurality of layers with respect to the differential input data (mental step directed to observation, evaluation – a person could determine a required or optimal memory size for inference of a neural network based on observed or determined elements of differential input data and an observed or determined maximum value of dimensions of a Jacobian matrix).
Claim 11 is a system claim and its limitation is included in claim 2. Claim 11 is rejected for the same reasons as claim 2.
Claim 12 is a system claim and its limitation is included in claim 3. Claim 12 is rejected for the same reasons as claim 3.
Claim 13 is a system claim and its limitation is included in claim 4. Claim 13 is rejected for the same reasons as claim 4.
Claim 14 is a system claim and its limitation is included in claim 5. Claim 14 is rejected for the same reasons as claim 5.
Claim 15 is a system claim and its limitation is included in claim 6. Claim 15 is rejected for the same reasons as claim 6.
Claim 16 is a system claim and its limitation is included in claim 7. Claim 16 is rejected for the same reasons as claim 7.
Claim 17 is a system claim and its limitation is included in claim 8. Claim 17 is rejected for the same reasons as claim 8.
Claim 19 recites wherein the differential data of the output data of the neural network is obtained with respect to the input data, based on differential data of an output activation of a corresponding layer with respect to the input data (obtaining output data of a neural network based on differential activation output data corresponding to input data of a given layer is interpreted as well-understood, routine, conventional activity directed to receiving data over a network and as aspects of the technological environment in which the claimed abstract ideas are performed. These additional elements do not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea (see MPEP 2106.05(d)(II) and MPEP 2106.05(h)).
Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. Therefore, the claims are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bai et al* (US 20210042606 A1, herein Bai).
*this document was included in the IDS dated 08/11/2023
Regarding claim 1, Bai teaches a processor-implemented method for inference-based differential consideration (para. [0008] recites “In accordance with a first aspect of the invention, a computer-implemented method and corresponding system are provided for training a neural network”), comprising: for each layer of a plurality of layers of a neural network for an input data provided to the neural network: obtain activation data of a corresponding layer of the plurality of layers, resulting from an inference operation of the corresponding layer (para. [0009] recites “providing a neural network which comprises an iterative function (z[i+ 1] = f(z[i], θ, c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. The first layer of the stack of layers may receive an initial activation as input, which may for example be an output of yet another layer of the neural network” (i.e., obtaining activation data for a plurality of layers during a forward pass, or inference operation));
generate differential data of the activation data of the corresponding layer with respect to input data by performing forward propagation of the corresponding layer without performing backpropagation (para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium. Herein, the vector Jacobian product may be efficiently computed via automatic differentiation tools for any x, without having to explicitly write out the Jacobian matrix” (i.e., calculating the Jacobian vector product, which one of ordinary skill in the art would recognize as the projection of the Jacobian matrix into a vector, for a given layer instead of performing backpropagation)),
and generate differential data of output data of the neural network with respect to the input data, based on the generated differential data of each layer during the forward propagation of each layer of the neural network (para. [0009] recites “In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. The first layer of the stack of layers may receive an initial activation as input, which may for example be an output of yet another layer of the neural network” (i.e., generating activation data for a given layer with respect to the input data). Para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium” (i.e., generating differential activation data for a given layer with respect to the input data. Examiner notes that the broadest reasonable interpretation of “differential” data includes an interpretation data that can be input to a differentiation technique, or “differentiable” data));
wherein a memory size for the generating of the differential data is determined based on a number of elements of differential input data and a maximum value of dimension of an output of performing forward propagation for each of the plurality of layers such that memory usage is reduced and execution speed is enhanced (Bai para. [0003] recites “the training itself of a deep neural network requires a large amount of memory, since in addition to the weights per layer, also a large amount of temporary data has to be stored for the forward passes ('forward propagation') and backward passes ('backward propagation') during the training. For example, the layer output of each individual layer ('hidden state') during forward propagation may need to be stored as temporary data as it may be used in the backward propagation. This way, the training of a deep neural network may require many gigabytes of memory, with the memory requirements being expected to further increase as the complexity of models increases”. Bai para. [0011] recites “As described in the background section, during training, the iterative execution of a stack of layers still requires a sizable memory footprint, since the layer output of each individual layer (even if weight-tied) during forward propagation may need to be stored as temporary data as it may be used in the subsequent backward propagation” (i.e., the amount of memory required is determined using at least the size of the data generated during the forward passes, which includes the input data described in at least para. [0009] and the output of the corresponding Jacobian matrix for each layer as described in at least para. [0022]. Examiner notes that “such that memory usage is reduced and execution speed is enhanced” is interpreted as the intended use or necessary outcome of determining the size of the memory and does not provide additional patentable weight to this limitation (see MPEP 2103))).
Regarding claim 2, Bai teaches the method of claim 1, wherein the generating of the differential data comprises: for each layer of the layers, calculating a Jacobian matrix with respect to the input data (Para. [0017]-[0020] recite “performing the backward propagation part comprises: computing respective partial derivatives of the iterative function with respect to the weights and the part of the input; computing a gradient of the input of the iterative execution layer for the weights and/or the part of the input as a function of the respective partial derivative and a backpropagated gradient at the output of the iterative execution layer. The derivatives which are indicated above may be implemented via their analytic equations or computed, e.g., implemented via their automatic differentiation tools”. Para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium” (i.e., differential output data obtained based on a Jacobian matrix calculated from the input data of a given layer)).
Regarding claim 3, Bai teaches the method of claim 1, wherein the generating of the differential data comprises: calculating a Jacobian matrix of the corresponding layer with respect to the input data by performing the inference operation of the corresponding layer (para. [0009] recites “providing a neural network which comprises an iterative function (z[i+ 1] = f(z[i], θ, c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. Para. [0017]-[0020] recite “performing the backward propagation part comprises: computing respective partial derivatives of the iterative function with respect to the weights and the part of the input; computing a gradient of the input of the iterative execution layer for the weights and/or the part of the input as a function of the respective partial derivative and a backpropagated gradient at the output of the iterative execution layer. The derivatives which are indicated above may be implemented via their analytic equations or computed, e.g., implemented via their automatic differentiation tools”. Para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium” (i.e., differential output data obtained based on a Jacobian matrix calculated from the input data of a given layer obtained during the inference, or forward pass)).
Regarding claim 4, Bai teaches the method of claim 1, wherein the generating of the differential data comprises: for each layer, calculating a Jacobian matrix of the corresponding layer with respect to the input data without performing backpropagation (para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium. Herein, the vector Jacobian product may be efficiently computed via automatic differentiation tools for any x, without having to explicitly write out the Jacobian matrix” (i.e., calculating the Jacobian vector product, which one of ordinary skill in the art would recognize as the projection of the Jacobian matrix into a vector, for a given layer instead of performing backpropagation)).
Regarding claim 5, Bai teaches the method of claim 1, further comprising: for each layer, performing the inference operation of the corresponding layer to generate the activation data of the corresponding layer (para. [0009] recites “providing a neural network which comprises an iterative function (z[i+ 1] = f(z[i], θ, c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. The first layer of the stack of layers may receive an initial activation as input, which may for example be an output of yet another layer of the neural network” (i.e., obtaining activation data for a plurality of layers during a forward pass, or inference operation)); and generating output data of the neural network based on the generated activation data of each of the layers (para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium”. Para. [0024] recites “outputting the trained neural network comprises representing the stack of layers in the trained neural network by at least i) a data representation of a layer of the stack of layers, and ii) a hyperparameter defining a number of layers of the stack of layers at which the output of the stack of layers reaches or to a selected degree approximates the equilibrium point during forward propagation” (i.e., generating output data of the neural network based on the stack of layers, which includes the activation data as described in at least para. [0009])).
Regarding claim 6, Bai teaches the method of claim 1, further comprising: generating differential input data comprising one or more elements for a differential value among a plurality of elements of the input data (para. [0009] recites “providing a neural network which comprises an iterative function (z[i+ 1] = f(z[i], θ, c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. The first layer of the stack of layers may receive an initial activation as input, which may for example be an output of yet another layer of the neural network” (i.e., generating input data comprising one or more elements with values capable of being differentiable as described in at least para. [0017]-[0020])).
Regarding claim 7, Bai teaches the method of claim 6, wherein the generating of the differential data comprises: for each layer, calculating a Jacobian matrix of the corresponding layer with respect to the differential input data (para. [0017]-[0020] recite “performing the backward propagation part comprises: computing respective partial derivatives of the iterative function with respect to the weights and the part of the input; computing a gradient of the input of the iterative execution layer for the weights and/or the part of the input as a function of the respective partial derivative and a backpropagated gradient at the output of the iterative execution layer. The derivatives which are indicated above may be implemented via their analytic equations or computed, e.g., implemented via their automatic differentiation tools”. Para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium” (i.e., differential output data obtained based on a Jacobian matrix calculated from the differentiable input data)).
Regarding claim 8, Bai teaches the method of claim 7, wherein a memory size for inference of the neural network is determined based on a number of elements of the differential input data and a maximum value of dimensions of each Jacobian matrix of the plurality of layers with respect to the differential input data (Bai para. [0003] recites “the training itself of a deep neural network requires a large amount of memory, since in addition to the weights per layer, also a large amount of temporary data has to be stored for the forward passes ('forward propagation') and backward passes ('backward propagation') during the training. For example, the layer output of each individual layer ('hidden state') during forward propagation may need to be stored as temporary data as it may be used in the backward propagation. This way, the training of a deep neural network may require many gigabytes of memory, with the memory requirements being expected to further increase as the complexity of models increases”. Bai para. [0011] recites “As described in the background section, during training, the iterative execution of a stack of layers still requires a sizable memory footprint, since the layer output of each individual layer (even if weight-tied) during forward propagation may need to be stored as temporary data as it may be used in the subsequent backward propagation” (i.e., the amount of memory required is determined using the size of the data generated during the forward and backward passes, which includes the input data described in at least para. [0009] and the corresponding Jacobian matrix for each layer as described in at least para. [0022])).
Claim 9 is a non-transitory computer readable medium claim which performs the inference method of claim 1. The only difference is that claim 9 requires a non-transitory computer readable medium (Bai para. [0008] recites “a computer-readable medium is provided comprising transitory or non-transitory data representing model data defining a trained neural network”). Therefore, claim 9 is rejected for the same reasons as claim 1.
Claim 10 is a system claim and its limitation is included in claim 1. The only difference is that claim 10 requires a system (Bai para. [0008] recites “In accordance with a first aspect of the invention, a computer-implemented method and corresponding system are provided for training a neural network”). Therefore, claim 10 is rejected for the same reasons as claim 1.
Claim 11 is a system claim and its limitation is included in claim 2. Claim 11 is rejected for the same reasons as claim 2.
Claim 12 is a system claim and its limitation is included in claim 3. Claim 12 is rejected for the same reasons as claim 3.
Claim 13 is a system claim and its limitation is included in claim 4. Claim 13 is rejected for the same reasons as claim 4.
Claim 14 is a system claim and its limitation is included in claim 5. Claim 14 is rejected for the same reasons as claim 5.
Claim 15 is a system claim and its limitation is included in claim 6. Claim 15 is rejected for the same reasons as claim 6.
Claim 16 is a system claim and its limitation is included in claim 7. Claim 16 is rejected for the same reasons as claim 7.
Claim 17 is a system claim and its limitation is included in claim 8. Claim 17 is rejected for the same reasons as claim 8.
Regarding claim 18, Bai teaches a processor-implemented method (para. [0008] recites “In accordance with a first aspect of the invention, a computer-implemented method and corresponding system are provided for training a neural network”), comprising: generating differential data of output data of a neural network based on respective differential data of each layer of the neural network (para. [0009] recites “providing a neural network which comprises an iterative function (z[i+ 1] = f(z[i], θ, c(x)). Such an iterative function is known in the field of machine learning to be representable by a stack of layers which have mutually shared weights. In such a stack of layers, each layer except for the first layer may receive, as input, i) an output of the previous layer and ii) (a part of) an input to the stack of layers, being either the original input (x) to the neural network or a transformation of that input (c(x)), for example by one or more previous layers preceding the stack of layers in the neural network. The first layer of the stack of layers may receive an initial activation as input, which may for example be an output of yet another layer of the neural network” (i.e., obtaining activation data for a plurality of layers during a forward pass, or inference operation)) generated during corresponding forward propagation operations of the neural network without performing backpropagation (para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium. Herein, the vector Jacobian product may be efficiently computed via automatic differentiation tools for any x, without having to explicitly write out the Jacobian matrix” (i.e., calculating the Jacobian vector product, which one of ordinary skill in the art would recognize as the projection of the Jacobian matrix into a vector, for a given layer instead of, or before performing backpropagation));
wherein the differential data of output data is obtained based on a Jacobian matrix for input data of a layer of the plurality of layers (para. [0017]-[0020] recite “performing the backward propagation part comprises: computing respective partial derivatives of the iterative function with respect to the weights and the part of the input; computing a gradient of the input of the iterative execution layer for the weights and/or the part of the input as a function of the respective partial derivative and a backpropagated gradient at the output of the iterative execution layer. The derivatives which are indicated above may be implemented via their analytic equations or computed, e.g., implemented via their automatic differentiation tools”. Para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium” (i.e., differential output data obtained based on a Jacobian matrix));
and wherein a memory size for the generating of the differential data is determined based on a number of elements of differential input data and a maximum value of dimension of an output of performing forward propagation for each of the plurality of layers such that memory usage is reduced and execution speed is enhanced (Bai para. [0003] recites “the training itself of a deep neural network requires a large amount of memory, since in addition to the weights per layer, also a large amount of temporary data has to be stored for the forward passes ('forward propagation') and backward passes ('backward propagation') during the training. For example, the layer output of each individual layer ('hidden state') during forward propagation may need to be stored as temporary data as it may be used in the backward propagation. This way, the training of a deep neural network may require many gigabytes of memory, with the memory requirements being expected to further increase as the complexity of models increases”. Bai para. [0011] recites “As described in the background section, during training, the iterative execution of a stack of layers still requires a sizable memory footprint, since the layer output of each individual layer (even if weight-tied) during forward propagation may need to be stored as temporary data as it may be used in the subsequent backward propagation” (i.e., the amount of memory required is determined using at least the size of the data generated during the forward passes, which includes the input data described in at least para. [0009] and the output of the corresponding Jacobian matrix for each layer as described in at least para. [0022]. Examiner notes that “such that memory usage is reduced and execution speed is enhanced” is interpreted as the intended use or necessary outcome of determining the size of the memory and does not provide additional patentable weight to this limitation (see MPEP 2103))).
Regarding claim 19, Bai teaches the method of claim 18, wherein the differential data of the output data of the neural network is obtained with respect to the input data, based on differential data of an output activation of a corresponding layer with respect to the input data (para. [0022] recites “backpropagation of the backward gradient through the stack of layers may be replaced by solving the above linear system which may involve using one step of matrix multiplications that involves the Jacobian at equilibrium”. Para. [0024] recites “outputting the trained neural network comprises representing the stack of layers in the trained neural network by at least i) a data representation of a layer of the stack of layers, and ii) a hyperparameter defining a number of layers of the stack of layers at which the output of the stack of layers reaches or to a selected degree approximates the equilibrium point during forward propagation” (i.e., generating differential output data of the neural network based on the stack of layers, which includes the input data and the activation data as described in at least para. [0009])).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
“Neural Network Learning Without Backpropagation” (Wilamowski et al) teaches a method for training complex feed-forward neural network architectures without the necessity of the backward computations.
“Gradients without Backpropagation” (Baydin et al) teaches a method for computing gradients based solely on a forward gradient, or an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent.
“On Neural Differential Equations” (Kidger) teaches an overview of neural ordinary differential equations and potential applications.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEAH M FEITL whose telephone number is (571) 272-8350. The examiner can normally be reached on M-F 0900-1700 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached on (571) 270-5871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/L.M.F./ Examiner, Art Unit 2147
/ERIC NILSSON/ Primary Examiner, Art Unit 2151
Prosecution Insights