DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant's claim for the benefit of a PCT Application PCT/RU2021/000630, filed December 30, 2021 is acknowledged.
Information Disclosure Statement
The information disclosure statements submitted on 01/08/2025, 11/26/2025 and 04/16/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-18 are directed to a method, claim 19 is directed to a system and claim 20 is directed to a medium. Therefore, the claims are eligible under Step 1 for being directed to a process, a machine and a manufacture respectively.
Independent claims 1, 19 and 20:
Step 2A Prong 1:
Claims recite:
transforming the neural network topology to an equivalent analog network of analog components, including replacing composition or superposition of linear transformations by a single linear transformation - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper;
computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between analog components of the equivalent analog network - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine a weight matrix for the equivalent analog network; and
generating a schematic model for implementing the equivalent analog network based on the weight matrix, including selecting component parameter values for the analog components - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses the mental process of evaluating data and generating data based on judgement, which is observing, evaluating and judging that is practically capable of being performed in the human mind with the assistance of pen and paper.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
obtaining a neural network topology and weights of a trained neural network - the steps recited at a high level of generality, and amounts to mere data gathering which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g));
A system for hardware realization of neural networks, comprising: one or more processors; and memory; wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions; A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
obtaining a neural network topology and weights of a trained neural network - the steps recited at a high level of generality, and amounts to mere data gathering which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g));
A system for hardware realization of neural networks, comprising: one or more processors; and memory; wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions; A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions - These limitations amount to components of a general purpose computer that applies a judicial exception, by use of conventional computer functions (see MPEP § 2106.05(b)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 2:
Step 2A Prong 1: The claim recites the abstract ideas of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
prior to transforming the neural network topology to the equivalent analog network: adding regularizers to the neural network topology to reduce nominal values of the weights or to a respective reduce weight sum for each neuron - the step recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)); and
retraining the trained neural network to obtain updated weights for the weight matrix - the step recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
prior to transforming the neural network topology to the equivalent analog network: adding regularizers to the neural network topology to reduce nominal values of the weights or to a respective reduce weight sum for each neuron - the step recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)); and
retraining the trained neural network to obtain updated weights for the weight matrix - the step recited at a high level of generality, and amounts to mere data modifying which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 3:
Step 2A Prong 1:
Claims recite:
wherein each predetermined regularizer treats each batch normalization layer as a normalization and calculates combined convolution-batch normalization multipliers applied to an input neural network signal in the signal's propagation path, and reduces the absolute value of combined weights for each neuron - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine combined convolution-batch normalization multipliers and the absolute value of combined weights for each neuron.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
wherein the regularizers include a respective predetermined regularizer for each convolution batch normalization ReLU block - the step recited at a high level of generality, and amounts to selecting a particular data source or type of data to be manipulated, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
wherein the regularizers include a respective predetermined regularizer for each convolution batch normalization ReLU block - viewed individually or in combination, describes selecting a particular data source or type of data to be manipulated similar to selecting information, based on types of information and availability of information in a power-grid environment, for collection, analysis and display described in MPEP § 2106.05(g).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 4:
Step 2A Prong 1:
Claims recite:
wherein transforming the neural network topology into the equivalent analog network comprises translating weights of each batch normalization layer to weights of its previous layer - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine weights of each batch normalization layer.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 5:
Step 2A Prong 1: The claim recites the abstract ideas of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
wherein transforming the neural network topology into the equivalent analog network comprises merging layers that do not have an activation function - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
wherein transforming the neural network topology into the equivalent analog network comprises merging layers that do not have an activation function - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 6:
Step 2A Prong 1: The claim recites the abstract ideas of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
wherein transforming the neural network topology into the equivalent analog network comprises transforming a linear transformation followed by another linear transformation into a single linear transformation - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
wherein transforming the neural network topology into the equivalent analog network comprises transforming a linear transformation followed by another linear transformation into a single linear transformation - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 7:
Step 2A Prong 1:
Claim recites:
wherein transforming the neural network topology into the equivalent analog network comprises transforming layers with ReLU into ReLU1 - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine weight correction.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 8:
Step 2A Prong 1:
Claim recites:
wherein transforming layers with ReLU into ReLU1 comprises maintaining normal operation of the trained neural network during the transformation by analyzing a passage of signals through the trained neural network and performing weight correction - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine weight correction.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 9:
Step 2A Prong 1:
Claim recites:
wherein performing weight correction comprises: adjusting weights so as to restrict signals in the trained neural network below a physical limit
- Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to adjust weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 10:
Step 2A Prong 1:
Claim recites:
wherein performing weight correction comprises: when weights of a layer N are divided by a factor, adjusting weights of layer N+1 by multiplying the weights by the factor - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to adjust weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 11:
Step 2A Prong 1:
Claim recites:
wherein performing weight correction comprises: adjusting weights and bias of one or more neurons and adjustment of weights of outgoing connections of the one or more neurons - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to adjust weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 12:
Step 2A Prong 1:
Claim recites:
wherein performing weight correction comprises: repeating weight correction for the trained neural network until complete compliance is achieved - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to adjust weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 13:
Step 2A Prong 1:
Claim recites:
wherein performing weight correction comprises scaling signals on layers with unlimited ReLU so that they do not exceed a physical limitation - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to perform weight correction.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 14:
Step 2A Prong 1:
Claim recites:
wherein transforming the neural network topology to the equivalent analog network comprises introducing additional intermediate layers that limit a number of input or output links of neurons by splitting inputs or outputs of the neurons - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to perform the splitting.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 15:
Step 2A Prong 1:
Claim recites:
pruning at least some connections of the neural network topology - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to perform the pruning.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 16:
Step 2A Prong 1:
Claim recites:
quantizing and/or restricting the weights of the neural network topology - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to quantize and/or restrict the weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Dependent claim 17:
Step 2A Prong 1: The claim recites the abstract ideas of claim 1.
Step 2A Prong 2: This judicial exception is not integrated into a practical application because they recite the additional elements:
identifying non-linear elements in the neural network topology - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The claims are thus directed to the abstract idea.
Step 2B: The claims do not include additional elements that amount to significantly more than the judicial exception.
The additional elements:
identifying non-linear elements in the neural network topology - the step recited at a high level of generality, and amounts to insignificant application, which is a form of insignificant extra-solution activity (see MPEP § 2106.05(g)).
Accordingly, these additional elements do not amount to significantly more than the judicial exception. As such, the claims are ineligible.
Dependent claim 18:
Step 2A Prong 1:
Claim recites:
further comprising (i) calculating a respective range of weights for each layer of the neural network topology and (ii) calculating a respective sum of the weights for each neuron of the neural network topology - Under its broadest reasonable interpretation in light of the specification, this limitation encompasses a mathematical concept of a mathematical calculation of calculating using mathematical methods to determine a respective range of weights and a respective sum of the weights.
Step 2A Prong 2 & Step 2B: There are no additional elements recited so the claims do not provide a practical application and is not considered to be significantly more. As such, the claims are ineligible.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 6, 16-17 and 19-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Vorenkamp et al. (hereinafter Vorenkamp), US 20190026625 A1.
Regarding independent claim 1, Vorenkamp teaches a method for hardware realization of neural networks (Abstract; [0002]), comprising:
obtaining a neural network topology and weights of a trained neural network (Fig. 2, 202; [0024] At block 202, a behavioral description of a machine learning logic module is determined. The behavioral description may comprise machine code, e.g., a set of one or more instructions that may be executed by a CPU, that defines an architecture of the machine learning logic module. In one embodiment, the machine learning logic module may represent at least a portion of a neural network; [0028] In one embodiment, a “fully connected layer” (e.g., within an integrated circuit) may be represented by one or more operations that multiply one input vector by a matrix of weights to yield one output vector. That matrix-vector multiplication corresponds to the operations performed by an array of memory elements. In such embodiments, the conversion of the computational graph to a circuit netlist includes, via a rules-based analysis, producing a representation of one or more circuit blocks based on arithmetic operations, including, but not limited or restricted to, vector-matrix multiplication operations that are representative of a machine learning logic module; [0041] Referring now to FIG. 4, a block diagram illustrating an exemplary computational graph, in the form of a long-short term memory (LSTM) circuit is shown. The LSTM circuit diagram 400 includes the following components: the matrix-vector operators 404 1-404 4; the sigmoid vector operators 408 1-408 4; the element-wise multiplier 412; the element-wise multiplier 420; the element-wise summation operator 424; the unit time delay element 416; the sigmoid vector operator 428; the element-wise multiplier 432; and the unit time delay element 436. The LSTM circuit diagram 400 receives the X-dimensional input vector 402, which is provided to the matrix-vector operators 404 1-404 4. Following the operations of the matrix-vector operators 404 1-404 4, output vectors 406 1-406 4 are generated. The matrix-vector operators 404 1-404 4 each apply a weight to the input vector 402, wherein one or more of the weights may differ from the others);
transforming the neural network topology to an equivalent analog network of analog components, including replacing composition or superposition of linear transformations by a single linear transformation ([0025] At block 204, the behavioral description of the machine learning logic module is converted into a computational graph that represents a plurality of levels of operations. Of these operations, some may be referred to as “primitive operations” such as add, multiply, half-wave rectification (often referred to in machine learning parlance as a Rectified Linear Unit, or ReLU), leaky ReLU, sigmoid, hyperbolic tangent function (“tanh”), winner-take-all (the output equal to the largest of multiple inputs), loser-take-all block (the output is equal to the smallest of several inputs), arg-max (the output indicates which of multiple inputs is the largest); arg-min (the output indicates which of multiple inputs is the smallest), averaging, exponentiation, logarithm, etc. For these primitive operations, there are corresponding analog circuits that implement the primitive operations in the analog circuit library. Further, complex operations may be comprised of a plurality of primitive operations, wherein examples of complex operations may include, but are not limited or restricted to, fully-connected layers, convolutions, matrix-vector multiplication, concatenation, vector addition, matrix inversion, LSTM layers, recurrent connections, gating units, delay operations, pooling operations, temporal convolutions, embedding layers, merge layers, and normalization. The primitive operations and complex operations may be referred to as “known neural network operations.” Additional operations may be constructed from a combination of complex operations and primitive operations. In one embodiment, a software algorithm called TensorFlow may be used to convert the architecture of the machine learning logic module to the computational graph; [0027] At block 206, the computational graph is converted into a circuit netlist based on a correlation of (i) operations described in the computational graph with (ii) a predetermined analog cell library representing computational operations. In some embodiments, the computational graph is comprised of a plurality of elements wherein each element corresponds directly to one or more circuit blocks within a circuit layout. The term “circuit block” refers to predetermined circuits that are part of a custom library. Each circuit block may have a fixed size or may be parametric. A parametric circuit block has a dynamic size that is defined by one or more parameters. For example, a circuit block, e.g., a memory array, might be instantiated with size parameters as follows, e.g.; “memory_array(rows=8, columns=16).” Such an example would result in an 8×16 array of core memory elements having eight input circuits with one input connected to each row and 16 output interface circuits with one connected to each column);
computing a weight matrix for the equivalent analog network based on the weights of the trained neural network, wherein each element of the weight matrix represents a respective connection between analog components of the equivalent analog network ([0028] In one embodiment, a “fully connected layer” (e.g., within an integrated circuit) may be represented by one or more operations that multiply one input vector by a matrix of weights to yield one output vector. That matrix-vector multiplication corresponds to the operations performed by an array of memory elements. In such embodiments, the conversion of the computational graph to a circuit netlist includes, via a rules-based analysis, producing a representation of one or more circuit blocks based on arithmetic operations, including, but not limited or restricted to, vector-matrix multiplication operations that are representative of a machine learning logic module. In one such embodiment, a circuit element comprising a plurality of multipliers and summation circuitry and implementing a matrix-vector multiplication with parameters defining the number of inputs and outputs may exist within a pre-defined library, e.g., the analog cell library 504 of FIG. 5. In that case, the neuromorphic synthesizer system 500 of FIG. 5 might instantiate the pre-defined matrix-vector multiplication circuit with the appropriate parameters; [0029] In a similar embodiment wherein a fully-connected layer is desired, and a complete matrix-vector multiplication circuit is not defined in the analog circuit library, but multiplication and summation elements do exist within the pre-defined library, the synthesizer might construct a matrix-vector multiplication circuit from the multiplication and summation elements according to a set of rules; [0031] an element in the computational graph may correspond to an entity that is to be generated using one or more elements within a circuit library. In such embodiments, representations of software routines will be stored within the circuit library, wherein each of the software routines expresses a given type of computational element (e.g., a convolution, or a recurrent layer) as a collection of circuit elements from a predefined and pre-populated analog cell library and the connections between those elements. In one embodiment, the analog cell library, stored in the non-transitory computer-readable as illustrated in FIG. 5, may include schematic representations of one or more elements as well as predefined circuit layouts, and/or physical implementation of each element); and
generating a schematic model for implementing the equivalent analog network based on the weight matrix ([0034] At block 208, a layout of a circuit, e.g., an IC, that corresponds to the machine learning logic module is generated by reformatting the syntax of the circuit netlist to represent a circuit layout. At optional block 210, an IC is generated using the layout of the circuit generated in block 208), including selecting component parameter values for the analog components ([0028] In one embodiment, a “fully connected layer” (e.g., within an integrated circuit) may be represented by one or more operations that multiply one input vector by a matrix of weights to yield one output vector. That matrix-vector multiplication corresponds to the operations performed by an array of memory elements. In such embodiments, the conversion of the computational graph to a circuit netlist includes, via a rules-based analysis, producing a representation of one or more circuit blocks based on arithmetic operations, including, but not limited or restricted to, vector-matrix multiplication operations that are representative of a machine learning logic module. In one such embodiment, a circuit element comprising a plurality of multipliers and summation circuitry and implementing a matrix-vector multiplication with parameters defining the number of inputs and outputs may exist within a pre-defined library, e.g., the analog cell library 504 of FIG. 5. In that case, the neuromorphic synthesizer system 500 of FIG. 5 might instantiate the pre-defined matrix-vector multiplication circuit with the appropriate parameters; [0031] an element in the computational graph may correspond to an entity that is to be generated using one or more elements within a circuit library. In such embodiments, representations of software routines will be stored within the circuit library, wherein each of the software routines expresses a given type of computational element (e.g., a convolution, or a recurrent layer) as a collection of circuit elements from a predefined and pre-populated analog cell library and the connections between those elements. In one embodiment, the analog cell library, stored in the non-transitory computer-readable as illustrated in FIG. 5, may include schematic representations of one or more elements as well as predefined circuit layouts, and/or physical implementation of each element).
Regarding dependent claim 6, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp further teaches wherein transforming the neural network topology into the equivalent analog network comprises transforming a linear transformation followed by another linear transformation into a single linear transformation ([0025] complex operations may be comprised of a plurality of primitive operations, wherein examples of complex operations may include, but are not limited or restricted to, fully-connected layers, convolutions, matrix-vector multiplication, concatenation, vector addition, matrix inversion, LSTM layers, recurrent connections, gating units, delay operations, pooling operations, temporal convolutions, embedding layers, merge layers, and normalization. The primitive operations and complex operations may be referred to as “known neural network operations.” Additional operations may be constructed from a combination of complex operations and primitive operations. In one embodiment, a software algorithm called TensorFlow may be used to convert the architecture of the machine learning logic module to the computational graph).
Regarding dependent claim 16, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp further teaches further comprising:
quantizing and/or restricting the weights of the neural network topology ([0020] While the initially fabricated neuromorphic IC 102 can include an initial firmware with custom synaptic weights between the nodes, the initial firmware can be updated as needed by the cloud 130 to adjust the weights).
Regarding dependent claim 17, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp further teaches further comprising: identifying non-linear elements in the neural network topology ([0032] In one non-limiting example, a long-short term memory (LSTM) is a type of layer often used in neural networks, that generally takes the form of the circuit shown in FIG. 4 (discussed below). The routine for a LSTM circuit would take parameters relating to the size of the inputs and outputs and the type of the nonlinearities and generates circuit netlist lines that lists each of the time delays, adders, multipliers, etc. and the connections between them).
Regarding independent claim 19, it is a system claim that corresponding to the method of claim 1. Therefore, it is rejected for the same reason as claim 1 above. Vorenkamp further teaches a system for hardware realization of neural networks (Fig. 5, 500; [0044]), comprising:
one or more processors (Fig. 5, 501; [0044]); and
memory (Fig. 5, 502; [0044]);
wherein the memory stores one or more programs configured for execution by the one or more processors, and the one or more programs comprising instructions ([0015]; [0044]).
Regarding independent claim 20, it is a medium claim that corresponding to the method of claim 1. Therefore, it is rejected for the same reason as claim 1 above. Vorenkamp further teaches a non-transitory computer readable storage medium storing one or more programs configured for execution by a computer system having one or more processors, the one or more programs comprising instructions (Fig. 5, 502; [0044]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-4 are rejected under 35 U.S.C. 103 as being unpatentable over Vorenkamp as applied in claim 1, in view of BELZER et al. (hereinafter BELZER), US 20200389188 A1.
Regarding dependent claim 2, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach
prior to transforming the neural network topology to the equivalent analog network:
adding regularizers to the neural network topology to reduce nominal values of the weights or to a respective reduce weight sum for each neuron; and
retraining the trained neural network to obtain updated weights for the weight matrix.
However, in the same field of endeavor, BELZER teaches prior to transforming the neural network topology to the equivalent analog network:
adding regularizers to the neural network topology to reduce nominal values of the weights or to a respective reduce weight sum for each neuron ( [0061] The weights w1 through w3 and magnitude limits T1 through T3 are optimized in order to minimize the BER at the channel decoder 14's output after all iterations are completed; [0074] The output oj of a given output layer's node j is computed from its hidden layer node inputs xi as oj=ƒ(zj), where zj=Σiwijxi+bj, and the wij and bj are trainable weights and a trainable bias term, respectively. The function ƒ(z) is the rectified linear unit (ReLU) function ƒ(z)=max (0, z); [0086] Every convolutional unit includes 3 layers: convolutional layer, batch normalization layer, and ReLU layer. Two of these convolutional units are shown in FIG. 5A, Convolutional Unit #1 502 and Convolutional Unit #5 503. The convolutional layer slides the filter over the input data, and the batch normalization layer normalizes the data to speed up network training and reduce sensitivity to the initial conditions (of the filter coefficients and interconnection weights) in the layers); and
retraining the trained neural network to obtain updated weights for the weight matrix ([0109] The DNN implemented herein for the DNN APP detector 702 of the system 700 consists of several layers. Interconnections between the layers are defined by functions. Parameters that specify these functions in the DNN include weights, biases, offsets and scales. Among these parameters, some are specified prior to training called as hyperparameters. The rest of the parameters are learnable parameters that are learned through training).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of training and optimizing weights and the magnitude limits as suggested in BELZER into Vorenkamp’s system because both of these systems are addressing multilayer neural network. This modification would have been motivated by the desire to optimize the learnable parameters so that they provide an accurate description of the input-output relationship between the bottom (input) layer and the top (output) layer of the DNN (BELZER, [0109]).
Regarding dependent claim 3, the combination of Vorenkamp and BELZER teaches all the limitations as set forth in the rejection of claim 2 that is incorporated. BELZER further teaches wherein the regularizers include a respective predetermined regularizer for each convolution batch normalization ReLU block, ([0086] Every convolutional unit includes 3 layers: convolutional layer, batch normalization layer, and ReLU layer. Two of these convolutional units are shown in FIG. 5A, Convolutional Unit #1 502 and Convolutional Unit #5 503. The convolutional layer slides the filter over the input data, and the batch normalization layer normalizes the data to speed up network training and reduce sensitivity to the initial conditions (of the filter coefficients and interconnection weights) in the layers; [0122] The convolutional layer is followed by a batch normalization layer to normalize the convolutional layer's output across a mini batch … This normalization process speeds up the training, reduces the network's sensitivity to initialization and increases the network's stability. Similar to FCDNN, the last layer component in a convolutional stage is the ReLU layer which activates the network. The batch normalization layer and the ReLU layer do not change the size of their input … Overall, the learnable parameters of the CNN include the coefficients of the filter banks in the convolutional layers, the parameters in the batch normalization layers and the weights and biases in the fully connected layers in the output stage) and wherein each predetermined regularizer treats each batch normalization layer as a normalization and calculates combined convolution-batch normalization multipliers applied to an input neural network signal in the signal's propagation path, and reduces the absolute value of combined weights for each neuron ([0060] After the inverse interleaver 10, the LLRs are multiplied using a multiplier 12 by a weight w1>0 and then limited in magnitude (while preserving their signs) to a value T1. The magnitude limiting prevents the LLRs from becoming too large and causing numerical overflow or underflow in the channel decoder 14. The multiplicative weight w1 is conventionally set to be less than or equal to 1.0. Setting this value to less than 1.0 is valuable if multiple iterations are done between the channel decoder 14 and the trellis detector 6, because down-weighting the LLRs slows the convergence of the iterative algorithm and results in a lower BER once convergence occurs; [0061] The decoded LLR estimate LLR1 of the channel decoder 14 is multiplied by weight w3 and magnitude limited to value T3 by multiplier 32. This magnitude limited value is interleaved using an interleaver 34 denoted as π before it enters the trellis detector 6. Then after the desired number of iterations when the BER is converged, the decoded LLR estimates LLR1 can either be compared with zero to generate a final decision, or it can be sent back to the DNN predictor 8 for a second DNN iteration. To send back the channel decoder 14's LLR estimates back to the DNN for a second iteration, the LLR estimate is multiplied by weight w2 and magnitude limited to value T2 by multiplier 16 and then interleaved by an interleaver 18 denoted by π before the LLR estimates enter the DNN media noise predictor 8. The weights w1 through w3 and magnitude limits T1 through T3 are optimized in order to minimize the BER at the channel decoder 14's output after all iterations are completed; [0068] The LDPC channel decoder 44 in the system 200 in FIG. 1C, is Nt channel decoders for TDMR architecture, one per track. (note that interleaver 48, inverse interleaver 40, multipliers 42, 46, 64, and respective weights w1, w2, w3 operate as similarly discussed above for FIG. 1B and are not discussed in detail for this example for simplicity) cannel decoding is done per track herein. Each of the Nt tracks are allowed to have a different channel coding rate, under the assumption that each track has a header which specifies the code rate. In addition, it is to be noted that the weights w1, w2 and w3 are vectors of Nt weights, with one weight per each of the Nt tracks, to allow for different LLR statistics on the different tracks. The magnitude limitation values T1, T2 and T3 are also vectors of Nt magnitude limits. In this way, the code rate of the Nt tracks for channel decoding, the weights and the magnitude limits are optimized over all Nt tracks, and a higher overall areal density is achieved than using a single code rate for all Nt tracks, or single weight and magnitude limit values for all Nt tracks).
Regarding dependent claim 4, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach wherein transforming the neural network topology into the equivalent analog network comprises translating weights of each batch normalization layer to weights of its previous layer.
However, in the same field of endeavor, BELZER teaches wherein transforming the neural network topology into the equivalent analog network comprises translating weights of each batch normalization layer to weights of its previous layer ([0122] The convolutional layer is followed by a batch normalization layer to normalize the convolutional layer's output across a mini batch, i.e., {circumflex over (x)}.sub.i=(x.sub.i−μ.sub.B)/√{square root over (σ.sub.B.sup.2+∈)}, y.sub.i=γ{circumflex over (x)}.sub.i+β, where μ.sub.B is the mini-batch mean, σ.sub.B is the mini-batch standard deviation, ∈ is the small denominator offset for numerical stability and β, γ are learnable offset and scale factors that are optimized during the training. This normalization process speeds up the training, reduces the network's sensitivity to initialization and increases the network's stability. Similar to FCDNN, the last layer component in a convolutional stage is the ReLU layer which activates the network. The batch normalization layer and the ReLU layer do not change the size of their input. The output size of each convolutional stage is 3×15×N.sub.ƒ, where N.sub.ƒ is the number of filter banks at that convolutional stage. For the choice of N.sub.ƒ herein, the dimensionality of the output at each convolutional stage is rather low and no max pooling layer is employed for down sampling. The last functional stage, i.e., the output stage of the CNN is identical to FCDNN. Overall, the learnable parameters of the CNN include the coefficients of the filter banks in the convolutional layers, the parameters in the batch normalization layers and the weights and biases in the fully connected layers in the output stage).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of normalizing the convolutional layer's output across a mini batch as suggested in BELZER into Vorenkamp’s system because both of these systems are addressing multilayer neural network. This modification would have been motivated by the desire for a normalization process which speeds up the training, reduces the network's sensitivity to initialization and increases the network's stability (BELZER, [0122]).
Claims 5 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Vorenkamp as applied in claim 1, in view of SU et al. (hereinafter SU), US 20210383233 A1.
Regarding dependent claim 5, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach wherein transforming the neural network topology into the equivalent analog network comprises merging layers that do not have an activation function.
However, in the same field of endeavor, SU teaches wherein transforming the neural network topology into the equivalent analog network comprises merging layers that do not have an activation function ([0036] since the output of the enlarged fully connected layer has no activation function, the enlarged fully connected layer and the reduced fully connected layer may be transformed into a miniaturized fully connected layer with an equivalent input and output of the second data processing capacity N. As an example, the enlarged fully connected layer and the reduced fully connected layer in the training student model after the distilling are merged into the third intermediate fully connected layer).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of merging the enlarged fully connected layer and the reduced fully connected layer when the output of the enlarged fully connected layer has no activation function as suggested in SU into Vorenkamp’s system because both of these systems are addressing training and distilling model. This modification would have been motivated by the desire to improve the efficiency and the overall effect of the training process (SU, [0004]).
Regarding dependent claim 14, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach wherein transforming the neural network topology to the equivalent analog network comprises introducing additional intermediate layers that limit a number of input or output links of neurons by splitting inputs or outputs of the neurons.
However, in the same field of endeavor, SU teaches wherein transforming the neural network topology to the equivalent analog network comprises introducing additional intermediate layers that limit a number of input or output links of neurons by splitting inputs or outputs of the neurons (Fig. 2; [0029] In order to speed up model prediction, in embodiments of the disclosure, as illustrated in FIG. 2, which is a schematic diagram of Embodiment 2 of the disclosure, the enlarged fully connected layer and the reduced fully connected layer that are split during training and that are of the training student model may be merged in a prediction stage to make the scale of the fully connected layer smaller. The specific implementation process is as follows; [0030] At block 201, a teacher model and a student model are obtained. The teacher model has a first intermediate fully connected layer. The student model has a second intermediate fully connected layer. An input of the first intermediate fully connected layer is a first data processing capacity M. An output of the first intermediate fully connected layer is the first data processing capacity M. An input of the second intermediate fully connected layer is a second data processing capacity N. An output of the second intermediate fully connected layer is the second data processing capacity N. M and N are positive integers. M is greater than N; [0031] At block 202, the second intermediate fully connected layer is transformed, based on the first data processing capacity M and the second data processing capacity N, into an enlarged fully connected layer and a reduced fully connected layer. The second intermediate fully connected layer is replaced with the enlarged fully connected layer and the reduced fully connected layer to generate a training student model; [0032] At block 203, the training student model is distilled based on the teacher model; [0033] In embodiments of the disclosure, description of blocks 201-203 may be referred to description of blocks 101-103 of embodiments illustrated in FIG. 1, and will not be repeated herein; [0034] At block 204, the training student model after the distilling is transformed to generate a prediction model; [0035] In some embodiments, the enlarged fully connected layer and the reduced fully connected layer in the training student model after the distilling are merged into a third intermediate fully connected layer to generate the prediction model).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of splitting the enlarged fully connected layer and the reduced fully connected layer during training and merging in a prediction stage to make the scale of the fully connected layer smaller as suggested in SU into Vorenkamp’s system because both of these systems are addressing training and distilling model. This modification would have been motivated by the desire to improve the efficiency and the overall effect of the training process (SU, [0004]).
Claims 7-13 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Vorenkamp as applied in claim 1, in view of KATAEVA et al. (hereinafter KATAEVA), US 20200110991 A1.
Regarding dependent claim 7, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach wherein transforming the neural network topology into the equivalent analog network comprises transforming layers with ReLU into ReLU1.
However, in the same field of endeavor, KATAEVA teaches wherein transforming the neural network topology into the equivalent analog network comprises transforming layers with ReLU into ReLU1 ([0022] Signals flow in one direction from input layers to output layers via intermediate layers. Each of the input neurons transmits the input value to the intermediate neuron without processing the input value. Each of the intermediate neurons and the output neurons calculates a sum of values each obtained by multiplying output values of a plurality of neurons of a previous layer by a positive or negative weight indicating strength of connection, and calculates an output value from the calculated value of the sum using an activation function; Fig. 1; [0032] the convolution neural network 20 includes seven convolution layers constituted by a first convolution layer 21, a second convolution layer 22, a third convolution layer 23, a fourth convolution layer 24, a fifth convolution layer 25, a sixth convolution layer 26 and a seventh convolution layer 27; [0082] The ReLU used as the activation function in the present embodiment is a linear function in a positive range. Accordingly, correction of the weights for the filters and the bias inputs using the scaling factors does not adversely affect recognition performance; [0026] Following situation may occur when generation of output values in accordance with an activation function is performed in an analog region in an integrated circuit (analog processing circuit) constituted by CMOS elements. The situation described below is particularly likely to occur in the case of use of an activation function, like a ReLU, which does not specify an upper limit for output values and therefore has a tendency of producing higher output values of neurons in an upper layer).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of transmission of the signals between the neurons and adjustment output level of the neurons of each of the layers as suggested in KATAEVA into Vorenkamp’s system because both of these systems are addressing multilayer neural network. This modification would have been motivated by the desire to use ReLU as the activation function since these functions are simple and therefore produce advantages such as fast calculation, no gradient loss, and possible high sparsity (KATAEVA, [0024]).
Regarding dependent claim 8, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 7 that is incorporated. KATAEVA further teaches wherein transforming layers with ReLU into ReLU1 comprises maintaining normal operation of the trained neural network during the transformation by analyzing a passage of signals through the trained neural network and performing weight correction ([0066] the learned weights for the filter and for the bias input are not set for the memristor 53 of the crossbar circuit 44 without change, but are corrected such that the output levels of the respective output neurons become appropriate before use of the weights; Fig. 13; [0067] In S100, a method such as a gradient descent method and a back propagation method is initially applied to the convolution neural network 20 constituted on the computer, and magnitudes of weights for filters and weights for bias inputs of the convolution layers 21 to 27 are learned; [0068] After completion of the learning, output values from the output neurons 56 of the respective convolution layers 21 to 27 are detected in S110 as output values at the time of input of an image to the convolution neural network 20 which is constituted on the computer and including optimized weights for the filters and the bias inputs; [0069] In S120, the maximum output in the output values of the output neurons 56 of each of the convolution layers 21 to 27 is selected for each of the output neurons 56; [0070] In subsequent S130, a scaling factor is calculated for each of the convolution layers 21 to 27 based on the output maximum value determined in S120 and a maximum output range of the activation function processing circuit; [0079] In S140, the magnitudes of the weights for the filters and the weights for the bias inputs of the convolution layers 21 to 27 learned in S100 are corrected using the scaling factors of the respective layers calculated in S130).
Regarding dependent claim 9, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 8 that is incorporated. KATAEVA further teaches wherein performing weight correction comprises:
adjusting weights so as to restrict signals in the trained neural network below a physical limit ([0081] In the manner described above, the scaling factors of the respective layers are calculated, and the weights for the filters and the weights for the bias inputs of the respective layers are corrected using the calculated scaling factors. Accordingly, the output values of the output neurons 56 of the respective layers fall within the set maximum output range of the analog processing circuit with a higher possibility, and do not exceed the write threshold voltages of the memristors 53.).
Regarding dependent claim 10, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 8 that is incorporated. KATAEVA further teaches wherein performing weight correction comprises:
when weights of a layer N are divided by a factor, adjusting weights of layer N+1 by multiplying the weights by the factor ([0072] A first layer scaling factor for the first convolution layer 21 is calculated by following Equation 2, for example.
Scaling_factor_layer1=maximum neuron output_layer1/max_range (Equation 2); [0074] Subsequently, a second layer scaling factor for the second convolution layer 22 is calculated by following Equation 3, for example.
Scaling_factor_layer2=(maximum neuron output_layer2/max_range)/Scaling_factor_layer1 (Equation 3); [0077] Respective scaling factors for the third to seventh convolution layers 23 to 27 are similarly calculated by following Equation 4, for example, in consideration of the scaling factors of the lower layers.
Scaling_factor_layerN=(maximum neuron outpt_layerN/max_range)/(Scaling_factor_layer1* . . . *Scaling_factor_layerN−1) (Equation 4); [0079] After completion of calculation of the scaling factors for the respective layers, the process proceeds to S140. In S140, the magnitudes of the weights for the filters and the weights for the bias inputs of the convolution layers 21 to 27 learned in S100 are corrected using the scaling factors of the respective layers calculated in S130. Specifically, correction filter weights and correction bias input weights are calculated for each of the convolution layers 21 to 27 by dividing learned original weights for the filters and weights for the bias inputs by the corresponding scaling factors as expressed by following Equations 5 and 6.
W scaled_layerN =W original_layerN/Scaling_factor_layerN (Equation 5)
W_Biasscaled_layerN =W_Biasorigirial_layerN/(Scaling_factor_layerN*Scaling_factor_layer (N−1)* . . . *Scaling_factor_layer2*Scaling_factor_layer1) (Equation 6)).
Regarding dependent claim 11, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 8 that is incorporated. KATAEVA further teaches wherein performing weight correction comprises:
adjusting weights and bias of one or more neurons and adjustment of weights of outgoing connections of the one or more neurons ([0079] correction filter weights and correction bias input weights are calculated for each of the convolution layers 21 to 27 by dividing learned original weights for the filters and weights for the bias inputs by the corresponding scaling factors as expressed by following Equations 5 and 6.
W scaled_layerN =W original_layerN/Scaling_factor_layerN (Equation 5)
W_Biasscaled_layerN =W_Biasorigirial_layerN/(Scaling_factor_layerN*Scaling_factor_layer (N−1)* . . . *Scaling_factor_layer2*Scaling_factor_layer1) (Equation 6); [0081] In the manner described above, the scaling factors of the respective layers are calculated, and the weights for the filters and the weights for the bias inputs of the respective layers are corrected using the calculated scaling factors. Accordingly, the output values of the output neurons 56 of the respective layers fall within the set maximum output range of the analog processing circuit with a higher possibility, and do not exceed the write threshold voltages of the memristors 53).
Regarding dependent claim 12, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 8 that is incorporated. KATAEVA further teaches wherein performing weight correction comprises:
repeating weight correction for the trained neural network until complete compliance is achieved ([0081] In the manner described above, the scaling factors of the respective layers are calculated, and the weights for the filters and the weights for the bias inputs of the respective layers are corrected using the calculated scaling factors. Accordingly, the output values of the output neurons 56 of the respective layers fall within the set maximum output range of the analog processing circuit with a higher possibility, and do not exceed the write threshold voltages of the memristors 53).
Regarding dependent claim 13, the combination of Vorenkamp and KATAEVA teaches all the limitations as set forth in the rejection of claim 8 that is incorporated. KATAEVA further teaches wherein performing weight correction comprises scaling signals on layers with unlimited ReLU so that they do not exceed a physical limitation ([0082] The ReLU used as the activation function in the present embodiment is a linear function in a positive range. Accordingly, correction of the weights for the filters and the bias inputs using the scaling factors does not adversely affect recognition performance; [0071] The maximum output range of the activation function processing circuit is set such that the input voltage to the subsequent layer does not become equal to or higher than a write threshold voltage of the memristor 53 (e.g., 0.9 V) even when the output of the activation function processing circuit reaches the upper limit of the maximum output range).
Regarding dependent claim 18, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly further comprising (i) calculating a respective range of weights for each layer of the neural network topology and (ii) calculating a respective sum of the weights for each neuron of the neural network topology.
However, in the same field of endeavor, KATAEVA teaches further comprising (i) calculating a respective range of weights for each layer of the neural network topology ([0051] According to the present embodiment, the output bars 51 and 52 are connected to the non-inverting input terminal and the inverting input terminal of the differential operational amplifier 54, respectively. Accordingly, the convolution operation can be performed using not only positive weights but also negative weights as a filter. Specifically, when a certain input signal is multiplied by a positive weight, conductance of the memristor 53 provided between the input bar 50 and the output bar 51 connected to the non-inverting input terminal is set to a value higher by a positive weight to be set than conductance of the memristor 53 provided between the input bar 50 and the output bar 52 connected to the inverting input terminal. Conversely, when a certain input signal is multiplied by a negative weight, conductance of the memristor 53 provided between the input bar 50 and the output bar 52 connected to the inverting input terminal is set to a value higher by a negative weight to be set than conductance of the memristor 53 provided between the input bar 50 and the output bar 51 connected to the non-inverting input terminal; [0052] Accordingly, in the present embodiment, a weight corresponding to each of filters 1, 2, 3, and others used for the corresponding one off the convolution layers 21 to 27 is set for each pair of the two output bars 51 and 52, and for the memristor 53 between the corresponding pair of the output bars 51 and 52 and the input bar 50 as shown in FIG. 3) and (ii) calculating a respective sum of the weights for each neuron of the neural network topology ([0022] Each of the intermediate neurons and the output neurons calculates a sum of values each obtained by multiplying output values of a plurality of neurons of a previous layer by a positive or negative weight indicating strength of connection, and calculates an output value from the calculated value of the sum using an activation function).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of the convolution operation being performed using not only positive weights but also negative weights as a filter as suggested in KATAEVA into Vorenkamp’s system because both of these systems are addressing multilayer neural network. This modification would have been motivated by the desire a method for adjusting output level of a neuron in a multilayer neural network (KATAEVA, [0004]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Vorenkamp as applied in claim 1, in view of Pescianschi, US 20160283842 A1.
Regarding dependent claim 15, Vorenkamp teaches all the limitations as set forth in the rejection of claim 1 that is incorporated. Vorenkamp does not explicitly teach pruning at least some connections of the neural network topology.
However, in the same field of endeavor, Pescianschi teaches pruning at least some connections of the neural network topology ([0246] The p-net 100A can be additionally configured to remove any number of unused inputs 102, neurons 116, along with respective additional neuron outputs 117, and synapses 118 before, during, or after either initial training or supplementary training, of the p-net. Such ability to remove elements of the p-net 100A that are not being used is intended to simplify structure and modify operational parameters of the p-net, i.e., condense the p-net, without loss of the p-net's output quality).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of remove at least one of an input, a synapse, and a neuron before, during, or after training of the neural network as suggested in Pescianschi into Vorenkamp’s system because both of these systems are addressing multilayer neural network. This modification would have been motivated by the desire to simplify structure and modify operational parameters of the neural network without loss of the network's output quality. (Pescianschi, [0013]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider these references fully when responding to this action.
Venkataramani et al. (US 20230099608 A1) discloses training an artificial neural network on an analog resistive processing unit system and, in particular, techniques for training convolutional neural networks on an analog resistive processing unit system using fast filtering functions such as, but not limited to, Winograd filtering functions.
It is noted that any citation to specific pages, columns, lines, or figures in the prior art references and any interpretation of the references should not be considered to be limiting in any way. A reference is relevant for all it contains and may be relied upon for all that it would have reasonably suggested to one having ordinary skill in the art. In re Heck, 699 F.2d 1331, 1332-33, 216 U.S.P.Q. 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006, 1009, 158 U.S.P.Q. 275, 277 (C.C.P.A. 1968)).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AMY P HOANG whose telephone number is (469)295-9134. The examiner can normally be reached M-TH 8:30-5:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, JENNIFER WELCH can be reached at 571-272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AMY P HOANG/ Examiner, Art Unit 2143
/JENNIFER N WELCH/ Supervisory Patent Examiner, Art Unit 2143
Prosecution Insights