DETAILED ACTION
The action is in response to the amendment filed on 9/25/2025. Claim 21 has been added. Claims 1-21 are pending and have been considered below. Claims 1, 16 and 20 are independent claims.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on January 4, 2022 and April 27, 2023 are being considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 16-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. The specification does not support that the relevance scores generated by the neural network are generated in the absence of or without input data to the neural network as recited in claim 16. The specification states that “Each relevance vector x can have m relevance scores, where m is the number of input features that the neural network model 120 is configured to receive (m is given by the number of input neurons in the neural network model) to perform a particular task.”(15). In other words, the relevance scores are tied to and calculated from the input a model is to receive. The relevance scores are generated using some type of input made to the model, which signifies that it cannot be in the absence of input data.
Claims 16-19 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention. The specification does not support that the relevance scores generated by the neural network are generated in the absence of or without input data to the neural network as recited in claim 16. The specification states that “Each relevance vector x can have m relevance scores, where m is the number of input features that the neural network model 120 is configured to receive (m is given by the number of input neurons in the neural network model) to perform a particular task.”(15). In other words, the relevance scores are tied to and calculated from the input a model is to receive. The relevance scores are generated using some type of input made to the model, which signifies that it cannot be in the absence of input data.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 16-19 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 16 recites “wherein the relevance scores are generated in the absence of input data to the neural network model”. It’s not clear how the relevance scores can be generated in the absence or without input to the neural network. The relevance scores are generated using some type of input made to the model, which signifies that it cannot be in the absence of input data. In this action, this limitation is being interpreted as being independent of the input features of the input vector.
Claim Rejections - 35 USC § 101
The rejection of claims 1-20 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more, has been withdrawn in light of the amendment, and arguments.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8 and 10-20 are rejected under 35 U.S.C. 103 as being unpatentable over Weber (“Towards a More Refined Training Process for Neural Networks: Applying Layer-wise Relevance Propagation to Understand and Improve Classification Performance on Imbalanced Datasets”) in view of Kursun (US 2021/0173905 A1) in further view of Lapuschkin (“Opening the Machine Learning Black Box with Layer-wise Relevance Propagation”).
Regarding claim 1, Weber teaches the neural network model comprising a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, a plurality of neuron biases, (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs; “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as where 𝑤𝑖𝑗 denotes the (learned) weight between 𝑥𝑖 and 𝑥𝑗. 𝑏𝑗 is the bias term of neuron 𝑥𝑗.”). The relevance scores are generated from the neurons, their weights and biases, and not from features of the input data or vector. and an input layer configured to receive an input vector with a plurality of input features; wherein generating the relevance scores is performed independently of the input features of the input vector
(Weber, Chapter 2 – Background, pp. 10, paragraph 4; “LRP is able to explain which parts of an input support or oppose a certain output and can even offer information about by which degree the various input features influence a decision,” thereby indicating a plurality of input features Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “The model f should be decomposable into different layers of computation, with the first being the input layer and the last being the classifier’s real-valued output. Then, the intermediate representation of data at layer l can be modeled as a vector, i.e.” Therefore, when l = 0, i.e., the input layer, z is an input vector.).
Weber does not explicitly teach receiving a request identifying a neural network model or in response to receiving the request. However, Kursun, in the area of explainable AI, teaches these limitations (Note that this claim is being interpreted as receiving a request identifying a neural network stored in memory in accordance with the embodiment disclosed at paragraph [0021] of the specification, “The request can include sufficient information to retrieve the neural network model from memory or storage device(s) or can include the neural network model.” Kursun, Fig. 3; Kursun, [0071]; “In one embodiment, the user application 238 may be configured to allow a user 102 to request and receive output from another system such as the misappropriation processing and alert system 130. In some embodiments, the memory device 234 may store information or data generated by the misappropriation processing and alert system 130 and/or by the processes described herein,” wherein “the misappropriation processing and alert system” comprises “memory device 306” housing “machine learning models 320,” or neural network[s].).
Weber and Kursun are analogous to the claimed invention as both are from the same field of endeavor, that is, explainable AI methods directed to modeling the relevance of individual nodes. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the neural network and layer-wise relevance propagation method of Weber with the memory calls of Kursun. The motivation to do so is inherent, that is a computer-implemented necessarily requires memory with which it can store its instructions.
Weber further teaches traversing the sequence of layers one or more times while generating a plurality of relevance scores each time based on the neuron weights and neuron biases, each relevance score of the plurality of relevance scores quantifying a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers; (“LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers, i.e.”
Here, the equality of the sums indicates how each relevance score of the plurality of relevance scores quantif[ies] a relevance of a neuron in a lower layer of the sequence of layers, 𝑅𝑖(𝑙), to a higher layer of the sequence of layers, 𝑅𝑖(𝑙+1).).
The combination of Weber and Kursun does not explicitly teach for global explanation, and generating a global explainability dataset comprising one or more relevance vectors populated with relevance scores from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and. However, Lapuschkin, in the area of layer-wise relevance propagation, teaches these limitations.
generating a global explainability dataset. (Lapuschkin, Figure 1.1; “LRP then decomposes the model’s evaluated prediction function into differential contributions for all input components, by performing a backward pass through the model. The resulting relevance map can then optionally be visualized for interpretation by a human observer,”, (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 4; “The application of LRP is not restricted to a model’s actual prediction. In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable,” wherein the ordinary meaning in the art for “map” in this context is a data structure comprising rows and columns thereby encompassing a global explanation or explainability dataset).
comprising one or more relevance vectors populated with relevance scores from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, pp. 28, paragraph 5; “Correspondingly, relevance values computed via LRP will also be available in vector form. For later visualization, the vectors of relevance scores can simply be reshaped into square images again,” wherein “vector form” indicates that the “relevance map” generated by LRP is itself one vector or comprises one or more relevance vectors populated with relevance scores. “The LeNet-5 directly receives as input square image matrices, zero padded to a (32 × 32 × 1) shaped input tensor with an additional channel axis,” wherein the “input tensor” refers to the input features whose relevance the “relevance map” is quantifying. In the context of an image classification task, the “relevance map” is quantifying a relevance of each input feature (in this case pixel) to the overall output. Lapuschkin, Fig. 2.6.).
Weber, Kursun and Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance.
Therefore, it would have been obvious, before the effective filing date of the claimed invention, to combine the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, with the relevance map of Lapuschkin. The motivation to use such a data structure would be to apply layer-wise propagation to image classification tasks in order to create heatmap visualization that highlight the importance of individual pixels (Lapuschkin, Figures 2.6-2.11 and 2.13. Lapuschkin, 2.4.3 Example Application on Toy Data with Different Kernels, pp. 41-42, paragraphs 8-1; “We apply LRP to predictions made by both models on the test image patches. Since test image patches are extracted from their respective source images with overlap, we first merge the pixel-wise relevance scores obtained for each patch at the original location of the patch by locally averaging overlapping relevance maps. After merging all relevance patches into a single, larger relevance map, color space mapping is applied as described in Section 2.3.3. The result is a single large heatmap visualization of the attributed relevance scores per model, as shown in Figure 2.13”).
Weber further teaches generating a global explanation of the neural network model based on the global explainability dataset (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “To visualize and quantify the decision making of a Neural Network (NN), we utilize the heatmaps generated by Layer-wise Relevance Propagation (LRP).” Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 12; “LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers.” Weber, Chapter 5: Validating LRP-Heatmaps as a Measure of Understanding, pp. 18, paragraph 1; “the heatmaps generated by LRP can offer insights into how a deep neural network (NN) arrives at its decisions.” Therefore, “the heatmaps generated by Layer-wise Relevance Propagation (LRP)” – because the algorithm “conserv[es] the relevance sum globally over all layers” of the neural network – act as global explanation[s] of the neural network model in accordance with the example embodiments listed at paragraph [0018] of the specification, “The interpretations unit 160 and/or explanations unit 170 can receive the global explainability dataset 140 and transform the dataset into percentages and/or ratios. These ratios can be converted into histograms, pie and/or line graphs, heatmaps, and the like to give further insights”(emphasis added).).
Regarding claim 2, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Weber further teaches wherein the sequence of layers comprises a plurality of
hidden layers and a last layer, wherein a last hidden layer of the plurality of hidden layers precedes the last layer, wherein a first hidden layer of the plurality of hidden layers succeeds the input layer, (Weber, Figure, 2.1; “Schematic of a simple Multilayer Perceptron (MLP). It is structured into multiple layers of neurons, including hidden layers (here one for simplicity), an input layer, and an output layer,” wherein a “Multilayer Perceptron” is a kind of neural network. The figure illustrates a single hidden layer that both precedes the last layer and succeeds the input layer with the caption adding that more hidden layers are possible. In this case one with ordinary skill in the art would recognize that a last hidden layer of the plurality of hidden layers would naturally precede the last layer and that a first hidden layer of the plurality of hidden layers would naturally succeed the input layer. Such would be the case in the models used by Weber.
Weber, Table A1; “This network consists of 16 layers, beginning with groups of convolutional layers with small kernel sizes (2 to 3) alternating with pooling operations, yielding a very deep model. Then follow two large dense layers, each consisting of 4096 hidden units,” wherein all layers that are not the input layer or output layer must be hidden layers.)
and wherein the sequence of layers is traversed in a reverse direction from the last layer to the first hidden layer (Weber Chapter 3: Layer-wise Relevance Propagation – An Overview,
pp. 11, paragraph 4; “Corresponding to the (forward) mappings 𝑧𝑖𝑗, the relevance 𝑅𝑖(𝑙) is expressed in terms of (backward) relevance messages 𝑅𝑖←𝑗(𝑙,𝑙+1)…The specific relevance propagation rule satisfying Equation (3.4) depends on the specific computational graph and the role of the components within it, however, it can be expressed in a general fashion as”
wherein the “propagation rule” of “(backward) relevance messages)” indicates the sequences of layers is traversed in a reverse direction from the last layer to the first hidden layer.).
Regarding claim 3, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 2 (and thus the rejection of claim 2 is incorporated).
Weber further teaches wherein generating the plurality of relevance scores comprises computing one or more relevance scores at the last layer based on neuron weights in the last layer and neuron biases in the last hidden layer (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “The model f should be decomposable into different layers of computation, with the first being the input layer and the last being the classifier’s real-valued output.” Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 12, paragraph 2; “𝑅𝑖(𝑙)” is then obtained by aggregating the incoming relevance messages: Equation (3.6) applies to all components except for those of the last layer, the relevances of which are initialized with the corresponding model outputs. The output of any given neuron is taught earlier in Weber with the following two equations:
When applying this equation to the final output, the weight term “𝑤𝑖𝑗(𝑙−1,𝑙)” represents a neuron weight in the last layer that connects a select neuron “i” in the last hidden layer “l – 1” to a select neuron “j” in the last layer “l – 1.” The bias term “𝑏𝑗(𝑙)” represents bias in the last layer, however the output term from the previous layer “𝑥𝑖(𝑙−1)” itself contains neuron biases in the last hidden layer “l – 1.” Using the two equations above we can restate Equation 2.2 equivalently as: 𝑥𝑗(𝑙)= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑥𝑖(𝑙−1)+ 𝑏𝑗(𝑙))= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(ℎ𝑖(𝑙−1))+ 𝑏𝑗(𝑙))= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(Σ𝑤𝑖𝑗(𝑙−2,𝑙−1)𝑥𝑖(𝑙−2)+ 𝑛𝑖=1𝑏𝑗(𝑙−1))+ 𝑏𝑗(𝑙))) wherein “𝑏𝑗(𝑙−1)” is a bias term that represents a neuron bias connected to the select neuron in the last hidden layer. This equation be further built out such that the final output of the model is represented by the activations of all the previous neurons in the model (i.e., with all the weights and biases from previous layers).)
Regarding claim 4, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 3 (and thus the rejection of claim 3 is incorporated). Weber further teaches wherein each relevance score computed at the last layer is a weighted linear combination of a weight term and a bias term, (Equations 2.1 and 2.2 from Weber can be applied to the output layer, say l, of the model to yield the final output: 𝑓(𝑥)=𝑥(𝑙)= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(Σ𝑤𝑖𝑗(𝑙−2,𝑙−1)𝑥𝑖(𝑙−2)+ 𝑛𝑖=1𝑏𝑗(𝑙−1))+ 𝑏𝑗(𝑙)) which amounts to a weighted linear combination of a weight term and a bias term as well as an activation function 𝑔𝑗(𝑙).) wherein the weight term comprises a neuron weight in the last layer that connects a select neuron in the last hidden layer to a select neuron in the last layer, (The weight term 𝑤𝑖𝑗(𝑙−1,𝑙) represents a neuron weight in the last layer that connects a select neuron i in the last hidden layer l – 1 to a select neuron j in the last layer l – 1.). (“𝑏𝑗(𝑙−1)” is the bias term that represents a neuron bias connected to the select neuron in the last hidden layer).) The weights and the biases scale the input of the neural network (pages4-5, fig.2.1)-- and wherein the bias term comprises a neuron bias connected to the select neuron in the last hidden layer, wherein the weight term is weighted by a first scalar scaling factor, and wherein the bias term is weighted by a second scalar scaling factor.
Regarding claim 5, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 3 (and thus the rejection of claim 3 is incorporated). Weber further teaches wherein generating the plurality of relevance scores further comprises computing one or more relevance scores at the plurality of hidden layers, wherein the one or more relevance scores are computed at each one of the hidden layers based on neuron weights in the each one of the hidden layers, relevance scores in the higher layer succeeding the each one of the hidden layers, and neuron biases in a lower layer preceding the each one of the hidden layers (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 2; “Under the assumption that each relevance score 𝑅𝑗(𝑙+1) for each component 𝑧𝑗(𝑙+1) at the following layer l + 1 is already known, LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers, i.e., Building out Equation 3.2, we have: Σ𝑅𝑎(0)=⋯=Σ𝑅𝑖(𝑙)𝑖= Σ𝑅𝑗(𝑙+1)𝑗𝑎=⋯=𝑓(𝑥) wherein 𝑅𝑎(0) represents a relevance score for a neuron at the input layer and f(x) represents the relevance score of the output layer, which as stated in the rejections of claims 3 and 4, is simply the final output of the model. Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 12, paragraph 2; “Equation (3.6) applies to all components except for those of the last layer, the relevances of which are initialized with the corresponding model outputs. Equation
(3.6) can be iteratively applied from output to input, yielding the relevances of all layers of the model.” Therefore, we can use Equation 3.6 to compute the relevance of a neuron j in a hidden layer “l” as: 𝑅𝑗(𝑙)= Σ𝑧𝑗𝑘𝑧𝑘𝑘𝑅𝑘(𝑙+1). Equations 3.3 and 3.12 of Weber give us 𝑧𝑗 and 𝑧𝑖𝑗, respectively. We can restate our equation as follows: 𝑅𝑗(𝑙)= Σ𝑥𝑗𝑤𝑗𝑘Σ𝑥𝑗𝑤𝑗𝑘𝑗𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1) wherein the term 𝑤𝑖𝑗 represents neuron weights in the each one of the hidden layers, the term 𝑅𝑘(𝑙+1)represents relevance scores in the higher layer succeeding the each one of the hidden layers and the term 𝑥𝑖 is necessarily computed using the bias term 𝑏𝑖 which represents neuron biases in a lower layer preceding the hidden layer l. This equivalence can be applied when calculating any hidden layer l in the neural network model in accordance with Equation 3.6; therefore, the term 𝑅𝑗(𝑙) represents one or more relevance scores at the plurality of hidden layers.)
Regarding claim 6, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 5 (and thus the rejection of claim 5 is incorporated).
Weber further teaches wherein each relevance score computed at the each one of
the hidden layers is a linear combination of a weighted relevance term and a bias term, (Equations 3.6, 3.3 and 3.12 from Weber can be applied to a hidden layer l to yield: 𝑅𝑗(𝑙)= Σ𝑥𝑗𝑤𝑗𝑘Σ𝑥𝑗𝑤𝑗𝑘𝑗𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1) which amounts to a linear combination of a weighted relevance term, (Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑅𝑘(𝑙+1), and a bias term, 𝑏𝑖, which can be taken from 𝑥𝑖.) wherein the weighted relevance term is based on the neuron weights in the each one of the hidden layers (The weight term 𝑤𝑖𝑗(𝑙−1,𝑙) represents neuron weights in the each one of the hidden layers (l + 1, l, l – 1, etc.) and the relevance scores in the higher layer succeeding the each one of the hidden layers, (𝑅𝑘(𝑙+1) represents the relevance scores in the higher layer, l + 1, succeeding the each one of the hidden layers, l.) and wherein the bias term is based on the neuron biases in the lower layer preceding the each one of the hidden layers. (𝑏𝑖 is the bias term used to calculate 𝑥𝑖, the output of a neuron at layer l – 1, that represents neuron biases in the lower layer, l - 1, preceding the each one of the hidden layers, l.).
Regarding claim 7, the combination the combination of Weber, Kursun and Lapuschkin teaches the method of claim 5 (and thus the rejection of claim 5 is incorporated).
Weber further teaches further comprising discarding the relevance scores computed at a higher layer succeeding the each one of the hidden layers after computing the relevance scores at the each one of the hidden layers (Weber, 2.3.3 Prediction Difference, CAM, and Layer-wise Relevance Propagation, pp. 10, paragraph 2; “However, here [using Layer-wise relevance propagation] the relevance score is obtained in a computationally efficient
way by decomposing the classification output consecutively into sums of feature relevances, until the input-level is reached and pixel-wise relevances are obtained. E.g., in the context of NNs, this decomposition is performed layer-wise backwards from output to input, costing O(1) in terms of standard NN backward passes,” wherein “decomposing the classification output consecutively into sums of feature relevances” encompasses discarding in the manner described at paragraph [0031] of the specification, “For example, once the relevance scores of a given layer L have been calculated using the relevance scores of a higher layer L+ 1, the relevance scores of the higher layer L + 1 can be discarded since they will not be needed to generate the relevance scores of the lower layer L - 1.” In other words, the relevance scores computed at higher layers are decomposed or discarded into the relevance scores at lower layers.).
Regarding claim 8, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 5 (and thus the rejection of claim 5 is incorporated).
The combination of Weber and Kursun does not explicitly teach wherein each of the one or more relevance vectors is populated with the one or more relevance scores computed at the first hidden layer during the corresponding time of traversing the sequence of layers.
However, Lapuschkin, in the area of layer-wise relevance propagation, teaches this limitation (Lapuschkin, 2.2.1 Conservation of Relevance across Layers of Computation, pp. 12, paragraph 7; “The first layer of the model to decompose is the input layer, e.g. receiving as inputs pixels from image, documents of text or other data, and the last layer of the model f is the real-valued prediction output. We assume that the intermediate representation of the data at the l-th layer is modeled as vector (or matrix, tensor, or single neuron) 𝒛=(𝑧𝑖(1))∈ℝ𝑑𝑖𝑚(𝑙). LRP assumes that we have a relevance score 𝑅𝑗(𝑙+1) for each component 𝑧𝑗(𝑙+1) of the representation 𝒛 at layer (𝑙+1). The objective is to compute relevance scores 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙)at layer (l), which is one “step” closer to the input layer, such that the following equation holds.” Therefore, in computing the relevance score “𝑅𝑖(𝑙)” for every neuron “i” at every layer “l,” the “vector” serving as “the intermediate representation of the data” must necessarily include one or more relevance scores computed at the first hidden layer.).
Weber, Kursun and Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to combine the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, with the relevance score vectors of Lapuschkin. The motivation to use vectors as intermediate representations of the relevance values is that they can be easily transformed into heatmap visualizations that highlight the importance of individual pixels (Lapuschkin, Figures 2.6-2.11 and 2.13. Lapuschkin, 2.4.3 Example Application on Toy Data with Different Kernels, pp. 41-42, paragraphs 8-1; “We apply LRP to predictions made by both models on the test image patches. Since test image patches are extracted from their respective source images with overlap, we first merge the pixel-wise relevance scores obtained for each patch at the original location of the patch by locally averaging overlapping relevance maps. After merging all relevance patches into a single, larger relevance map, color space mapping is applied as described in Section 2.3.3. The result is a single large heatmap visualization of the attributed relevance scores per model, as shown in Figure 2.13”).
Regarding claim 10, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 2 (and thus the rejection of claim 2 is incorporated).
Weber further teaches wherein the last layer comprises a plurality of neurons (Weber, Table A.1; The last layer “dense_3” has a “shape” of 8 thus denoting a plurality of neurons).
The combination of Weber and Kursun does not explicitly teach wherein the sequence of layers is traversed a plurality of times corresponding to the plurality of neurons and wherein the global explainability dataset comprises a plurality of relevance vectors corresponding to the plurality of neurons in the last layer. However, Lapuschkin, in the area of layer-wise relevance propagation, teaches these limitations (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 4; “The application of LRP is not restricted to a model’s actual prediction. In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable,” wherein creating “multiple relevance maps” entails travers[ing] a plurality of times corresponding to the plurality of neurons in the last layer, or each “class output.” Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 31, paragraph 4; “In some cases selecting multiple output neurons for
relevance back propagation instead of single neurons can be beneficial… In such cases, the simultaneous relevance backwards pass for multiple neurons will yield a weighted combination of relevance responses from all output classes of interest,” wherein “simultaneous relevance backwards pass[es]” constitute multiple traversals corresponding to the plurality of neurons in the last layer. Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 31, paragraphs 5-6; “In general, however, explaining all output neurons representing f(x) at once will yield nonsensical and unspecific relevance maps. Figure 2.8 demonstrates the relevance map for input digit digit “7” from Figure 2.6 as a simultaneous response from all output neurons at the same time. The figure also demonstrates that the relevance response wrt [with respect to] all output neurons at once is equal to the sum over relevance maps wrt to all individual output neurons due to the relevance conservation property.” Therefore, the plurality of relevance vectors, or “relevance maps,” corresponding to the plurality of neurons in the last layer are combined in Figure 2.8 to yield a single relevance map or global explainability dataset.)
Weber, Kursun and Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to apply the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, to multi-class settings, as taught by Lapuschkin. The motivation to do so is to generate relevance information for all possible classes that may allow a user to deduce the network’s reasoning when arriving at a prediction (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 5; “Computing relevance maps for non-dominantly firing output neurons can reveal interesting information about the learned rejection strategy of the model, why a certain class i.e. has been not picked for prediction.”).
Regarding claim 11, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 10 (and thus the rejection of claim 10 is incorporated).
The combination of Weber and Kursun does not explicitly teach wherein each of the plurality of relevance vectors is populated using a subset of the plurality of relevance scores generated during the corresponding time of traversing the sequence of layers. However, Lapuschkin, in the area of layer-wise relevance propagation, teaches this limitation (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 4; “In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable,” wherein “multiple relevance maps” denote a plurality of relevance vectors that are populated using a subset of relevance scores. Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 31, paragraph 4; “In some cases selecting multiple output neurons for relevance back propagation instead of single neurons can be beneficial… In such cases, the simultaneous relevance backwards pass for multiple neurons will yield a weighted combination
of relevance responses from all output classes of interest,” wherein the “simultaneous relevance backwards pass[es]” denote the corresponding time of traversing the sequence of layers.).
Weber, Kursun and Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to apply the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, to multi-class settings, as taught by Lapuschkin. The motivation to do so is to generate relevance information for all possible classes that may allow a user to deduce the network’s reasoning when arriving at a prediction (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 5; “Computing relevance maps for non-dominantly firing output neurons can reveal interesting information about the learned rejection strategy of the model, why a certain class i.e. has been not picked for prediction.”).
Regarding claim 12, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Weber further teaches wherein the neural network model is a trained neural
network model, (Weber, 4.1 Dataset and Model, pp. 15, paragraph 3; “The training of this network is done in a batch-wise setup, as shown schematically in Figure 4.1: We partition the training data into larger data-batches, and the network is trained on an entire data-batch before moving on to the next one,” thereby denoting a trained neural network model.) and further comprising retraining the neural network model based at least in part on the global explanation (Weber, 6.1.1 Setup and Method, pp. 36; paragraph 2; “For this experiment, the
division of the training process into batch-epochs is especially important, as it allows for data resampling on the larger data-batches,” wherein each “batch-epoch” constitutes retraining the neural network. Weber, 6.1.1 Setup and Method, Entropy of LRP-Heatmaps, pp. 36-37, paragraphs 4-1; “After the network has completed the training of an arbitrary batch-epoch 𝑡 ∈ {1,...,𝑇}, we obtain the class-wise weights 𝑤𝑐𝑡+1∈]0,1[… Given a series of LRP-heatmaps (𝑅𝑖,𝑐𝑖𝑡) for samples 𝑖 ∈ {1,...𝐼} w.r.t. the predicted class 𝑐𝑖𝑡 of the respective samples each, we calculate the classwise weights by first applying Equation (5.2) to every LRP-heatmap (𝑅𝑖,𝑐𝑖𝑦𝑡)…For each class c, the values in 𝐻𝑐𝑡 are then transformed into the interval ]0, 1[ by computing the L1-norm over all 𝐻𝑐𝑡 to yield the final class weights 𝑤𝑐𝑡+1 of the next batch-epoch,” thereby retraining the neural network with a new batch epoch whose weights are based at least in part on the global explanation, or “LRP-heatmap.”).
Regarding claim 14, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Weber does not explicitly teach further comprising storing the global explainability dataset in a data storage in association with the neural network model. However, Kursun, in the area of explainable AI, teaches this limitation (Kursun, Fig. 3; Kursun, [0079]; “Data stored in the data storage 308 may comprise a user information database 314, a historical interaction database 316, misappropriation database 318, and machine learning models 320.”).
Weber and Kursun are analogous to the claimed invention as both are from the same field of endeavor, that is, explainable AI methods directed to modeling the relevance of individual nodes. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the neural network and layer-wise relevance propagation method of Weber with the storage media of Kursun. The motivation to do so is inherent, that is a computer-implemented method necessarily requires memory with which it can store its instructions.
Regarding claim 15, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 1 (and thus the rejection of claim 1 is incorporated).
Weber further teaches wherein the plurality of relevance scores are generated each time without using neuron activations of the neurons in the sequence of layers (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2; “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as,”
wherein Equation 3.12 does not require an activation function. Weber, Equation 3.6;
A given relevance score “𝑅𝑖(𝑙)” of a plurality of relevance scores is calculated using terms defined in Equation 3.12 and relevance scores from higher layers. Therefore, the plurality of relevance scores are generated each time without using neuron activations of the neurons in the sequence of layers.).
Regarding claim 16, Weber teaches the neural network model comprising a plurality of neurons arranged in a sequence of layers, a plurality of neuron weights, a plurality of neuron biases, (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs; “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as where 𝑤𝑖𝑗 denotes the (learned) weight between 𝑥𝑖 and 𝑥𝑗. 𝑏𝑗 is the bias term of neuron 𝑥𝑗.”). The relevance scores are generated from the neurons, their weights and biases, and not from features of the input data--and an input layer configured to receive an input vector with a plurality of input features; wherein the relevance scores are generated in the absence of input data to the neural network model.
(Weber, Chapter 2 – Background, pp. 10, paragraph 4; “LRP is able to explain which parts of an input support or oppose a certain output and can even offer information about by which degree the various input features influence a decision,” thereby indicating a plurality of input features Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “The model f should be decomposable into different layers of computation, with the first being the input layer and the last being the classifier’s real-valued output. Then, the intermediate representation of data at layer l can be modeled as a vector, i.e.” Therefore, when l = 0, i.e., the input layer, z is an input vector.).
Weber does not explicitly teach one or more non-transitory computer-readable storage media storing computer-executable instructions for causing a computer system to perform operations, receiving a request identifying a neural network model or in response to receiving the request. However, Kursun, in the area of explainable AI, teaches these limitations.
One or more non-transitory computer-readable storage media storing computer-executable instructions for causing a computer system to perform operations (Kursun, Fig. 2; “Computer Readable Instructions 236.” Kursun, [0107]; “It will be understood that any suitable computer-readable medium may be utilized. The computer-readable medium may include, but is not limited to, a non-transitory computer-readable medium.”)
receiving a request identifying a neural network model…in response to receiving the request (Note that this claim is being interpreted as receiving a request identifying a neural network stored in memory in accordance with the embodiment disclosed at paragraph [0021] of the specification, “The request can include sufficient information to retrieve the neural network model from memory or storage device(s) or can include the neural network model.” Kursun, Fig. 3; Kursun, [0071]; “In one embodiment, the user application 238 may be configured to allow a user 102 to request and receive output from another system such as the misappropriation processing and alert system 130. In some embodiments, the memory device 234 may store information or data generated by the misappropriation processing and alert system 130 and/or by the processes described herein,” wherein “the misappropriation processing and alert system” comprises “memory device 306” housing “machine learning models 320,” or neural network[s].).
Weber and Kursun are analogous to the claimed invention as both are from the same field of endeavor, that is, explainable AI methods directed to modeling the relevance of individual nodes. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the neural network and layer-wise relevance propagation method of Weber with the storage media of Kursun. The motivation to do so is inherent, that is a computer system necessarily requires memory with which it can store its instructions.
Weber further teaches traversing the sequence of layers one or more times while generating a plurality of relevance scores each time based on the neuron weights and neuron biases, each relevance score of the plurality of relevance scores quantifying a relevance of a neuron in a lower layer of the sequence of layers to a higher layer of the sequence of layers; (“LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers, i.e.”
Here, the equality of the sums indicates how each relevance score of the plurality of relevance scores quantif[ies] a relevance of a neuron in a lower layer of the sequence of layers, 𝑅𝑖(𝑙), to a higher layer of the sequence of layers, 𝑅𝑖(𝑙+1).)
The combination of Weber and Kursun does not explicitly teach for global explanation, and generating a global explainability dataset comprising one or more relevance vectors populated with relevance scores from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and. However, Lapuschkin, in the area of layer-wise relevance propagation, teaches these limitations.
generating a global explainability dataset (Lapuschkin, Figure 1.1; “LRP then decomposes the model’s evaluated prediction function into differential contributions for all input components, by performing a backward pass through the model. The resulting relevance map can then optionally be visualized for interpretation by a human observer,” wherein the ordinary meaning in the art for “map” in this context is a data structure comprising rows and columns thereby encompassing for global explanation, and a global explainability dataset).
comprising one or more relevance vectors populated with relevance scores from the plurality of relevance scores generated at the one or more times, each of the relevance scores in each relevance vector quantifying a relevance of one of the input features to a task the neural network model is trained to perform; and (Lapuschkin, 2.3.3 Example Application on MNIST Data and Neural Networks, pp. 28, paragraph 5; “Correspondingly, relevance values computed via LRP will also be available in vector form. For later visualization, the vectors of relevance scores can simply be reshaped into square images again,” wherein “vector form” indicates that the “relevance map” generated by LRP is itself one vector or comprises one or more relevance vectors populated with relevance scores.)
Weber, Kursun and Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to combine the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, with the relevance map of Lapuschkin. The motivation to use such a data structure would be to apply layer-wise propagation to image classification tasks in order to create heatmap visualization that highlight the importance of individual pixels (Lapuschkin, Figures 2.6-2.11 and 2.13. Lapuschkin, 2.4.3 Example Application on Toy Data with Different Kernels, pp. 41-42, paragraphs 8-1; “We apply LRP to predictions made by both models on the test image patches. Since test image patches are extracted from their respective source images with overlap, we first merge the pixel-wise relevance scores obtained for each patch at the original location of the patch by locally averaging overlapping relevance maps. After merging all relevance patches into a single, larger relevance map, color space mapping is applied as described in Section 2.3.3. The result is a single large heatmap visualization of the attributed relevance scores per model, as shown in Figure 2.13”).
Weber further teaches generating a global explanation of the neural network model based on the one or more relevance vectors (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “To visualize and quantify the decision making of a Neural Network (NN), we utilize the heatmaps generated by Layer-wise Relevance Propagation (LRP).” Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 12; “LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers.” Weber, Chapter 5: Validating LRP-Heatmaps as a Measure of Understanding, pp. 18, paragraph 1; “the heatmaps generated by LRP can offer insights into how a deep neural network (NN) arrives at its decisions.” Therefore, “the heatmaps generated by Layer-wise Relevance Propagation (LRP)” – because the algorithm “conserv[es] the relevance sum globally over all layers” of the neural network – act as global explanation[s] of the neural network model in accordance with the example embodiments listed at paragraph [0018] of the specification, “The interpretations unit 160 and/or explanations unit 170 can receive the global explainability dataset 140 and transform the dataset into percentages and/or ratios. These ratios can be converted into histograms, pie and/or line graphs, heatmaps, and the like to give further insights”(emphasis added).).
Claim 17 is a non-transitory computer-readable storage media claim corresponding to the steps of claim 2 and is rejected for the same reasons as claim 2.
Regarding claim 18, the combination of Weber, Kursun and Lapuschkin teaches the one or more non-transitory computer-readable storage media (and thus the rejection of claim 17 is incorporated).
Weber further teaches computing one or more relevance scores at the last layer based on neuron weights in the last layer and neuron biases in the last hidden layer; (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 1; “The model f should be decomposable into different layers of computation, with the first being the input layer and the last being the classifier’s real-valued output.” Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 12, paragraph 2; “𝑅𝑖(𝑙)” is then obtained by aggregating the incoming relevance messages:
Equation (3.6) applies to all components except for those of the last layer, the relevances of which are initialized with the corresponding model outputs. The output of any given neuron is taught earlier in Weber with the following two equations:
When applying this equation to the last layer, the weight term “𝑤𝑖𝑗(𝑙−1,𝑙)” represents neuron weights in the last layer “l.” The bias term “𝑏𝑗(𝑙)” represents bias in the last layer, however the output term from the previous layer “𝑥𝑖(𝑙−1)” itself contains neuron biases in the last hidden
layer “l – 1.” Using the two equations above we can restate Equation 2.2 equivalently as: 𝑥𝑗(𝑙)= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑥𝑖(𝑙−1)+ 𝑏𝑗(𝑙))= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(ℎ𝑖(𝑙−1))+ 𝑏𝑗(𝑙))= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(Σ𝑤𝑖𝑗(𝑙−2,𝑙−1)𝑥𝑖(𝑙−2)+ 𝑛𝑖=1𝑏𝑗(𝑙−1))+ 𝑏𝑗(𝑙))) wherein “𝑏𝑗(𝑙−1)” is a bias term that represents neuron biases in the last hidden layer. This equation be further built out such that the final output of the model is represented by the activations of all the previous neurons in the model (i.e., with all the weights and biases from previous layers).)
computing one or more relevance scores at each one of the hidden layers based on neuron weights in the each one of the hidden layers, relevance scores in the higher layer succeeding the each one of the hidden layers, and neuron biases in the lower layer preceding the each one of the hidden layers; and (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 11, paragraph 2; “Under the assumption that each relevance score 𝑅𝑗(𝑙+1) for each component 𝑧𝑗(𝑙+1) at the following layer l + 1 is already known, LRP then aims to compute the relevance score 𝑅𝑖(𝑙) for each component 𝑧𝑖(𝑙), while conserving the relevance sum globally over all layers, i.e.,
Building out Equation 3.2, we have: Σ𝑅𝑎(0)=⋯=Σ𝑅𝑖(𝑙)𝑖= Σ𝑅𝑗(𝑙+1)𝑗𝑎=⋯=𝑓(𝑥) wherein 𝑅𝑎(0) represents a relevance score for a neuron at the input layer and f(x) represents the relevance score of the output layer, which as stated in the rejections of claims 3 and 4, is simply the final output of the model. Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 12, paragraph 2; “Equation (3.6) applies to all components except for those of the last layer, the relevances of which are initialized with the corresponding model outputs. Equation (3.6) can be iteratively applied from output to input, yielding the relevances of all layers of the model.” Therefore, we can use Equation 3.6 to compute the relevance of a neuron j in a hidden layer “l” as: 𝑅𝑗(𝑙)= Σ𝑧𝑗𝑘𝑧𝑘𝑘𝑅𝑘(𝑙+1). Equations 3.3 and 3.12 of Weber give us 𝑧𝑗 and 𝑧𝑖𝑗, respectively.
We can restate our equation as follows: 𝑅𝑗(𝑙)= Σ𝑥𝑗𝑤𝑗𝑘Σ𝑥𝑗𝑤𝑗𝑘𝑗𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑧𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1)= Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘(Σ(Σ𝑥𝑖𝑤𝑖𝑗+𝑏𝑗𝑖)𝑤𝑗𝑘𝑗)𝑘𝑅𝑘(𝑙+1) wherein the term 𝑤𝑖𝑗 represents neuron weights in the each one of the hidden layers, the term 𝑅𝑘(𝑙+1)represents relevance scores in the higher layer succeeding the each one of the hidden layers and the term 𝑥𝑖 is necessarily computed using the bias term 𝑏𝑖 which represents neuron biases in the lower layer preceding the each one of the hidden layers. This equivalence can be applied when calculating any hidden layer l in the neural network model in accordance with Equation 3.6; therefore, the term 𝑅𝑗(𝑙) represents one or more relevance scores at the hidden layer l.)
after generating one or more relevance scores at each one of the hidden layers, discarding the one or more relevance scores generated at the higher layer succeeding the each one of the hidden layers. (Weber, 2.3.3 Prediction Difference, CAM, and Layer-wise Relevance Propagation, pp. 10, paragraph 2; “However, here [using Layer-wise relevance propagation] the relevance score is obtained in a computationally efficient way by decomposing the classification output consecutively into sums of feature relevances, until the input-level is reached and pixel-wise relevances are obtained. E.g., in the context of NNs, this decomposition is performed layer-wise backwards from output to input, costing O(1) in terms of standard NN backward passes,” wherein “decomposing the classification output consecutively into sums of feature relevances” encompasses discarding in the manner described at paragraph [0031] of the specification, “For example, once the relevance scores of a given layer L have been calculated using the relevance scores of a higher layer L+ 1, the relevance scores of the higher layer L + 1 can be discarded since they will not be needed to generate the relevance scores of the lower layer L - 1.” In other words, the relevance scores computed at higher layers are decomposed or discarded into relevance scores at lower layers.).
Regarding claim 19, the combination of Weber, Kursun and Lapuschkin teaches the one or more non-transitory computer-readable storage media of claim 16 (and thus the rejection of claim 16 is incorporated).
Weber further teaches wherein the operations further comprise retraining the neural network model based at least in part on the global explanation (Weber, 6.1.1 Setup and Method, pp. 36; paragraph 2; “For this experiment, the division of the training process into batch-epochs is especially important, as it allows for data resampling on the larger data-batches,” wherein each “batch-epoch” constitutes retraining the neural network. Weber, 6.1.1 Setup and Method, Entropy of LRP-Heatmaps, pp. 36-37, paragraphs 4-1; “After the network has completed the training of an arbitrary batch-epoch 𝑡 ∈ {1,...,𝑇}, we obtain the class-wise weights 𝑤𝑐𝑡+1∈]0,1[… Given a series of LRP-heatmaps (𝑅𝑖,𝑐𝑖𝑡) for samples 𝑖 ∈ {1,...𝐼} w.r.t. the predicted class 𝑐𝑖𝑡 of the respective samples each, we calculate the classwise weights by first
applying Equation (5.2) to every LRP-heatmap (𝑅𝑖,𝑐𝑖𝑦𝑡)…For each class c, the values in 𝐻𝑐𝑡 are then transformed into the interval ]0, 1[ by computing the L1-norm over all 𝐻𝑐𝑡 to yield the final class weights 𝑤𝑐𝑡+1 of the next batch-epoch,” thereby retraining the neural network with a new batch epoch whose weights are based at least in part on the global explanation, or “LRP-heatmap.”).
Regarding claim 20, Weber teaches “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as where 𝑤𝑖𝑗 denotes the (learned) weight between 𝑥𝑖 and 𝑥𝑗. 𝑏𝑗 is the bias term of neuron 𝑥𝑗.”( Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs;)-- wherein the relevance scores are computed as linear combinations of the neuron weights weighted by a first scalar scaling factor and the neuron biases weighted by a second scalar scaling factor.
Weber does not explicitly teach one or more processing units coupled to memory; one or more computer readable storage media storing instructions that when executed cause the computing system to perform operations comprising. However, Kursun, in the area of explainable AI, teaches these limitations.
one or more processing units coupled to memory; (Kursun, Fig. 2; “Processing Device 202.” Kursun, [0070]; “The user device 110 may generally include a processing device or processor 202 communicably coupled to devices such as, a memory device 234.”) one or more computer readable storage media storing instructions that when executed cause the computing system to perform operations comprising: (Kursun, Fig. 2; “Computer Readable Instructions 236.” Kursun, [0107]; “It will be understood that any suitable computer-readable medium may be utilized. The computer-readable medium may include, but is not limited to, a non-transitory computer-readable medium.”). Weber and Kursun are analogous to the claimed invention as both are from the same field of endeavor, that is, explainable AI methods directed to modeling the relevance of individual nodes. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the neural network and layer-wise relevance propagation method of Weber with the storage media of Kursun. The motivation to do so is inherent, that is a computing system necessarily requires memory with which it can store its instructions.
The remaining steps of claim 20 correspond to the steps of claim 1 and are rejected for the same reasons as claim 1.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Weber in view of Kursun, Lapuschkin and Guerrero-Gómez-Olmedo et al. (“LRP-Based path relevances for global explanation of deep architectures,” hereinafter Guerrero).
Regarding claim 9, the combination of Weber, Kursun and Lapuschkin teaches the method of claim 2 (and thus the rejection of claim 2 is incorporated).
…
The combination of Weber and Kursun does not explicitly teach and wherein one relevance vector is populated using a subset of the plurality of relevance scores generated during the one time of traversing the sequence of layers. However, Lapuschkin, in the area of layer-wise relevance propagation, teaches this limitation (Lapuschkin, Figure 1.1; “LRP then decomposes the model’s evaluated prediction function into differential contributions for all input components, by performing a backward pass through the model. The resulting relevance map can then optionally be visualized for interpretation by a human observer,” wherein “a backward pass” corresponds to the one time of traversing the sequence of layers and “the resulting relevance map” corresponds to one relevance vector that is populated using a subset of the plurality of relevance scores.).
Weber, Kursun, Lapuschkin are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI methods for calculating the feature relevance. Weber teaches a method of generating relevance scores during reverse traversals or backwards passes but does not specify that the number of traversals corresponds to the number of output neurons or predictions. Lapuschkin teaches this step. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to apply the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun, to single output settings specifically, as taught by Lapuschkin. The motivation to focus on a single output rather than multiple is that generating a single final relevance vector or relevance map yields an individual representation or visualization that can be interpreted easily and at once. Generating multiple outputs, while providing a greater breadth of information, yields representations that are difficult to interpret individually in the context of the final prediction (Lapuschkin, Figures 2.6 and 2.7; Figure 2.6 illustrates relevance maps for single predictions of a dominant output neuron that allow the user to view the relevance of each individual pixel as it relates to the output. Figure 2.7, however, offers a number of relevance maps with pixel relevances that fluctuate wildly, thus making it difficult for a user to interpret the neural network’s prediction globally.
Lapuschkin, 2.3.3 Example Application of MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 31, paragraph 5; “In general, however, explaining all output neurons representing f(x) at once will yield nonsensical and unspecific relevance maps. Figure 2.8 demonstrates the relevance map for input digit digit “7” from Figure 2.6 as a simultaneous response from all output neurons at the same time.”).
The combination of Weber, Kursun and Lapuschkin does not explicitly teach wherein the last layer comprises a single neuron, wherein the sequence of layers is traversed one time corresponding to the single neuron. However, Guerrero, in the area of calculating path relevances in neural networks using layer-wise relevance propagation, teaches these limitations (Guerrero, Figure 1; The figure illustrates an output layer that comprises a single neuron.
Guerrero, Table 1; The table depicts the architecture of a specific model used in the experiments of Guerrero, whose last layer also comprises a single neuron.
Guerrero, 2.1 Layer-wise relevance propagation, pp. 253, col. 2, paragraph 6; “Fig. 1 shows the flow from the input sample 𝑋𝑖 through the network up to obtain the prediction 𝑓(𝑋𝑖). Then, the relevance is distributed across each layer and sent back to the input,” thereby indicating that “each layer” in the sequence of layers is traversed one time corresponding to the single neuron output of the last layer, “𝑓(𝑋𝑖).”).
Weber, Kursun, Lapuschkin and Guerrero are analogous to the claimed invention as all are from the same field of endeavor, that is explainable AI. Therefore, it would have been obvious, before the effective filing date of the claimed invention, to modify the layer-wise relevance propagation algorithm of Weber, implemented with the added hardware components of Kursun and the relevance score vectors of Lapuschkin, for a neural network architecture with a single output neuron, as is taught by Guerrero. The motivation to do so is to visualize how relevant a given path of neurons through the layers is to the final output (Guerrero, Figure 2;
Guerrero, 3.2 Critical paths with TOPSIS, pp. 255, col. 1; paragraph 3; “We propose to compute all possible paths within the architecture and sort them using a multi-criteria algorithm, TOPSIS. This way, we can retrieve the most critical paths and the least critical ones as shown in Fig. 2.”).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Weber (“Towards a More Refined Training Process for Neural Networks: Applying Layer-wise Relevance Propagation to Understand and Improve Classification Performance on Imbalanced Datasets”) in view of Kursun (US 2021/0173905 A1) further view of Lapuschkin (“Opening the Machine Learning Black Box with Layer-wise Relevance Propagation”), and further in view of Totsuka et al (hereinafter Totsuka) USPGPUB 20230147985 (filed 2/10/2021).
Regarding claim 21, Weber teaches (Weber, Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs; “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as where 𝑤𝑖𝑗 denotes the (learned) weight between 𝑥𝑖 and 𝑥𝑗. 𝑏𝑗 is the bias term of neuron 𝑥𝑗.”). The relevance scores are generated from the neurons, their weights and biases, and not from features of the input data. Lapuschkin teaches “LRP then decomposes the model’s evaluated prediction function into differential contributions for all input components, by performing a backward pass through the model. The resulting relevance map can then optionally be visualized for interpretation by a human observer,”, “The application of LRP is not restricted to a model’s actual prediction. In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable,” -- the global explanation ( Figure 1.12.3.3 Example Application on MNIST Data and Neural Networks, Output Neuron Selection and Relevance Map Normalization, pp. 30, paragraph 4).
Weber does not explicitly teach, but Totsuka teaches receiving a reaction from a user in response to an explanation presented to the user [104]-- receiving a user response to the explanation. Totsuka also teaches retraining the trained DNN based on the submitted user’s reaction (233)-- and responsive to the user response, adjusting one or both of the first scalar scaling factor and the second scalar scaling factor to achieve an improved global explanation. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the teachings of Weber, the prior art, and Totsuka, because Totsuka teaches presenting an explanation about a DNN that is easily understood (63).
Response to Arguments
Applicant's arguments filed 9/25/2025 have been fully considered but they are not persuasive. Regarding the abstract idea rejection under 35 USC 101, the Applicant states that “While the claim does recite "relevance scores" and "vector" it does not attempt to preempt or claim a mathematical formula per se. The claim ties operations to a specific neural network structure: traversing layers, and using neuron weights and biases…” (p.11-12). The examiner agrees and is withdrawing the rejection.
In addition, the Applicant argues that “…As claimed, the technology improves functioning of the computer system itself. Explainability enables neural networks to be deployed more safely and effectively. These technical improvements are described at least in paragraphs [0079]-[0081] of the original specification…”(p.12-13). The examiner agrees and withdraws the rejection.
Additionally, the Applicant states that “Finally, under Step 2B, the claim has additional elements that are directed to non- conventional operations. As noted elsewhere below, it is not well-understood, routine, conventional to perform explainability as recited (e.g., independently of the input features, in the absence of input data, etc.)…”(p.13). The examiner agrees and withdraws the rejection.
In addition, the Applicant argues that “The disclosures of Weber and/or Lapuschkin would not lead one to a method in which "generating the relevance scores is performed independently of the input features of the input vector" as recited in amended independent claim 1.…”(p.15-16). The examiner disagrees, because the relevance scores of Weber are generated from the neurons, their weights and biases, and not from features of the input data or vector-- independently of the input features of the input vector.
Moreover, the Applicant indicates that “the disclosure of Kursun would not lead one to a method that includes "receiving a request identifying a neural network model for global explanation" as recited in amended independent claim 1…”(p.16-17). The examiner disagrees, because Lapuschkin teaches the resulting relevance map can then optionally be visualized for interpretation by a human observer( Figure 1.1). Also, Lapuschkin shows that the application of LRP is not restricted to a model’s actual prediction. In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable (2.3.3, p. 30, paragraph 4).
Claims 2-12, and 14-15 are unpatentable at least based on the rationale set forth above.
Regarding claim 4, the Applicant indicates that “The expression provided in the Action does not represent a quantity that is "computed as a weighted linear combination of a weight term and a bias term ... the weight term is weighted by a first scalar scaling factor, and wherein the bias term is weighted by a second scalar scaling factor" as recited in amended claim 4…”(p.19). The examiner disagrees, because Equations 2.1 and 2.2 from Weber can be applied to the output layer, say l, of the model to yield the final output: 𝑓(𝑥)=𝑥(𝑙)= 𝑔𝑗(𝑙)(Σ𝑤𝑖𝑗(𝑙−1,𝑙)𝑛𝑖=1𝑔𝑖(𝑙−1)(Σ𝑤𝑖𝑗(𝑙−2,𝑙−1)𝑥𝑖(𝑙−2)+ 𝑛𝑖=1𝑏𝑗(𝑙−1))+ 𝑏𝑗(𝑙)) which amounts to a weighted linear combination of a weight term and a bias term.
Regarding claim 16, the Applicant indicates that “The disclosures of Weber and/or Lapuschkin would not lead one to a method in which "the relevance scores are generated in the absence of input data to the neural network model"…”(p.20-21). The examiner disagrees, because in light of the 35 USC 112a, b rejections, this limitation is being interpreted as , which is taught by Weber’s relevance scores, which are generated from the neurons, their weights and biases, and not from features of the input data or vector(Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs).
Regarding claim 20, the Applicant argues that “The disclosures of Weber and/or Lapuschkin would not lead one to operations in which "the relevance scores are computed as linear combinations of the neuron weights weighted by a first scalar scaling factor and the neuron biases weighted by a second scalar scaling factor"…”(p.23). The examiner disagrees, because Weber teaches “In the NN case, the components 𝑧𝑖 of LRP correspond to the network’s neurons 𝑥𝑖 at layer l, with the output of neurons 𝑥𝑗 at the following layer l + 1 often being computed via linear projection as where 𝑤𝑖𝑗 denotes the (learned) weight between 𝑥𝑖 and 𝑥𝑗. 𝑏𝑗 is the bias term of neuron 𝑥𝑗.”( Chapter 3: Layer-wise Relevance Propagation – An Overview, pp. 13, paragraph 2, and last 3 paragraphs;).
Additionally, the Applicant states that “the disclosures of Weber and/or Lapuschkin would not lead one to operations in which "generating the global explainability dataset comprises generating the global explainability dataset independent of the input vector"…”(p.23). The examiner disagrees, because The examiner disagrees, because Lapuschkin teaches the resulting relevance map can then optionally be visualized for interpretation by a human observer( Figure 1.1). Also, Lapuschkin discloses that the application of LRP is not restricted to a model’s actual prediction. In a multi class setting, as it is the case with most DNN-based applications, LRP can be configured to decompose the prediction for any class output. In such a setting, the joint normalization of multiple relevance maps might desirable (2.3.3, p. 30, paragraph 4).
Applicant’s arguments with respect to claim(s) 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
The following is a summary of cited prior art, but is not relied upon:
Gu WO 2020104252 A1—Training a CNN for visual classification by applying a contrastive layer-wise relevance propagation algorithm (CLRP), after a backward and forward pass.
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CESAR PAULA whose telephone number is (571)272-4128. The examiner can normally be reached Monday - Friday, 6.30am- 4:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Wiley can be reached at (571)272-3923. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CESAR B PAULA/Supervisory Patent Examiner, Art Unit 2145
Prosecution Insights