DETAILED ACTION
This nonfinal office action is responsive to claims filed on April 12, 2024. Claims 1-20 are pending. Claims 1, 8, and 15 are independent.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Zoldi and Rahman (US20190354853), hereinafter Zoldi, in view of Sreeramachandra Murthy and Tsia (US20240112026), hereinafter Murthy.
Regarding claim 1, Zoldi teaches the method:
accessing a multi-layer model, wherein the multi-layer model comprises an input layer, one or more hidden layers, and an output layer, wherein the input layer comprises a plurality of input features for the multi-layer model, (Zoldi, paragraph 0025: “Neural network models can have multiple hidden layers, and such models have a single input layer with n input variables, and a single output layer with one or more outputs, along with multiple hidden layers.”) each hidden layer of the one of more hidden layers comprises a plurality of latent features, (Zoldi, paragraph 0026: “The hidden nodes are configured to learn complex interactions between all their inputs. These complex interactions are called latent factors and represent the underlying drivers of the relationships being modeled by the associated neural network model.”) wherein each latent feature is connected to the input layer or a preceding hidden layer by a limited number of input connections, (Zoldi, paragraph 0054: “Each hidden node connected to the input layer represents an interpretable latent feature and can be represented as a non-linear transform of weighted sum of up to N input variables.”)
wherein each latent feature is associated with an activation state matrix, (Zoldi, paragraph 0060: “FIG. 10 is an activation matrix corresponding to processing p records through the reference neural network model with J hidden nodes. Correlation between the activations across all nodes is sought to determine the natural grouping of records which yield similar activations.”)
wherein an explanation of a latent feature is selected based on an activation state of said latent feature, and the output layer is capable of generating an output based at least in part on activation states of terminal latent features in a final hidden layer, (Zoldi, paragraph 0025: “In this notation, w0j represents the bias term for the hidden node j and w0k represents the bias term for the output node k.”, paragraph 0028: “Accordingly, in exemplary implementations, a system and method simplifies a neural network's architecture, and creates explainable latent features.” and paragraphs 0054-0055: “In the FIG. 7, five interpretable latent features are shown. For the sake of completeness and to act as a guide, the listing of all the interpretable latent factors is shown in FIG. 8. FIG. 8 illustrates a representation of interpretable latent features based on the example in FIG. 7, where σ is the activation function and in this example the activation function is a sigmoidal function. The nomenclature scheme defined above is followed, as noted by the bias terms, w0j.” – The activation function is used to determine the activation state of the latent features while the explainable latent features represents an explanation of the latent features based on the activation state. The output is generated based on the latent features, see e.g. Fig. 7, where the latent features h are generated based on the activation state (using the activation function).)
wherein the terminal latent features are a subset of the latent features that are connected directly to the output layer, and the final hidden layer comprises the terminal latent features; (Zoldi, paragraph 0059: “FIG. 9 shows the schematic representation of different activations for hidden nodes corresponding to different records, to illustrate that different hidden nodes will have different activation for different input records. Shaded sections represent the strength of activation in this schematic example. These two records lie far from each other in the latent space represented by the hidden nodes. Note that for record j, one node does not activate.” – Records I and j of Fig. 9 show that the terminal latent features have differing activations and therefore represents a subset of latent features connected directly to the output layer.)
upon receiving an output generated by the multi-layer model, generating an output explanation by:
removing the input features; (Zoldi, paragraph 0028: “The methodology described herein simplifies the interactions so that the learned latent factors are interpretable, so as to be possible to reduce hidden nodes to a function of a limited subset of predictive input variables, as opposed to the entire set of input variables, without perceptible loss in predictive power.” – The limited subset of input variables indicates that input features are removed.)
removing the latent features that are in non-activation state; selecting an explanation of each of the remaining latent features based at least in part on a number of input connections after the removals; and (Zoldi, paragraph 0008: “ The resulting neural network with all hidden nodes at complexity of N non-zero inputs or less yields an explainable latent feature neural network.” – The hidden nodes only being used for a complexity of N non-zero inputs indicates that the non-activation state latent features are not used. The explainable latent feature neural network is analogous to the explanation of the remaining latent features based on a number of input connections, represented with N here.)
Zoldi does not explicitly teach:
combining the explanations associated with the remaining latent features to generate the output explanation.
However, Murthy teaches:
combining the explanations associated with the remaining latent features to generate the output explanation. (Murthy, paragraph 0141: “The aspect of explainability in the framework was demonstrated in FIG. 20 . The feature ranking of the predicted latent explained the important latent factors, and the interpretable heatmap generated from the training dataset provided answers regarding the critical factors on which the model had based its decisions.” – The heatmap is the combination of explanations as it is used to provide answers regarding critical factors of the model, e.g. the remaining latent features as taught by Zoldi.)
Murthy is considered analogous to the claimed invention as it is in the same field of endeavor, machine learning and providing explainable results in multi-layer models. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to have modified Zoldi, which already teaches generating explanations of latent features in an activation state but does not explicitly teach combining the explanations into an output explanation, to include the teachings of Murthy which does teach combining the explanations into an output in order to “understand the specific attributes that contributed significantly to the variations within the latent space.” (Murthy, paragraph 0139)
Regarding claim 2, Zoldi and Murthy teach the method of claim 1, as cited above.
Zoldi further teaches:
ranking pre-activation terms associated with the remaining terminal latent features, respectively; and (Zoldi, claim 7: “forming a dataset of hidden nodes of size L×W, where L is the number of input records in the training dataset and W is the number of latent feature hidden nodes; applying a matrix factorization technique to define relationships between commonly occurring combinations of latent feature hidden node activations to form a rank ordered matrix of most important functions of the latent feature hidden nodes;” – The dataset formed is from the input data to the hidden nodes which would be preactivation, where the number of latent features were already reduced by Zoldi in claim 1.)
that the explanations are associated with the remaining terminal latent features in an order of the ranking. (Zoldi, claim 7: “using the highest ranking latent feature hidden node relationships to segment data between groups of data records and customers in the training dataset; and the learned segmentations of data records and customers forming explainability on dynamics of the training dataset for use in explanation of which neural network to score the data records or customers” – Using the highest ranking latent features for the explainability indicates that the explanations are associated with the latent features in order of a ranking.)
Zoldi does not explicitly teach:
generating the output explanation by combining the explanations…
However, Murthy further teaches:
generating the output explanation by combining the explanations (Murthy, paragraph 0141: “The feature ranking of the predicted latent explained the important latent factors, and the interpretable heatmap generated from the training dataset provided answers regarding the critical factors on which the model had based its decisions.” – The feature ranking used to generate the heatmap is analogous to generating the output explanation by combining the explanations which Zoldi teaches above.)
Regarding claim 3, Zoldi and Murthy teach the method of claim 1, as cited above.
Zoldi further teaches:
retrieving pre-activation terms for each input connections connected to the latent feature by multiplying a value of each input connection by a corresponding weight of the input connection; transforming the pre-activation terms by an activation function to determine an activation mode for each input connection, wherein the activation modes include negative activation, non-activation, and positive activation; (Zoldi, paragraph 0055: “FIG. 8 illustrates a representation of interpretable latent features based on the example in FIG. 7, where σ is the activation function and in this example the activation function is a sigmoidal function. The nomenclature scheme defined above is followed, as noted by the bias terms, w0j.” and paragraph 0024: “The non-linear transform can be a sigmoid activation function, or one of the many other non-linear transforms such as tan h, or a rectifier, etc. FIG. 1 shows a perceptron with n input variables, and a sigmoidal activation.” – Fig. 8 shows that the input term is multiplied by the bias term as the pre-activation term which is then used in a sigmoidal function which represents the transformation by an activation function. Further, the non-linear transformation being a tan h indicates that the result of the activation function, e.g., the activation mode, would be negative, positive, or zero which is analogous to negative activation, positive activation, or non-activation, respectively.)
identifying an activation state for the latent feature by combining the activation modes of the input connections of the latent feature, wherein the activation state corresponds to a cell in the activation state matrix; and (Zoldi, paragraph 0046: “For example, the four inputs going into h3, and the subsequent output value out of h3 as a target can be recorded. This provides the activation of each of the hidden node.” And claim 7: “applying a matrix factorization technique to define relationships between commonly occurring combinations of latent feature hidden node activations to form a rank ordered matrix of most important functions of the latent feature hidden nodes;” – The activation of each hidden node being provided is analogous to identifying the activation state for the latent feature. The matrix factorization technique to define relationships indicates that there is a corresponding cel in the activation matrix which represents the activation state. See also Fig. 10.)
selecting an explanation that corresponds to the identified activation state for the latent feature. (Zoldi, claim 7: “using the highest ranking latent feature hidden node relationships to segment data between groups of data records and customers in the training dataset; and … the learned segmentations of data records and customers forming explainability on dynamics of the training dataset for use in explanation of which neural network to score the data records or customers.” – The ranked features representing relationships which are then used to form explainability is analogous to selecting an explanation of the activation sate for the latent feature.)
Regarding claim 4, Zoldi and Murthy teach the method of claim 3, as cited above.
Zoldi further teaches:
selecting an explanation that corresponds to the identified activation state for the latent feature further comprises resolving bimodal states by comparing magnitudes of the transformed pre-activation terms of the input connections to determine which input connection has a stronger influence on the latent feature. (Zoldi, paragraph 0035: “FIG. 3 shows a schematic representation of an arbitrary objective function C, as a function of two weights, w1 and w2. Any of a number of optimization algorithms can be used for minimizing the cost functions to identify the most optimal weights. In some implementations, a gradient descent method can be used. In this approach, the gradient vector of the curve described by the cost function is computed.” – There are weights from the input layer to the first hidden layer, see e.g. Fig. 7, thus the gradient descent method of optimizing the weights is comparing the magnitudes of the input connections to determine the input connection with the stronger influence, i.e. the gradient descent method would converge on the weights that have the biggest influence on the latent features. Additionally, fig 3 shows the weights ranging from -1 to 1 which is thus the bimodal states of the input connections.)
Regarding claim 5, Zoldi and Murthy teach the method of claim 1, as cited above.
Zoldi further teaches:
ranking pre-activation terms associated with the terminal latent features, respectively; (Zoldi, claim 7: “forming a dataset of hidden nodes of size L×W, where L is the number of input records in the training dataset and W is the number of latent feature hidden nodes;” applying a matrix factorization technique to define relationships between commonly occurring combinations of latent feature hidden node activations to form a rank ordered matrix of most important functions of the latent feature hidden nodes; - The dataset formed is from the input data to the hidden nodes which would be preactivation, where the number of latent features were already reduced by Zoldi in claim 1.)
identifying the terminal latent features with the pre-activation term that exceeds a threshold, and (Zoldi, claim 7: “using the highest ranking latent feature hidden node relationships to segment data between groups of data records and customers in the training dataset;” – Using the highest ranking latent feature hidden node relationships indicates that those ranks exceed a threshold.)
that the explanations are associated with the identified terminal latent features in an order of the ranking. (Zoldi, claim 7: “the learned segmentations of data records and customers forming explainability on dynamics of the training dataset for use in explanation of which neural network to score the data records or customers”)
Zoldi does not explicitly teach:
generating the output explanation by combining the explanations…
However, Murthy further teaches:
generating the output explanation by combining the explanations (Murthy, paragraph 0141: “The feature ranking of the predicted latent explained the important latent factors, and the interpretable heatmap generated from the training dataset provided answers regarding the critical factors on which the model had based its decisions.” – The feature ranking used to generate the heatmap is analogous to generating the output explanation by combining the explanations which Zoldi teaches above.)
Regarding claim 6, Zoldi and Murthy teach the method of claim 1, as cited above.
Zoldi further teaches:
the limited number is restricted to either one or two input connections per latent feature. (Zoldi, paragraph 0030: “Extract any hidden node that has up to N inputs only. From interpretability perspective, N is set to 2, meaning nodes of size 1 and 2 non-zero inputs are automatically extracted after regularization.” – Zoldi teaches that the limited number is restricted to N above, having N set to 2 indicates two input connections while being able to set N indicates that the method is capable of restricting to one input connection per latent feature.)
Regarding claim 7, Zoldi and Murthy teach the method of claim 6, as cited above.
Zoldi further teaches:
selecting an explanation for a remaining latent feature based on an activation state of said remaining latent feature if the number of input connections is zero after the removals; and (Zoldi, claim 7: “using the highest ranking latent feature hidden node relationships to segment data between groups of data records and customers in the training dataset; and … the learned segmentations of data records and customers forming explainability on dynamics of the training dataset for use in explanation of which neural network to score the data records or customers.” And paragraph 0007: “applying regularization to the cost function of the neural network model such that weights of one or more edges connecting the predictive input variables to the hidden nodes converge towards zero.” And paragraph 0040: “The process for simplifying the model architecture includes forcing most of the hidden weights to converge to 0. By doing so, the fully connected model is converted into a sparsely connected neural network model, where many of the weights are near zero.” – The weights of the edges converging toward zero indicates that there are edges whose weights are zero and would therefore be removed as an input connection. Therefore, with the inputs being restricted toward one or two there would then be some latent features with zero input connections. The explainability being formed is therefore the explanation of the features with zero input connections after the removals.)
selecting an explanation for remaining latent feature based on an activation state of an input connection if the number of input connections is one after the removals. (Zoldi, claim 7: “using the highest ranking latent feature hidden node relationships to segment data between groups of data records and customers in the training dataset; and … the learned segmentations of data records and customers forming explainability on dynamics of the training dataset for use in explanation of which neural network to score the data records or customers.” and paragraph 0030: “Extract any hidden node that has up to N inputs only. From interpretability perspective, N is set to 2, meaning nodes of size 1 and 2 non-zero inputs are automatically extracted after regularization.” – The nodes having 1 input connection after extraction is analogous to the remaining latent features having one input connection. The explainability being formed is therefore the explanation of the features with one input connection after the removals.)
Regarding claim 8, claim 8 has all the same limitations of claim 1 which are taught by Zoldi and Murthy -see claim 1 above.
Zoldi additionally teaches:
A computer program product comprising a non-transient machine-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: (Zoldi, paragraph 0063: “These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.”)
Regarding claim 9, Zoldi and Murthy teach the computer program product of claim 8, as cited above.
Claim 9 additionally has the same limitations of claim 2 which are taught by Zoldi and Murthy – see claim 2 above.
Regarding claim 10, Zoldi and Murthy teach the computer program product of claim 8, as cited above.
Claim 10 additionally has the same limitations of claim 3 which are taught by Zoldi and Murthy – see claim 3 above.
Regarding claim 11, Zoldi and Murthy teach the computer program product of claim 10, as cited above.
Claim 11 additionally has the same limitations of claim 4 which are taught by Zoldi and Murthy – see claim 4 above.
Regarding claim 12, Zoldi and Murthy teach the computer program product of claim 8, as cited above.
Claim 12 additionally has the same limitations of claim 5 which are taught by Zoldi and Murthy – see claim 5 above.
Regarding claim 13, Zoldi and Murthy teach the computer program product of claim 8, as cited above.
Claim 13 additionally has the same limitations of claim 6 which are taught by Zoldi and Murthy – see claim 6 above.
Regarding claim 14, Zoldi and Murthy teach the computer program product of claim 13, as cited above.
Claim 14 additionally has the same limitations of claim 7 which are taught by Zoldi and Murthy – see claim 7 above.
Regarding claim 15, claim 15 has all the same limitations of claim 1 which are taught by Zoldi and Murthy -see claim 1 above.
Zoldi additionally teaches:
a programmable processor; and a non-transient machine-readable medium storing instructions that, when executed by the processor, cause the at least one programmable processor to perform operations comprising: (Zoldi, paragraph 0063: “These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.”)
Regarding claim 16, Zoldi and Murthy teach the system of claim 15, as cited above.
Claim 16 additionally has the same limitations of claim 2 which are taught by Zoldi and Murthy – see claim 2 above.
Regarding claim 17, Zoldi and Murthy teach the system of claim 15, as cited above.
Claim 17 additionally has the same limitations of claim 3 which are taught by Zoldi and Murthy – see claim 3 above.
Regarding claim 18, Zoldi and Murthy teach the system of claim 17, as cited above.
Claim 18 additionally has the same limitations of claim 4 which are taught by Zoldi and Murthy – see claim 4 above.
Regarding claim 19, Zoldi and Murthy teach the system of claim 15, as cited above.
Claim 19 additionally has the same limitations of claim 5 which are taught by Zoldi and Murthy – see claim 5 above.
Regarding claim 20, Zoldi and Murthy teach the system of claim 15, as cited above.
Claim 20 additionally has the same limitations of claim 6 which are taught by Zoldi and Murthy – see claim 6 above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Zoldi and Rahman (US20210004703)
Pardo et al. (Coloring the Black Box: Visualizing Neural Network Behavior with a Self-Introspective Model)
Alvarez-Melis and Jaakkola (Towards Robust Interpretability with Self-Explaining Neural Networks)
Cunningham et al. (Sparse Autoencoders Find Highly Interpretable Features in Language Models)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACQUELINE MEYER whose telephone number is (703)756-5676. The examiner can normally be reached M-F 8:00 am - 4:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tamara Kyle can be reached at 571-272-4241. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.C.M./Examiner, Art Unit 2144
/TAMARA T KYLE/Supervisory Patent Examiner, Art Unit 2144
