DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claim the benefit of a prior-filed U.S. Appl. No. 63/375,262 filed September 22, 2022, which is acknowledged by the examiner.
Drawings
The drawings were received on 04/11/2023. These drawings are acceptable.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 06/22/2023 is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 13 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Hanazawa (US 12387361, hereinafter ‘Han’) in view of Huts et al. (US 20230067026, hereinafter ‘Huts’).
Regarding independent claim 1, Han teaches a computer-implemented method comprising: (in 19:40-43: The above-described series of processing can also be executed by hardware or software. In a case where the series of processing is executed by software, a program that constitutes the software is installed on a computer…)
identifying, by one or more processors, a set of Shapley values corresponding to the at least one machine-learning model, ( As depicted in Fig. 6; And in 14:14-23: In addition, the interpreting unit 252 performs SHAP (SHapley Additive exPlanations) processing with respect to the attention area image using the approximate image recognition model and calculates a SHAP value with respect to each grid segment in the attention area. In other words, the interpreting unit 252 calculates a SHAP value (hereinafter, referred to as a type recognition contribution) indicating a contribution by each grid segment toward recognition (class classification) of a type of an attention object in the attention area. )
the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set; (in 14:14-23: In addition, the interpreting unit 252 performs SHAP (SHapley Additive exPlanations) processing with respect to the attention area image using the approximate image recognition model and calculates a SHAP value with respect to each grid segment in the attention area. In other words, the interpreting unit 252 calculates a SHAP value (hereinafter, referred to as a type recognition contribution) indicating a contribution by each grid segment [at least one machine-learning model for each different portion of plurality of portions of the training data set] toward recognition (class classification) of a type of an attention object in the attention area.…; And in 13:9-16: In step S1, the recognizing unit 212 performs object recognition. Specifically, the recognizing unit 212 performs object recognition with respect to each sample image using an object recognition model generated in advance by machine learning performed by the learning unit 211. The recognizing unit 212 supplies the area setting unit 251 with data indicating a result of object recognition of each sample image.; And claim 13: … calculating a plurality of SHAP (SHapley Additive explanations) values for each of the horizontal segment and the vertical segment, wherein each SHAP value of the plurality of SHAP values indicates a contribution by each of the horizontal segment and the vertical segment [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set], and the contribution is toward recognition of a type of the attention object; and generating, a plurality of explanatory images, based on the plurality of SHAP values, wherein each explanatory images of the plurality of explanatory images corresponds to a SHAP value of the plurality of SHAP values, and each explanatory image represents the contribution by each of the horizontal segment and the vertical segment.)
generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation. (12:64-66: The explaining unit 253 generates an explanatory image [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values] for explaining the interpretation of the recognition result of the object recognition model and outputs the explanatory image [and outputting, by the one or more processors, at least the animation]…; And in 16:7-23: For example, the explaining unit 253 generates an image [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values] (hereinafter, referred to as a left-end recognition explanatory image) in which vertical segments in an attention area of an attention area image (or a sample image to be a source of an attention area image) are colored according to left-end recognition contribution. The explaining unit 253 generates an image (hereinafter, referred to as a right-end recognition explanatory image) in which vertical segments of an attention area image (or a sample image to be a source of an attention area image) are colored according to right-end recognition contribution. The explaining unit 253 generates an image (hereinafter, referred to as an upper-end recognition explanatory image) in which horizontal segments of an attention area image (or a sample image to be a source of an attention area image) are colored according to upper-end recognition contribution…; And claim 13.)
While Han teaches generating an explanatory image as a Shapley image based on a set of Shapley values.
Han does not expressly associate the Shapley image with an animation.
Huts does expressly associate the Shapley image with an animation, in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations of model inferences [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]… For example, a feature that is highly correlated with the target of a computer vision/data analytics problem generally has high expected utility for inferring the solution to that problem. Any suitable technique or metric may be used to assess feature importance including, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]. The foregoing techniques/metrics for assessing feature importance are described in further detail below… [0266] Another type of explanatory visualization is the image embedding visualization [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]... Referring to FIG. 15, an example of an image embedding visualization of bedroom images [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation] from the residential real estate data set is shown. In the example of FIG. 15, a small number of images 1502 of unfurnished or sparsely furnished bedrooms are clustered together, apart from the images of fully furnished bedrooms… [0267] In some embodiments, a model development system 100 and/or a model deployment system 1100 may be capable of generating image embedding visualizations of the images in a data set [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]. Image embedding visualizations may be generated using any suitable technique. In some embodiments, the highest-level image features extracted from each image in the set of images may be converted into 2D coordinates (e.g., Cartesian coordinates)…; And in [0260] Image inference explanations may help the user understand individual inferences of image-based models [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]. For example, some embodiments of the model development system 100 and/or the model deployment system 1100 may provide an explanation user interface (explanation UI)… [0269] In some embodiments, feature importance metrics [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set] used by a model development system 100 and/or a model deployment system 1100 may include, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]…. [0335] In step 1914, an image inference explanation visualization is generated based on the feature importance scores for the constituent image features, the values of the constituent image features, and the activation maps. The image inference explanation visualization [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation] may identify portions of the image data [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set] that contribute to the determination of the value of the target… [0354] Insurance companies can conduct more consistent and accurate vehicle damage assessments to help reduce fraud and streamline the claims process. Healthcare providers can use image-based neural networks to automate the examination and diagnosis of health issues from MRI's, CAT scans and X-rays. [0355] Other applications range from using images of gas stations to help better plan where to focus marketing spend, to the automated labeling of apparel from fashion photography for eCommerce websites.)
Huts and Han are analogous art because both involve developing information retrieval and data processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for implementing automated machine learning techniques to develop and deploy data analytics tools for image data based on Shapley feature importance metric as disclosed by Huts with the method of developing information retrieval and object recognition models based on interpreted in units of segments as disclosed by Han.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Huts and Han, as noted above; Doing so allows for enabling interpretation of machine learning models, in which the “payout” is the model prediction, the “team members” are the features or variables taken into consideration by the model, and a goal of the exercise is to assign importance to each feature, even though the features may not all be equally influential to the model (Huts, 0290).
Regarding claim 13 and 20, the limitations are similar to claim 1 limitations and rejected under the same rationale.
Claims 1-3, 9-15, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Öztireli et al. (US 20210042613, hereinafter ‘Oz’) in view of Huts et al. (US 20230067026, hereinafter ‘Huts’).
Regarding independent claim 1, Oz teaches a computer-implemented method comprising: (in [0018] The compute instance 110 is configured to implement one or more applications or subsystems of applications. For explanatory purposes only, each application is depicted as residing in the memory 116 of a single compute instance 110 and executing on a processor 112 of the single compute instance 110. However, in alternate embodiments, the functionality of each application may be distributed across any number of other applications that reside in the memories 116 of any number of compute instances 110 and execute on the processors 112 of any number of compute instances 110 in any combination.…)
identifying, by one or more processors, a set of Shapley values corresponding to the at least one machine-learning model, the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set; ( in [0027] To address the above problems, the compute instance 110 implements a relevance application 140 that automatically estimates Shapley values for the trained neural network 120 [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]. The relevance application 140 resides in the memory 116 and executes on the processor 112. As shown, the relevance application 140 computes a Shapley value matrix 190 based on the trained neural network 120 and the input point set 130. The Shapley value matrix 190 includes, without limitation, Shapley value vectors 192(1) to 192(N) [identifying, by one or more processors, a set of Shapley values corresponding to the at least one machine-learning model], where the Shapley value vector 192(i) is associated with the input point 132(i) and is denoted herein as The Shapley value vector 192(i) includes, without limitation, M estimated Shapley values (not shown), where the estimated Shapley value (denoted herein as r.sub.i,j) accurately quantifies a contribution of the input point 132(i) to the output point 142(j).)
generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation. (in [0050] For example, to understand why the trained neural network 120 misclassified a particular image as female, a user could configure the relevance application 140 to compute the Shapley value matrix 190 for the input point set 130 representing the image. The relevance application 140 could then display the Shapley value vector 192 corresponding to the output point 142 specifying the probability of female as a “Shapley image.” [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation. ] The relevance application 140 could set the color of each pixel in the Shapley image based on the Shapley value of the associated input point 132. Accordingly, the coloring of the Shapley image would visually illustrate [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation] the contribution of each of the inputs points 132 with respect to the misclassification…)
Additionally, Huts teaches the Shapley image visualization for computer vision animation, in [0093] The models (e.g., data analytics models) and techniques (e.g., modeling techniques, automation techniques, techniques for determining the importance of certain data relative to other data, techniques for interpreting the outputs of models and tools, etc.) described herein are generally described in the context of performing computer visions tasks (e.g., tasks related to the analysis and/or interpretation of images or videos) [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values] or solving data analytics problems using both image data and non-image data…; And in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations of model inferences [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]… For example, a feature that is highly correlated with the target of a computer vision/data analytics problem generally has high expected utility for inferring the solution to that problem. Any suitable technique or metric may be used to assess feature importance including, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]. The foregoing techniques/metrics for assessing feature importance are described in further detail below… [0266] Another type of explanatory visualization is the image embedding visualization [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]... Referring to FIG. 15, an example of an image embedding visualization of bedroom images [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation] from the residential real estate data set is shown. In the example of FIG. 15, a small number of images 1502 of unfurnished or sparsely furnished bedrooms are clustered together, apart from the images of fully furnished bedrooms… [0267] In some embodiments, a model development system 100 and/or a model deployment system 1100 may be capable of generating image embedding visualizations of the images in a data set [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]. Image embedding visualizations may be generated using any suitable technique. In some embodiments, the highest-level image features extracted from each image in the set of images may be converted into 2D coordinates (e.g., Cartesian coordinates)…; And in [0260] Image inference explanations may help the user understand individual inferences of image-based models [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation]. For example, some embodiments of the model development system 100 and/or the model deployment system 1100 may provide an explanation user interface (explanation UI)… [0269] In some embodiments, feature importance metrics [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set] used by a model development system 100 and/or a model deployment system 1100 may include, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set]…. [0335] In step 1914, an image inference explanation visualization is generated based on the feature importance scores for the constituent image features, the values of the constituent image features, and the activation maps. The image inference explanation visualization [generating, by the one or more processors, a plurality of frames of an animation based on the set of Shapley values; and outputting, by the one or more processors, at least the animation] may identify portions of the image data [the set of Shapley values comprising a Shapley value generated after completion of training of the at least one machine-learning model for each different portion of plurality of portions of the training data set] that contribute to the determination of the value of the target… [0354] Insurance companies can conduct more consistent and accurate vehicle damage assessments to help reduce fraud and streamline the claims process. Healthcare providers can use image-based neural networks to automate the examination and diagnosis of health issues from MRI's, CAT scans and X-rays. [0355] Other applications range from using images of gas stations to help better plan where to focus marketing spend, to the automated labeling of apparel from fashion photography for eCommerce websites.)
Huts and Oz are analogous art because both involve developing information retrieval and data processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for implementing automated machine learning techniques to develop and deploy data analytics tools for image data based on Shapley feature importance metric as disclosed by Huts with the method of developing information retrieval models and approaches for understanding trained models as disclosed by Oz.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Huts and Oz, as noted above; Doing so allows for enabling interpretation of machine learning models, in which the “payout” is the model prediction, the “team members” are the features or variables taken into consideration by the model, and a goal of the exercise is to assign importance to each feature, even though the features may not all be equally influential to the model (Huts, 0290).
Regarding claim 2, the rejection of claim 1 is incorporated and Oz in combination with Huts further teaches the computer-implemented method of claim 1, further comprising: generating at least one synthetic Shapley value by at least performing an interpolation between a first Shapley value of the set of Shapley values and a second Shapley value of the set of Shapley values; and generating an updated set of Shapley values by updating the set of Shapley values to include the at least one synthetic Shapley value, wherein the plurality of frames is generated based on the updated set of Shapley values. (in [0027] To address the above problems, the compute instance 110 implements a relevance application 140 that automatically estimates Shapley values [generating at least one synthetic Shapley value by at least performing an interpolation between a first Shapley value of the set of Shapley values and a second Shapley value of the set of Shapley values] for the trained neural network 120. The relevance application 140 resides in the memory 116 and executes on the processor 112. As shown, the relevance application 140 computes a Shapley value matrix 190 based on the trained neural network 120 and the input point set 130. The Shapley value matrix 190 includes, without limitation, Shapley value vectors 192(1) to 192(N), where the Shapley value vector 192(i) is associated with the input point 132(i) and is denoted herein as The Shapley value vector 192(i) includes, without limitation, M estimated Shapley values (not shown), where the estimated Shapley value (denoted herein as r.sub.i,j) accurately quantifies a contribution of the input point 132(i) to the output point 142(j)… [0030] Instead of computing exact Shapley values, the relevance application 140 implements a probabilistic framework to compute estimated Shapley values [generating at least one synthetic Shapley value by at least performing an interpolation between a first Shapley value of the set of Shapley values and a second Shapley value of the set of Shapley values]… As shown, the relevance application 140 includes, without limitation, a neural network conversion engine 150, an input converter 152, a probabilistic neural network 160, a coalition size list 154, and N instances of a Shapley value engine 170. In alternate embodiments, the relevance application 140 may include any number of instances of the Shapley value engine 170…[0048] For instance, in some embodiments, the relevance application 140 may be configured to compute Shapley values for any number of the internal points in the trained neural network 120 in any technically feasible fashion. For example, the relevance application 140 could configure the probabilistic neural network 160 to store and append internal distributions to the output distribution set [generating at least one synthetic Shapley value by at least performing an interpolation between a first Shapley value of the set of Shapley values and a second Shapley value of the set of Shapley values]. The neural network wrapper 170 could compute and append internal marginal contributions to the marginal contribution vector 172, and the Shapley value engine 170 could compute and append internal Shapley values to the Shapley value vector 192 [and generating an updated set of Shapley values by updating the set of Shapley values to include the at least one synthetic Shapley value, wherein the plurality of frames is generated based on the updated set of Shapley value]. )
Regarding claim 3, the rejection of claim 1 is incorporated and Oz further teaches the computer-implemented method of claim 1, further comprising: applying at least the set of Shapley values to an (in [0008] At least one technical advantage of the disclosed techniques relative to the prior art is that the disclosed techniques can more efficiently and more reliably quantify how a trained neural network operates across a wide range of architectures and input types [applying at least the set of Shapley values to an ]. In particular, contrary to prior art approaches that use unreliable and/or non-robust heuristics, the disclosed techniques use statistical approximation to compute estimated Shapley values that more accurately quantify the contributions of input points to output points. Further, estimating Shapley values using the disclosed techniques is computationally more efficient than computing exact Shapley values [applying at least the set of Shapley values to an ], as is done in prior art approaches. In this regard, the number of network evaluations required to estimate the Shapley values using the disclosed techniques is linearly related to the number of inputs, as opposed to exponentially related, which is the case in prior art approaches…; And in [0045] The relevance application 140 stores the N Shapley value vectors 192(1)-192(N) as the Shapley value matrix 190. The relevance application 140 then displays and/or transmits any portion of the Shapley value matrix 190 to any number of software applications in any technically feasible fashion to provide insight into what the trained neural network 120 has learned… [0049] Advantageously, relative to the prior art, the estimated Shapley values [applying at least the set of Shapley values to an ] included in the Shapley value matrix 190 can more efficiently and more reliable quantify the behavior of the trained neural network 120 across a wide range of architectures and input types Accordingly, the relevance application 140 can be used to increase confidence in the trained neural network 120, improve the accuracy of the trained neural network 120, implement portions of the trained neural network 120 to solve a new or different problem [applying at least the set of Shapley values to an ], prune neurons to increase the efficiency of the trained neural network 120, etc.)
Additionally, Huts further teaches the computer-implemented method of claim 1, further comprising: applying at least the set of Shapley values to an LSTM model configured to predict a model type based on the set of Shapley values. (in [0143] As another example, the feature extraction module may include a pre-trained text feature extraction model, which may extract text features (e.g., low-, medium-, high-, and/or highest-level text features) from text data and/or natural language data. A pre-trained text feature extraction model may use a convolutional neural network (CNN), a recurrent neural network (e.g., RNN, including but not limited to long short-term memory (LSTM) RNN) [applying at least the set of Shapley values to an LSTM model configured to predict a model type based on the set of Shapley values],…; And in[0266] Another type of explanatory visualization is the image embedding visualization. In an image embedding visualization, a set of images (e.g., from a training data set or an inference data set) are clustered and displayed on a 2D plot, such that images that appear similar to a model (e.g., a data analytics model) are located relatively close together [applying at least the set of Shapley values to an LSTM model configured to predict a model type based on the set of Shapley values], and images that appear dissimilar to the model are located relatively far apart.. From the perspective of an image feature extraction model or a downstream data analytics model, the images 1502 in this cluster may be anomalous. More generally, the image embedding visualization may help users to identify anomalous images or sets of anomalous images in a data set, because such images or sets (e.g., clusters) of images may be spaced apart from the other images in an image embedding visualization… In some embodiments, feature importance metrics used by a model development system 100 and/or a model deployment system 1100 may include, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) [applying at least the set of Shapley values to an LSTM model configured to predict a model type based on the set of Shapley values]. These metrics and some embodiments of techniques for assessing (or “scoring”) the feature importance of non-tabular features (e.g., image features) according to these metrics are described below. )
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 9, the rejection of claim 1 is incorporated and Oz in combination with Huts teaches the computer-implemented method of claim 1, wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation. (in [0050] For example, to understand why the trained neural network 120 misclassified a particular image as female, a user could configure the relevance application 140 to compute the Shapley value matrix 190 for the input point set 130 representing the image. The relevance application 140 could then display the Shapley value vector 192 corresponding to the output point 142 specifying the probability of female as a “Shapley image.” The relevance application 140 could set the color of each pixel in the Shapley image based on the Shapley value of the associated input point 132. Accordingly, the coloring of the Shapley image would visually illustrate [wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation] the contribution of each of the inputs points 132 with respect to the misclassification. The user could subsequently enhance the training dataset based on the insights provided by the Shapley image and retrain the trained neural network 120 to improve the accuracy. [0051] In another example, as part of an effort to generate a more efficient, smaller trained neural network 120, a user could configure [wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation] the relevance application 140 to compute the Shapley value matrix 190 that included all the internal Shapley values for each of a wide variety of input point sets 130. Based on the Shapley value matrices, the user could determine which of the neurons are contributing least to the output value sets 140 and, therefore, would be good candidates for pruning. And in [0078] The neural network conversion engine 150 may acquire the internal evaluation list 240 in any technically feasible fashion. For instance, in some embodiments, the relevance engine 140 determines the internal evaluation list 240 based on user input received via a graphical user interface (“GUI”) and then transmits the internal evaluation list 240 to the neural network conversion engine 150…)
Additionally, Huts teaches wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation, in [0158] In the example of FIG. 3, the “feature importance” values for the image features may be univariate feature importance values that can be quantitatively compared with the feature importance values for the other, non-image features. This comparison may help the user gain intuition about the importance of including image data in the data set. In the example of FIG. 3, the images of a house's bedroom and kitchen are in the top 5 most important features in the data set [wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation], complemented by numeric features indicating the house's number of bathrooms and square footage, and a location (geospatial) feature indicating the boundaries of the zip code area in which the house is located. And in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations [wherein outputting the animation comprises: causing rendering of an interface comprising at least the animation] of model inferences. A feature's “feature importance” may indicate the feature's expected utility (on an absolute scale or relative to other features) for inferring the solution to a computer vision problem or data analytics problem…
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 10, the rejection of claim 9 is incorporated and Oz in combination with Huts teaches the computer-implemented method of claim 9, wherein the interface further comprises at least one other animation corresponding to a second machine-learning model of a second model type, wherein the animation is generated based on a second set of Shapley values corresponding to training of the second machine-learning model. (in [0271] In general, the “univariate feature importance” of a feature F for a modeling problem P is an estimate of the correlation between the target of the modeling problem P and the feature [wherein the interface further comprises at least one other animation corresponding to a second machine-learning model of a second model type]…[0277] In some embodiments, the model development system 100 may determine univariate feature importance scores for one or more (e.g., all) the features of a data set during the exploratory data analysis phase of the model development process. [0278] In some embodiments, the model development system 100 may determine ACE scores for each of the constituent features F.sub.C (e.g., constituent image features) extracted from a column of non-tabular data elements (e.g., images) by a feature extraction model (e.g., an image feature extraction model) [wherein the animation is generated based on a second set of Shapley values corresponding to training of the second machine-learning model], and may concatenate those ACE scores to form a non-tabular (e.g., image) feature importance vector...)
Additionally, Huts teaches wherein the interface further comprises at least one other animation corresponding to a second machine-learning model of a second model type, wherein the animation is generated based on a second set of Shapley values corresponding to training of the second machine-learning model. (in [0291] In some embodiments, the Shapley values of a linear model's features may be used to determine feature importance values for those features. In some embodiments, Model-Specific Approximations of SHAP values of a tree-based model's features, as described in the literature for SHAP Tree Explainer, may be used to determined feature importance values for those features… [0301] Generate an image inference explanation visualization based on the non-tabular data element (e.g., image) 1701, the activation maps 1705, the feature vector 1710 derived from the non-tabular data element 1701, and the feature importance vector 1730. The image inference explanation visualization may indicate (e.g., highlight) the portions of the image 1701 in the inference data sample that contributed most to the output generated by the two-stage model 130 for the inference data sample. The image inference visualization explanation may be generated by forming a weighted combination of the individual activation maps of the constituent image features [wherein the interface further comprises at least one other animation corresponding to a second machine-learning model of a second model type, wherein the animation is generated based on a second set of Shapley values corresponding to training of the second machine-learning model], with each of the individual activation maps weighted by a value derived from the feature importance score and feature value of the corresponding constituent image feature. For example, the weight applied to the activation map for a particular constituent feature may be the product of the feature importance score and the feature value for that feature. And the plurality of trained models in [0320] In step 1811, inference data that include first data of a non-tabular data type (e.g., image data, textual data, natural language data, speech data, auditory data, spatial data, or a combination thereof) are obtained… [0322] In step 1813, a value of a data analytics target is determined based on the values of the constituent features. The value of the data analytics target may be determined by a trained machine learning model. In some cases, the inference data further include second data of a tabular data type (e.g., numeric data, categorical data, time-series data, etc.) [..corresponding to a second machine-learning model of a second model type, … to training of the second machine-learning model]…
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 11, the rejection of claim 9 is incorporated and Huts further teaches the computer-implemented method of claim 9, wherein the interface comprises a plurality of animations representing different plot type. (in [0281] In general, the feature impact of a non-tabular feature F for a trained model M may be determined by (1) using the model M to generate one set of inferences for a validation data set in which the data samples contain the actual values of the feature F, (2) using the model M to generate another set of inferences for a version of the validation data set in which the values of the feature F have been altered to destroy (e.g., reduce, minimize, etc.) the feature's predictive value, and (3) comparing the performance P1 (e.g., accuracy) of the first set of inferences to the performance P2 (e.g., accuracy) of the second set of inferences. In general, as the difference between P1 and P2 increases, the feature impact of the feature F increases… [0286] In some embodiments, the feature impact scores determined for various features (e.g., features of the same type, features of different types, tabular features, non-tabular features, image features, non-image features, etc.) [wherein the interface comprises a plurality of animations representing different plot type] can be quantitatively compared to each other. This comparison may help the user understand the importance of including various non-tabular data elements (e.g., images) in the data set. Likewise, the model-specific feature impact scores of a particular feature (e.g., a non-tabular feature) for a set of models may be compared. This comparison may help the user understand which models are doing a good job exploiting the information represented by the feature and which are not. And in [0322] In step 1813, a value of a data analytics target is determined based on the values of the constituent features [wherein the interface comprises a plurality of animations representing different plot type]. The value of the data analytics target may be determined by a trained machine learning model. In some cases, the inference data further include second data of a tabular data type (e.g., numeric data, categorical data, time-series data, etc.). In some embodiments, the determining of the value of the data analytics target is also based on values of one or more features derived from the second data… [0323] In some embodiments, the method 1810 further includes a step of arranging the values of the constituent features of the first data and the values of the features derived from the second data in a table [wherein the interface comprises a plurality of animations representing different plot type]. In some embodiments, the determining of the value of the data analytics target is performed by applying the trained machine learning model to the table. In some embodiments, the trained machine learning model includes a gradient boosting machine. In some embodiments, the value of the data analytics target includes a prediction based on the inference data, a description of the inference data, a classification associated with the inference data, and/or a label associated with the inference data. )
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 12, the rejection of claim 1 is incorporated and Oz in combination with Hutz teaches the computer-implemented method of claim 1, further comprising: training, by the one or more processors, the at least one machine-learning model utilizing the plurality of portions of the training data set. ( in [0027] To address the above problems, the compute instance 110 implements a relevance application 140 that automatically estimates Shapley values for the trained neural network 120 [… the plurality of portions of the training data set]. The relevance application 140 resides in the memory 116 and executes on the processor 112. As shown, the relevance application 140 computes a Shapley value matrix 190 based on the trained neural network 120 and the input point set 130. The Shapley value matrix 190 includes, without limitation, Shapley value vectors 192(1) to 192(N) […the plurality of portions of the training data set], where the Shapley value vector 192(i) is associated with the input point 132(i) and is denoted herein as The Shapley value vector 192(i) includes, without limitation, M estimated Shapley values (not shown), where the estimated Shapley value (denoted herein as r.sub.i,j) accurately quantifies a contribution of the input point 132(i) to the output point 142(j). And in [0049] Advantageously, relative to the prior art, the estimated Shapley values included in the Shapley value matrix 190 can more efficiently and more reliable quantify the behavior of the trained neural network 120 across a wide range of architectures and input types [training, by the one or more processors, the at least one machine-learning model utilizing the plurality of portions of the training data set] Accordingly, the relevance application 140 can be used to increase confidence in the trained neural network 120, improve the accuracy of the trained neural network 120, implement portions of the trained neural network 120 to solve a new or different problem [training, by the one or more processors, the at least one machine-learning model utilizing the plurality of portions of the training data set], prune neurons to increase the efficiency of the trained neural network 120, etc. And in [0089] At least one technical advantage of the disclosed techniques relative to the prior art is that the relevance application can more efficiently and more reliably quantify how a trained neural network operates across a wide range of architectures and input types… Further, estimating Shapley values using the disclosed techniques is computationally more efficient than computing exact Shapley values, as is done in prior art approaches… As described previously herein, insights into a trained neural network provided by the estimated Shapley values [utilizing the plurality of portions of the training data set] can be used to efficiently re-train [training, by the one or more processors, the at least one machine-learning model utilizing the plurality of portions of the training data set]] the trained neural network to increase accuracy and reliability. These technical advantages provide one or more technological advancements over the prior art...)
Regarding claims 13 and 20, the limitations are similar to claim 1 limitations and rejected under the same rationale.
Regarding claim 14, the limitations are similar to claim 2 limitations and rejected under the same rationale.
Regarding claim 15, the limitations are similar to claim 3 limitations and rejected under the same rationale.
Claims 4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Öztireli et al. (US 20210042613, hereinafter ‘Oz’) in view of Huts et al. (US 20230067026, hereinafter ‘Huts’) in further view of Schwiep et al. (US 20220292308, hereinafter ‘Jill’)
Regarding claim 4, the rejection of claim 3 is incorporated and Huts further teaches the computer-implemented method of claim 3, further comprising: normalizing the set of Shapley values before applying the set of Shapley values to the LSTM model. (in [0010] …In some embodiments, the feature importance score includes a univariate feature importance score, a feature impact score, or a Shapley value. The actions of the method may further include, prior to determining the feature importance score for the aggregate image feature based on the feature importance scores of the constituent image features, normalizing and/or standardizing the feature importance scores [normalizing the set of Shapley values before applying the set of Shapley values to the LSTM model] for the constituent image features. And in [0143] As another example, the feature extraction module may include a pre-trained text feature extraction model, which may extract text features (e.g., low-, medium-, high-, and/or highest-level text features) from text data and/or natural language data. A pre-trained text feature extraction model may use a convolutional neural network (CNN), a recurrent neural network (e.g., RNN, including but not limited to long short-term memory (LSTM) RNN) [normalizing the set of Shapley values before applying the set of Shapley values to the LSTM model],…; And in[0266] Another type of explanatory visualization is the image embedding visualization… In some embodiments, feature importance metrics used by a model development system 100 and/or a model deployment system 1100 may include, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”) normalizing the set of Shapley values before applying the set of Shapley values to the LSTM model]. These metrics and some embodiments of techniques for assessing (or “scoring”) the feature importance of non-tabular features (e.g., image features) according to these metrics are described below… [0328] In some embodiments, the method 1900 further includes a step of normalizing and/or standardizing the feature importance scores for the constituent image features. The normalizing and/or standardizing may be performed prior [normalizing the set of Shapley values before applying the set of Shapley values to the LSTM model] to determining the feature importance score for the aggregate image feature.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Huts teaches applying the Shapley values based on the model type an associated model task as noted above and normalizing the set of Shapley values as part of the pre-feature engineering process as would be understood by a person of ordinary skill in the art.
Jill expressly teaches normalizing the set of Shapley values as part of the pre-feature engineering process as would be understood by a person of ordinary skill in the art, as depicted in Fig. 1
PNG
media_image1.png
574
830
media_image1.png
Greyscale
And in [0044] Referring to FIG. 1, in certain examples, the systems and methods described herein provide a complete technological solution for a large-scale data science workflow that includes several independent modules or components for data processing and exploration, feature engineering [normalizing the set of Shapley values before applying the set of Shapley values to the … model] and reduction, model development and selection, and model deployment and monitoring. In brief overview, the system 100 can include, interface, access, or otherwise use a data processing module 104... The system 100 can include, interface, access, or otherwise use a feature engineering module 106. The feature engineering module 106 can receive the processed/segmented data and perform feature engineering, feature reduction, and/or data partitioning… The features and partitioned data can be provided to the model development module 108 that develops and trains one or more predictive models [normalizing the set of Shapley values before applying the set of Shapley values to the … model]. The system 100 can include, interface, access, or otherwise use a model management module 110. The model management module 110 can deploy the models for end users and can monitor model performance and output model results…; And in [0090] Referring again to FIG. 2, once the feature derivation process has been performed, the feature engineering module 106 [ …before applying the set of Shapley values to the …model] can begin a feature reduction process in which one or more features that are redundant or not impactful can be removed or ignored from further consideration. Feature reduction can be performed using proprietary algorithms based on GBT (Gradient Boosting-based Tree) and/or SHAP (SHapley Additive exPlanations) algorithms [normalizing the set of Shapley values before applying the set of Shapley values to the … model]. Feature reduction can involve fitting a Light Gradient Boosting Machine model (LGBM) on all derived features (e.g., in a matrix having a size or shape of num_observations×num_features). A tree explainer (e.g., TreeExplainer from a SHAP library) can be fit using the LGBM model. Next, using the tree explainer, shapley values can be obtained for each observation from the data. The resulting shapley values can be provided in a matrix (e.g., having a size or shape of num_observations×num_features). Each value in the shapley values matrix can provide a measure of how much a feature contributes to a prediction. For example, the shapley value in element (i,j) of the matrix can be a contribution that feature j has on the prediction for observation (i). Next, a mean(abs(shapley values)) is calculated (e.g., in a direction along the rows) to obtain a single vector (of shape num_features) in which each value j is a mean contribution that feature j has on all the predictions. The vector can then be normalized to sum to 1 by dividing each value by a total sum of all elements in this vector, and the normalized vector can be sorted (e.g., from largest to smallest). Finally, features corresponding to large values in the normalized vector [normalizing the set of Shapley values before applying the set of Shapley values to the … model] can be retained and all other features can be eliminated or removed from further consideration. For example, the top features (e.g., features with the highest contributions) that sum to a specified threshold (e.g., 0.98) can be retained and used as a reduced set of features.
Jill, Huts and Oz are analogous art because both involve developing information retrieval and data processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing and implementing forecasting models based on Shapley feature engineering process as disclosed by Jill with the method of developing information retrieval models and approaches for understanding trained models as collectively disclosed by Huts and Oz.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Jill, Huts and Oz, as noted above; Doing so allows for building and testing the most accurate models, while allowing data scientists and analysts to perform model evaluation and assessments, (Jill, 0007).
Regarding claim 16, the limitations are similar to claim 4 limitations and rejected under the same rationale.
Claims 5-8 and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Öztireli et al. (US 20210042613, hereinafter ‘Oz’) in view of Huts et al. (US 20230067026, hereinafter ‘Huts’) in further view of Brouwers (US 20250044271, hereinafter ‘Brou’).
Regarding claim 5, the rejection of claim 3 is incorporated and Oz in combination with Huts further teaches the computer-implemented method of claim 3, further comprising: identifying a set of (in [0023] In general, each of the neurons in a non-input layer of the trained neural network 120 combines the inputted internal points based on learned parameter values (i.e., weights and biases) to generate an internal point a [further comprising: identifying a set of ]. Each of the neurons in a hidden layer can be configured to output the internal point a to the next layer or apply a non-linear activation function to the internal point a to generate an internal point b and then output the internal point b to the next layer. Each of the neurons in the output layer can be configured to output the internal point a as the associated output point 142 or apply a non-linear activation function [identifying a set of ] to the internal point a to generate an internal point b and then output the internal point b as the associated output point 142.[0024] Each of the internal point combinations and each of the non-linear activation functions executed in the layers of the trained neural network 120 [identifying a set of ] is referred to herein as a “point transformation,” …; [0068] In equations (10a) and (10b), W and b denote, respectively, the weights and the biases specified in the parameter value set 214(p), and W.sup.2 denotes the element-wise square of W. For explanatory purposes only, the element-wise square of a matrix or a vector is denoted herein with a post-pended, non-parenthetical superscript of 2. Notably, as persons skilled in the art will recognize, equations (10a) and (10b) can be modified to apply to other linear functions [generating an encoded activations representation by at least applying the set of], such as convolutions and mean pooling. [0069] The closed-form conversion 270(2) is applicable to rectified linear unit (“ReLU”) activations. A ReLU activation can be expressed as the following equation (11): x.sub.relu=max(0,x) (11) [0070] The probabilistic version of a ReLU activation is a rectified Gaussian distribution with mean and variance that can be expressed based on the closed-form conversion 270(2) that includes the following equations (12a) and (12b): … [0087] In sum, the disclosed techniques may be used to efficiently quantify what trained neural networks have learned. In one embodiment, a relevance application includes, without limitation, a neural network conversion engine and a Shapley value engine. The neural network conversion engine generates a probabilistic representation of a trained neural network. First, the neural network conversion engine converts the initial point transformation of the trained neural network to an input converter. The initial point transformation maps N input points included in an input point set to M internal points [mapping the encoded activations representation to an embedding space]…)
While Oz teaches the use of shapley values in embedding data into a an embedding space associated with a neural network.
Oz does not expressly teach the neural network as a LSTM-autoencoder model.
Brou does expressly teaches the neural network as a LSTM-autoencoder model, in [0066] Malhotra et al. (2016) propose to use an LSTM-based autoencoder to learn to reconstruct normal univariate time series behaviour of three publicly available data sets [generating an encoded activations representation by at least applying the set of LSTM activations to an autoencoder]. After learning normal behaviour, the reconstruction error is used to detect anomalous time series within power demand, space shuttle and electrocardiogramata. Their experiments show the model is able to detect both anomalies from short time-series as well as long time-series. In case of a multivariate time series data set, the authors first reduce the multivariate time series to univariate using the first principal component of PCA. Similar, Assendorp (2017) developed multiple LSTM-based autoencoder models for anomaly detection in washing cycles using multivariate sensor data… And in [0214] Hochreiter and Schmidhuber (1997) introduced an adaptation of classic RNNs to overcome its issues. Their introduced network is called the Long Short-Term Memory network (LSTM). Since its introduction, the networks have evolved and are now the most popular types of RNNs. LSTM is better capable of learning long term dependencies over substantial long time intervals without being affected by the vanishing or exploding gradient problem. The architecture of RNNs, as illustrated in FIG. 15, is adapted. Gates are activation functions [generating an encoded activations representation by at least applying the set of LSTM activations to an autoencoder], which can add or remove information from the cell state... An LSTM unit is a gated cell which contains information outside the flow of an RNN. The memory of an LSTM is the cell state. The LSTM unit decides what to store on the cell state. Using gates, which can be opened or closed, it determines when the cell state can be read, written or deleted. Gates are opened or closed based on a signal. The signals strength determines whether information is passed or blocked. Similar to classic RNNs, BPTT learning is used to adjust and optimize the weights associated with the gates, such that the LSTM network learns when to allow the reading, writing or deletion of information. A simplified representation of an LSTM unit is illustrated in FIG. 16. And in [0293] One major advantage of the hybrid anomaly detection method, which uses the reconstruction error per minute as input feature, is that interpretability is facilitated using SHaPley values. SHapley Additive explanations (SHAP) algorithm was first published by Lundberg and Lee (2017) and is a way to reverse-engineer the output of a machine learning algorithm…)
Brou, Huts and Oz are analogous art because both involve developing information retrieval and data processing techniques using machine learning systems and algorithms.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the prior art for developing information processing models using LSTM-based autoencoder as disclosed by Brou with the method of developing information retrieval models and approaches for understanding trained models as collectively disclosed by Huts and Oz.
One of ordinary skill in the arts would have been motivated to combine the disclosed methods disclosed by Brou, Huts and Oz, as noted above; Doing so allows for achieving an attention mechanism that is better at detecting false data injections compared to normal autoencoders or unsupervised one-class SVMs (Brou, 0068).
Regarding claim 6, the rejection of claim 5 is incorporated and Oz in combination with Huts and Brou teaches the computer-implemented method of claim 5, further comprising: outputting the embedding space. (in [0022] The trained neural network 120 includes, without limitation, any number of neurons that are arranged into a series of layers. The first layer of neurons is also referred to herein as the input layer, the last layer of neurons is also referred to herein as the output layer, and the remaining layers are also referred to herein as hidden layers. The input layer includes N neurons that each receives a different one of the input points 132 that is subsequently inputted into one or more of the neurons in the first hidden layer based on the connectivity of the trained neural network 120. Based on the “internal point(s)” received from the preceding layer [outputting the embedding space], each neuron in a hidden layer computes new internal points and inputs the new internal points to one or more of the neurons in the subsequent layer. The output layer includes M neurons that generate output points 142 based on the internal point(s) [outputting the embedding space] received from the preceding layer.)
Additionally, Huts teaches, in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations of model inferences [outputting the embedding space]. A feature's “feature importance” may indicate the feature's expected utility (on an absolute scale or relative to other features) for inferring the solution to a computer vision problem or data analytics problem. For example, a feature that is highly correlated with the target of a computer vision/data analytics problem generally has high expected utility for inferring the solution to that problem. Any suitable technique or metric may be used to assess feature importance including, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”). The foregoing techniques/metrics for assessing feature importance are described in further detail below… [0255] Neural network visualizations may be used to show key attributes of neural networks [outputting the embedding space]. Such attributes may include, without limitation, the number and sequence of layers in the network, each layer's type (e.g., input, activation, pooling, output, etc.), the number of inputs to each layer, the number of outputs from each layer, the type of activation function used by each activation layer, the type of pooling function used by each pooling layer, etc. An example of a neural network visualization is shown in FIG. 13…; And in [0259] The above-described examples of occlusion-based and multicolor-based image inference explanations are not limiting… More generally, any visualization that indicates the extent to which a model has relied on the various portions of an image in a data sample to generate an inference based on the data sample may be used. Some embodiments of image inference explanations may, for example, display arrows pointing to the significant portions of the image, with attributes of the arrows (e.g., length, line weight, color, etc.) indicating the level of significance of the portion of the image to which the arrow points. In some embodiments, a “topographical map” may be drawn on top of the image, such that areas of less significance are shown at “lower elevations” and areas of greater significance are shown at “higher elevations.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 7, the rejection of claim 5 is incorporated and Oz in combination with Huts and Brou teaches the computer-implemented method of claim 5, wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space. ([0050] For example, to understand why the trained neural network 120 misclassified a particular image as female, a user could configure the relevance application 140 to compute the Shapley value matrix 190 for the input point set 130 representing the image. The relevance application 140 could then display the Shapley value vector 192 corresponding to the output point 142 specifying the probability of female as a “Shapley image.” The relevance application 140 could set the color of each pixel in the Shapley image based on the Shapley value of the associated input point 132. Accordingly, the coloring of the Shapley image would visually illustrate [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space] the contribution of each of the inputs points 132 with respect to the misclassification. The user could subsequently enhance the training dataset based on the insights provided by the Shapley image and retrain the trained neural network 120 to improve the accuracy.)
Additionally, Huts teaches, in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations of model inferences [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space]. A feature's “feature importance” may indicate the feature's expected utility (on an absolute scale or relative to other features) for inferring the solution to a computer vision problem or data analytics problem. For example, a feature that is highly correlated with the target of a computer vision/data analytics problem generally has high expected utility for inferring the solution to that problem. Any suitable technique or metric may be used to assess feature importance including, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”). The foregoing techniques/metrics for assessing feature importance are described in further detail below… [0255] Neural network visualizations may be used to show key attributes of neural networks [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space]. Such attributes may include, without limitation, the number and sequence of layers in the network, each layer's type (e.g., input, activation, pooling, output, etc.), the number of inputs to each layer, the number of outputs from each layer, the type of activation function used by each activation layer, the type of pooling function used by each pooling layer, etc. An example of a neural network visualization is shown in FIG. 13…; And in [0259] The above-described examples of occlusion-based and multicolor-based image inference explanations are not limiting… More generally, any visualization [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space] that indicates the extent to which a model has relied on the various portions of an image in a data sample to generate an inference based on the data sample may be used. Some embodiments of image inference explanations may, for example, display arrows pointing to the significant portions of the image, with attributes of the arrows (e.g., length, line weight, color, etc.) indicating the level of significance of the portion of the image to which the arrow points. In some embodiments, a “topographical map” may be drawn on top of the image, such that areas of less significance are shown at “lower elevations” and areas of greater significance are shown at “higher elevations.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 8, the rejection of claim 7 is incorporated and Oz in combination with Hutz and Brou teaches the computer-implemented method of claim 7, wherein the plot representation of the embedding space is configured to receive highlight criteria, the computer-implemented method further comprising: visually distinguishing at least a portion of embedded representation in the embedding space based on the highlight criteria. ([0050] For example, to understand why the trained neural network 120 misclassified a particular image as female, a user could configure the relevance application 140 to compute the Shapley value matrix 190 for the input point set 130 representing the image. The relevance application 140 could then display the Shapley value vector 192 corresponding to the output point 142 specifying the probability of female as a “Shapley image.” The relevance application 140 could set the color of each pixel in the Shapley image based on the Shapley value of the associated input point 132. Accordingly, the coloring of the Shapley image would visually illustrate [wherein the plot representation of the embedding space is configured to receive highlight criteria,] the contribution of each of the inputs points 132 with respect to the misclassification. The user could subsequently enhance the training dataset [wherein the plot representation of the embedding space is configured to receive highlight criteria, the computer-implemented method further comprising: visually distinguishing at least a portion of embedded representation in the embedding space based on the highlight criteria] based on the insights provided by the Shapley image and retrain the trained neural network 120 to improve the accuracy. [0051] In another example, as part of an effort to generate a more efficient, smaller trained neural network 120, a user could configure the relevance application 140 to compute the Shapley value matrix 190 that included all the internal Shapley values for each of a wide variety of input point sets 130. Based on the Shapley value matrices, the user could determine [wherein the plot representation of the embedding space is configured to receive highlight criteria, the computer-implemented method further comprising: visually distinguishing at least a portion of embedded representation in the embedding space based on the highlight criteria] which of the neurons are contributing least to the output value sets 140 and, therefore, would be good candidates for pruning. And in [0078] The neural network conversion engine 150 may acquire the internal evaluation list 240 in any technically feasible fashion. For instance, in some embodiments, the relevance engine 140 determines the internal evaluation list 240 based on user input received via a graphical user interface (“GUI”) [wherein the plot representation of the embedding space is configured to receive highlight criteria, the computer-implemented method further comprising: visually distinguishing at least a portion of embedded representation in the embedding space based on the highlight criteria] and then transmits the internal evaluation list 240 to the neural network conversion engine 150. In alternate embodiments, the relevance application 140, the neural network conversion engine 150, the Shapley value engine 170, and the neural network wrapper 180 may be configured to compute and store Shapley values for any number of internal points associated with the trained neural network in any technically feasible fashion.)
Additionally, Huts teaches, in [0251] Some of the explanatory techniques described below rely on various “feature importance” metrics to generate visual explanations of model inferences [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space]. A feature's “feature importance” may indicate the feature's expected utility (on an absolute scale or relative to other features) for inferring the solution to a computer vision problem or data analytics problem. For example, a feature that is highly correlated with the target of a computer vision/data analytics problem generally has high expected utility for inferring the solution to that problem. Any suitable technique or metric may be used to assess feature importance including, without limitation, univariate feature importance, feature impact, and SHapley Additive exPlanations (“SHAP”). The foregoing techniques/metrics for assessing feature importance are described in further detail below… [0255] Neural network visualizations may be used to show key attributes of neural networks [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space]. Such attributes may include, without limitation, the number and sequence of layers in the network, each layer's type (e.g., input, activation, pooling, output, etc.), the number of inputs to each layer, the number of outputs from each layer, the type of activation function used by each activation layer, the type of pooling function used by each pooling layer, etc. An example of a neural network visualization is shown in FIG. 13…; And in [0259] The above-described examples of occlusion-based and multicolor-based image inference explanations are not limiting… More generally, any visualization [wherein outputting the embedding space comprises: causing rendering of an interface comprising at least a plot representation of the embedding space] that indicates the extent to which a model has relied on the various portions of an image in a data sample to generate an inference based on the data sample may be used. Some embodiments of image inference explanations may, for example, display arrows pointing to the significant portions of the image, with attributes of the arrows (e.g., length, line weight, color, etc.) indicating the level of significance of the portion of the image to which the arrow points. In some embodiments, a “topographical map” may be drawn on top of the image, such that areas of less significance are shown at “lower elevations” and areas of greater significance are shown at “higher elevations.”
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the teachings of Huts and Oz for the same reasons disclosed above.
Regarding claim 17, the limitations are similar to claim 5 limitations and rejected under the same rationale.
Regarding claim 18, the limitations are similar to claim 6 limitations and rejected under the same rationale.
Regarding claim 19, the limitations are similar to claim 7 limitations and rejected under the same rationale.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Kollada et al. (US 20220392637): teaches multimodal and multi-sensor diagnostic devices, that utilize machine learning algorithms by computing Shapley values based on a Shapley additive explanations (SHAP) technique. SHapley Additive exPlanations (SHAP), an additive feature attribution method is used to explain the output of the machine learning model. SHAP based interpretability method enabled the computation of the SHAP values for all the input features (which are time series features) over each point of time. For examples, clinicians could use these type of data to isolate the impacts of specific words, facial or vocal expressions, or to find particularly salient portions of the video in larger clips.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWATOSIN ALABI whose telephone number is (571)272-0516. The examiner can normally be reached Monday-Friday, 8:00am-5:00pm EST..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWATOSIN ALABI/Primary Examiner, Art Unit 2129