This page provides a high level overview on algorithms and their features currently implemented in TSInterpret.
+| Method | +Backend | +Type | +Data | +Training Set | +
|---|---|---|---|---|
| COMTE 1 | +TF,PYT, SK, BB | +InstanceBased | +multi | +y | +
| LEFTIST 2 | +TF,PYT, SK, BB | +FeatureAttribution | +uni | +y | +
| NUN-CF 3 | +TF,PYT, SK | +InstanceBased | +uni | +y | +
| TSR 4 | +TF,PYT | +FeatureAttribution | +multi | +n | +
BB: black-box, TF: tensorflow, PYT: PyTorch, SK: sklearn, uni: univariate time series, multi: multivariate time series, y: yes, n: no
+ +NUN-CF Delany et al.3 proposed using a native guide to generate counterfatuals. The native guide is selected via K-nearest neighbor. For generating the counterfactual they offer three options: the native guide, the native guide with bary centering, and the counterfactual based on the native guide and class activation mapping.
+COMTE Ates et al. 1 proposed COMTE as a heuristic pertubations approach for multivariate time series counterfactuals. The goal is to change a small number of features to obtain a counterfactual.
+LEFTIST Agnostic Local Explanation for Time Series Classification by Guilleme et al.2 adapted LIME for time series classification task and propose to use prefixed (both the length and the position) shapelets as the interpretable components, and provide the feature importance of each shapelet.
+TSR Temporal Saliency Rescaling Ismail et al. 4 calculates the importance of each timestep, followed by the feature importance on basis of different Saliency Methods, both Back-propagation based and perturbation based. For a full list of implemented methods, we refer the reader to our code documentation. The implementation in TSInterpret is based on tf-explain 5, shap and captum [@kokhlikyan_captum_2020].
+Emre Ates, Burak Aksar, Vitus J. Leung, and Ayse K. Coskun. Counterfactual Explanations for Machine Learning on Multivariate Time Series Data. 2021 International Conference on Applied Artificial Intelligence (ICAPAI), pages 1–8, May 2021. arXiv: 2008.10781. URL: http://arxiv.org/abs/2008.10781 (visited on 2022-03-25), doi:10.1109/ICAPAI49758.2021.9462056. ↩↩
+Mael Guilleme, Veronique Masson, Laurence Roze, and Alexandre Termier. Agnostic Local Explanation for Time Series Classification. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), 432–439. Portland, OR, USA, November 2019. IEEE. ↩↩
+Eoin Delaney, Derek Greene, and Mark T. Keane. Instance-Based Counterfactual Explanations for Time Series Classification. In Antonio A. Sánchez-Ruiz and Michael W. Floyd, editors, Case-Based Reasoning Research and Development, volume 12877, pages 32–47. Springer International Publishing, Cham, 2021. ↩↩
+Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Deep Learning Interpretability in Time Series Predictions. arXiv:2010.13924 [cs, stat], October 2020. arXiv: 2010.13924. ↩↩
+Raphael Meudec. Tf-explain. February 2021. URL: https://zenodo.org/record/5711704 (visited on 2022-05-24), doi:10.5281/ZENODO.5711704. ↩
+TSInterpret works with Python 3.6+ and can be installed from PyPI or cloning the github repository by following the instructions below.
+Via PyPI:
+pip install tsinterpret
+You can install the latest development version from GitHub as so:
+pip install https://github.com/fzi-forschungszentrum-informatik/TSInterpret.git --upgrade
+Or, through SSH:
+pip install git@github.com:fzi-forschungszentrum-informatik/TSInterpret.git --upgrade
+Feel welcome to open an issue on GitHub if you are having any trouble.
+TSInterpret takes inspiration from scikit-learn, consisting of distinct initialize, explain and plot steps. We will use the Nun-CF method on to illustrate the API.
+First, we import the explainer:
+from TSInterpret.InterpretabilityModels.counterfactual.NativeGuideCF import NativeGuideCF
+Next, we initialize it by passing it a model (or in this case also possible a predict function) and any other necessary arguments:
+exp_model=NativeGuideCF(model,shape,(train_x,train_y), backend='PYT', mode='feat',method='dtw_bary_center')
+Finally, we can call the explainer on a test instance which will return an Tuple or a list containing the explanation:
+exp,label=exp_model.explain(item, np.argmax(y_target,axis=1))
+Afterwads we can use the Plot Function to obtain a visualiation of the explanation:
+exp_model.plot(item,np.argmax(y_target,axis=1)[0],exp,label)
+The input and output details vary for each method. Therefore, getting familiar with the different algorithms makes sense.
+ + + + + + +Explainable Artificial Intelligence (XAI) is an emerging field trying to make AI systems more understandable to humans. The goal of XAI, according to DARPA1, is to “produce more explainable models, while maintaining a high level of learning performance (prediction accuracy); and enable human users to understand, appropriately, trust, and effectively manage the emerging generation of artificially intelligent partners”. Especially on deep networks in high-stake scenarios understanding the decision process or the decision is crucial to prevent harm (e.g., autonomous driving or anything health-related).
+Multiple terms are strongly related to XAI. Most famously "Explainability" and "Interpretability". Both terms are often used interchangeably with no consent on definitions existing the literature.
+"Interpretability" in the context of TSInterpret refers to the ability to support user understanding and comprehension of the model decision-making process and predictions. Used to provide user uunderstanding of model decisions, "Explainability" algorithms are used. Thereby, it is often the case that multiple explainability algorithms are necessary for the user to understand the decision process.
+"Explainability" tries to provide algorithms that give insights into model predictions:
+How does a prediction change dependent on feature inputs?
+What features are or are not important for a given prediction?
+What features would you have to change minimally to obtain a new prediction of your choosing?
+How does each feature contribute to a model’s prediction?
+Interpretability is the end goal. Explanations and explainability are tools to reach interpretability 2.
+TSInterpret provides a set of algorithms or methods known as explainers specifically for time series. Each explainer provides a different kind of insight about a model (- i.e., answers different questions). The set of algorithms available to a specific model depends on several factors. For instance, some approaches need a gradient to function and can only be applied to models providing such. A full listing can be found in the section Algorithm Overview.
+As machine learning methods have become increasingly complex, with many practitioners applying machine and, specifically, deep learning methods, the need to understand the decisions made by models is only increasing.
+Trust: Explanations can build trust in the machine learning systems and increase social acceptance by providing insights into the basis of a decision.
+Debugging and Auditing: An explanation for an erroneous prediction helps to understand the cause of the error (e.g., by showing the model focus) and delivers a direction for how to fix the system. Further, by computing feature attributions toward a model's prediction, users can check whether those attributions are consistent with their understanding.
+Research: Explainability allows us to understand how and why models make decisions, thereby helping to understand the effects of the particular model or training schema.
+Explanations Methods and Techniques for Model Interpretability can be classified according to different criteria. In this section, we only introduce the criterias most relevant to TSInterpret.
+Instrinct Interpretability refers to models that are interpretable by design. This can be achieved by constraining model complexity or including explanation components in the model design.
+Post-Hoc Interpretability refers to explanation methods applied after model training and are usually decoupled from the model.
+Model-Specific methods are limited to specific model classes and usually rely on a specific model internal (e.g., Gradients).
+Model-Agnostic methods can be applied to any model and rely on analyzing the connection between inputs and output. Those methods cannot access the model's internal functions.
+
Take, for example, a Decision Support System to classify heart rates as depicted in the figure below. While the data scientist knows that the machine learning model can obtain an accuracy of over 90 % to classify a heart rate as abnormal or normal, the decision process of such a system is still intransparent resulting in unsureness about the decision process of a model. Wrong Classifications in both directions can have long-lasting effects on a patient relying on the system. If a heart rate is wrongly classified as normal, a patient will not get the necessary medication. If a heart rate is wrongly classified as Abnormal, a patient would get medication and endure side effects although his heart might still be health +To make this decision process more opaque, a data scientist might decide to use algorithms for explainable and interpretable machine learning, to learn: + - Which features are important. + - Which feature influences the decision of a model positively or negatively. + - How a counter-example would look like.
+David Gunning. Explainable artificial intelligence (xai). Defense advanced research projects agency (DARPA), nd Web, 2(2):1, 2017. ↩
+Milo R Honegger. Shedding light on black box machine learning algorithms. Development of an Axiomatic Framework to Assess the Quality of Methods that Explain Individual Predictions, MA Karlsruhe, 2018. ↩
+CF(mlmodel, mode)
+
+
+ Bases: InstanceBase
Abstract class to implement Coutnterfactual Methods for time series data
+ +Initialization of CF. +Arguments: + mlmodel [torch.nn.Module, Callabale, tf.keras.Model]: Machine Learning Model to be explained. + mode str : Second dimension is feature --> 'feat', is time --> 'time'
+ + + + +explain()
+
+Explains instance or model.
+ +plot(original, org_label, exp, exp_label, vis_change=True, all_in_one=False, save_fig=None, figsize=(6.4, 4.8))
+
+Basic Plot Function for visualizing Coutnerfactuals.
+Arguments:
+ original np.array: Instance to be explained. Shape: mode = time -> (time, feat) or mode = time -> (feat, time)
+ org_label int: Label of instance to be explained.
+ exp np.array: Explanation. mode = time -> (time, feat) or mode = time -> (feat, time)
+ exp_label int: Label of Explanation.
+ vis_change bool: Change to be visualized as heatmap.
+ all_in_one bool: Original and Counterfactual in one plot.
+ save_fig str: Path to save fig at.
plot_in_one(item, org_label, exp, cf_label, save_fig=None, figsize=(6.4, 4.8))
+
+Plot Function for Counterfactuals in uni-and multivariate setting. In the multivariate setting only the changed features are visualized.
+Arguments:
+ item np.array: original instance. Shape: mode = time -> (time, feat) or mode = time -> (feat, time)
+ org_label int: originally predicted label.
+ exp np.array: returned explanation. Shape: mode = time -> (time, feat) or mode = time -> (feat, time)
+ cf_label int: lebel of returned instance.
+ figsize Tuple: Size of Figure (x,y).
+ save_fig str: Path to Save the figure.
plot_multi(item, org_label, exp, cf_label, figsize=(6.4, 4.8), save_fig=None)
+
+Plot Function for Ates et al., used if multiple features are changed in a Multivariate Setting.
+Also called via plot_in_one. Preferably, do not use directly.
+Arguments:
+ item np.array: original instance. Shape: mode = time -> (time, feat) or mode = time -> (feat, time)
+ org_label int: originally predicted label.
+ exp np.array: returned explanation. Shape: mode = time -> (time, feat) or mode = time -> (feat, time)
+ cf_label int: lebel of returned instance.
+ figsize Tuple: Size of Figure (x,y).
+ save_fig str: Path to Save the figure.
COMTECF(model, data, backend, mode, method='opt', number_distractors=2, max_attempts=1000, max_iter=1000, silent=False)
+
+
+ Bases: CF
Calculates and Visualizes Counterfactuals for Multivariate Time Series in accordance to the paper [1].
+[1] Ates, Emre, et al. + "Counterfactual Explanations for Multivariate Time Series." + 2021 International Conference on Applied Artificial Intelligence (ICAPAI). IEEE, 2021.
+| PARAMETER | +DESCRIPTION | +
|---|---|
model |
+
+
+
+ Model to be interpreted. +
+
+ TYPE:
+ |
+
ref |
+
+
+
+ Reference Dataset as Tuple (x,y). +
+
+ TYPE:
+ |
+
backend |
+
+
+
+ desired Model Backend ('PYT', 'TF', 'SK'). +
+
+ TYPE:
+ |
+
mode |
+
+
+
+ Name of second dimension:
+
+ TYPE:
+ |
+
method |
+
+
+
+ 'opt' if optimized calculation, 'brute' for Brute Force +
+
+ TYPE:
+ |
+
number_distractors |
+
+
+
+ number of distractore to be used +
+
+ TYPE:
+ |
+
silent |
+
+
+
+ logging. +
+
+ TYPE:
+ |
+
explain(x, orig_class=None, target=None)
+
+Calculates the Counterfactual according to Ates.
+Arguments:
+ x (np.array): The instance to explain. Shape : mode = time -> (1,time, feat) or mode = time -> (1,feat, time)
+ target int: target class. If no target class is given, the class with the secon heighest classification probability is selected.
| RETURNS | +DESCRIPTION | +
|---|---|
+
+ ([array], int)
+
+ |
+
+
+
+ Tuple of Counterfactual and Label. Shape of CF : |
+
LEFTIST(model, data=None, mode='time', backend='F', transform_name='straight', segmentator_name='uniform', learning_process_name='Lime', nb_interpretable_feature=10, nb_neighbors=1000, explanation_size=1)
+
+
+ Bases: FeatureAttribution
Local explainer for time series classification. Wrapper for LEFTIST from [1].
+[1] Guillemé, Maël, et al. "Agnostic local explanation for time series classification." +2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2019.
+Initization.
+Arguments:
+ model_to_explain [torch.nn.Module, Callable, tf.keras.model]: classification model to explain.
+ data Tuple: Reference Dataset as Tuple (x,y).
+ mode str: Name of second dimension: time -> (-1, time, feature) or feat -> (-1, feature, time)
+ backend str: TF, PYT or SK
+ transform_name str: Name of transformer
+ learning_process_name str: 'Lime' or 'Shap'
+ nb_interpretable_feature int: number of desired features
+ nb_neighbors int: number of neighbors to generate.
+ explanation_size int: number of feature to use for the explanations
explain(instance, idx_label=None, random_state=0)
+
+Compute the explanation.
+ + + +| PARAMETER | +DESCRIPTION | +
|---|---|
instance |
+
+
+
+ item to be explained. Shape :
+
+ TYPE:
+ |
+
idx_label |
+
+
+
+ index of label to explain. If None, return an explanation for each label. +
+
+ TYPE:
+ |
+
random_state |
+
+
+
+ fixes seed +
+
+ TYPE:
+ |
+
| RETURNS | +DESCRIPTION | +
|---|---|
+ List
+ |
+
+
+
+ Attribution weight |
+
Implementation after Delaney et al . https://github.com/e-delaney/Instance-Based_CFE_TSC
+ + + +NativeGuideCF(model, data, backend='PYT', mode='feat', method='NUN-CF', distance_measure='dtw', n_neighbors=1, max_iter=500)
+
+
+ Bases: CF
NUN_CF according to [1] for both torch and tensorflow.
+[1] Delaney, E., Greene, D., Keane, M.T.: Instance-Based Counterfactual Explanations + for Time Series Classification. In: Sanchez-Ruiz, A.A., Floyd, M.W. (eds.) Case- + Based Reasoning Research and Development, vol. 12877, pp. 32–47. Springer + International Publishing, Cham (2021), series Title: Lecture Notes in Computer + Science.
+In this case differentiation between time & feat not necessary as implicitly given by CNN. Only works for CNNs due to the attribution methods.
+Arguments:
+ model [torch.nn.Module, Callable, tf.keras.model]: classification model to explain
+ data Tuple: reference set as tuple (x,y)
+ backend str: 'PYT' or 'TF'
+ mode str: model either 'time' or 'feat'. time -> (-1, time, feature) or feat -> (-1, feature, time)
+ method str: 'Nun_CF', 'dtw_bary_center' or 'native_guide'.
+ distance_measure str: sklearn appreviation for distance of knn.
+ n_neighbore int: # neighbors to select from.
+ max_iter int : max number of runs
explain(x, y)
+
+' +Explains a specific instance x. +Arguments: + x np.array : instance to be explained. + y int: predicted label for instance x. +Returns: + Tuple: (counterfactual, counterfactual label)
+ +SETSCF(model, data, backend, mode, min_shapelet_len, max_shapelet_len, num_candidates_to_sample_per_case=20, time_contract_in_mins_per_dim=10, predefined_ig_rejection_level=0.001, max_shapelets_to_store_per_class=30, random_state=42, remove_self_similar=True, silent=False, fit_shapelets=True)
+
+
+ Bases: CF
Calculates and Visualizes Counterfactuals for Multivariate Time Series in accordance to the paper [1]. + The shapelet transofor adapted by [1] is based on prior work of [2].
+[1] Bahri, Omar, et al. +"Shapelet-based Temporal Association Rule Mining for Multivariate Time Series Classification". SIGKDD 2022 Workshop on Mining and Learning from Time Series (MiLeTS)" +[2] ostrom, Aaron and Bagnall, Anthony}, +"Binary shapelet transform for multiclass time series". Bostrom, Aaron and Bagnall, Anthony
+| PARAMETER | +DESCRIPTION | +
|---|---|
model |
+
+
+
+ Model to be interpreted. +
+
+ TYPE:
+ |
+
dataset |
+
+
+
+ Reference Dataset of training and test data. ++ + |
+
backend |
+
+
+
+ desired Model Backend ('PYT', 'TF', 'SK'). +
+
+ TYPE:
+ |
+
mode |
+
+
+
+ Name of second dimension:
+
+ TYPE:
+ |
+
min_shapelet_len |
+
+
+
+ Value for min length of extracted shapelets / must be greater than 0 +
+
+ TYPE:
+ |
+
max_shapelet_len |
+
+
+
+ Value for max length of extracted shapelets < timeseries must be smaller or equal than timeseries length +
+
+ TYPE:
+ |
+
num_candidates_to_sample_per_case |
+
+
+
+ number of assesed candiates per series +
+
+ TYPE:
+ |
+
time_contract_in_mins_per_dim |
+
+
+
+ Max time for shapelet extraction per dimension +
+
+ TYPE:
+ |
+
predefined_ig_rejection_levl |
+
+
+
+ Min Information Gain of candidate shapelet to keep +
+
+ TYPE:
+ |
+
random_state |
+
+
+
+ RandomState used throughout the shapelet transform +
+
+ TYPE:
+ |
+
remove_self_similar |
+
+
+
+ removes similar shapelets from a timeseries +
+
+ TYPE:
+ |
+
initial_num_shapelets_per_case |
+
+
+
+ Initial number of shapelets per case. +
+
+ TYPE:
+ |
+
silent |
+
+
+
+ logging. +
+
+ TYPE:
+ |
+
explain(x, orig_class=None, target=None)
+
+Calculates the Counterfactual according to Ates.
+Arguments:
+ x (np.array): The instance to explain. Shape : mode = time -> (1,time, feat) or mode = time -> (1,feat, time)
+ target int: target class. If no target class is given, the class with the secon heighest classification probability is selected.
| RETURNS | +DESCRIPTION | +
|---|---|
+
+ ([array], int)
+
+ |
+
+
+
+ Tuple of Counterfactual and Label. Shape of CF : |
+
fit(occlusion_threshhold=0.1, remove_multiclass_shapelets=True)
+
+Calculates the occurences of shapelets and removes shapelets belonging to more than one class. This process can be triggered with different parameter options without a new shapelet transform run.
+Arguments:
+ x (np.array): The instance to explain. Shape : mode = time -> (1,time, feat) or mode = time -> (1,feat, time)
+ target int: target class. If no target class is given, the class with the secon heighest classification probability is selected.
| RETURNS | +DESCRIPTION | +
|---|---|
+
+ ([array], int)
+
+ |
+
+
+
+ Tuple of Counterfactual and Label. Shape of CF : |
+
FeatureAttribution(mlmodel, mode)
+
+
+ Bases: InterpretabilityBase
mlmodel:
+explain: + explains a instance +Returns
+None
+ + + + +explain()
+
+
+ abstractmethod
+
+
+Explains instance or model. +Parameters
+instance: np.array + Not encoded and not normalised factual examples in two-dimensional shape (m, n). +Returns
+pd.DataFrame + Encoded and normalised counterfactual examples.
+ +plot(item, exp, figsize=(6.4, 4.8), heatmap=False, normelize_saliency=True, vmin=-1, vmax=1, save=None)
+
+Plots expalantion on the explained Sample. +Parameters
+instance: np.array + timeseries instance in two-dimensional shape (m, n). +exp: expalantaion +Returns
+matplotlib.pyplot.Figure + Visualization of Explanation
+Saliency(model, NumTimeSteps, NumFeatures, method='GRAD', mode='time')
+
+
+ Bases: FeatureAttribution
Base Method for Saliency Calculation based on [1]. +Please use the designated Subclasses SaliencyMethods_PYT.py +for PyTorch explanations and SaliencyMethods_TF.py +for Tensforflow explanations.
+[1] Ismail, Aya Abdelsalam, et al. +"Benchmarking deep learning interpretability in time series predictions." +Advances in neural information processing systems 33 (2020): 6441-6452.
+| PARAMETER | +DESCRIPTION | +
|---|---|
model |
+
+
+
+ model to be explained. +
+
+ TYPE:
+ |
+
NumTimeSteps |
+
+
+
+ number of timesteps. +
+
+ TYPE:
+ |
+
NumFeatures |
+
+
+
+ number of features. +
+
+ TYPE:
+ |
+
method |
+
+
+
+ Saliency Method to be used. +
+
+ TYPE:
+ |
+
mode |
+
+
+
+ Second dimension 'time'->
+
+ TYPE:
+ |
+
plot(item, exp, figsize=(6.4, 4.8), heatmap=False, save=None)
+
+Plots explanation on the explained Sample.
+ + + +| PARAMETER | +DESCRIPTION | +
|---|---|
item |
+
+
+
+ instance to be explained,if
+
+ TYPE:
+ |
+
exp |
+
+
+
+ explanation, ,if
+
+ TYPE:
+ |
+
figsize |
+
+
+
+ desired size of plot. +
+
+ TYPE:
+ |
+
heatmap |
+
+
+
+ 'True' if only heatmap, otherwise 'False'. +
+
+ TYPE:
+ |
+
save |
+
+
+
+ Path to save figure. +
+
+ TYPE:
+ |
+
TSEvo(model, data, transformer='authentic_opposing_information', epochs=500, mode='time', backend='PYT', verbose=0)
+
+
+ Bases: CF
Calculates and Visualizes Counterfactuals for Uni- and Multivariate Time Series in accordance to the paper [1].
+[1] Höllig, Jacqueline , et al. + "TSEvo: Evolutionary Counterfactual Explanations for Time Series Classification." + 21st IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2022.
+| PARAMETER | +DESCRIPTION | +
|---|---|
model |
+
+
+
+ Model to be interpreted. +
+
+ TYPE:
+ |
+
data |
+
+
+
+ Reference Dataset as Tuple (x,y). +
+
+ TYPE:
+ |
+
transformer |
+
+
+
+ ['authentic_opposing_information','mutate_both','mutate_mean','frequency_band_mapping'] +
+
+ TYPE:
+ |
+
epochs |
+
+
+
+ Maximal Number of Itertions +
+
+ TYPE:
+ |
+
mode |
+
+
+
+ Name of second dimension: time -> (-1, time, feature) or feat -> (-1, feature, time) +
+
+ TYPE:
+ |
+
backend |
+
+
+
+ desired Model Backend ('PYT', 'TF', 'SK'). +
+
+ TYPE:
+ |
+
verbose |
+
+
+
+ Logging Level +
+
+ TYPE:
+ |
+
explain(original_x, original_y, target_y=None)
+
+Entry Point to explain a instance.
+Arguments:
+ original_x (np.array): The instance to explain. Shape mode = time -> (1,time, feat) or mode = time -> (1,feat, time)
+ original_y (np.array): Classification Probability of instance.
+ target_y int: Class to be targeted
| RETURNS | +DESCRIPTION | +
|---|---|
| + | +
+
+
+ [np.array, int]: Returns the Counterfactual and the class. Shape of CF : |
+
TSR
+
+
+Wrapper Class for Saliency Calculation. +Automatically calls the corresponding PYT or TF implementation.
+ + + + +__new__(model, NumTimeSteps, NumFeatures, method='GRAD', mode='time', device='cpu')
+
+Initialization
+Arguments:
+ model [torch.nn.Module, tf.keras.Model]: model to be explained
+ NumTimeSteps int : Number of Time Step
+ NumFeatures int : Number Features
+ method str: Saliency Methode to be used
+ mode str: Second dimension 'time'->(1,time,feat) or 'feat'->(1,feat,time)
Saliency_PTY(model, NumTimeSteps, NumFeatures, method='GRAD', mode='time', tsr=True, device='cpu')
+
+
+ Bases: Saliency
PyTorch Implementation for Saliency Calculation based on [1]. The Saliency Methods are based on the library captum [2]. +For PyTorch the following saliency methods are available: + + Gradients (GRAD) + + Integrated Gradients (IG) + + Gradient Shap (GS) + + DeepLift (DL) + + DeepLiftShap (DLS) + + SmoothGrad (SG) + + Shapley Value Sampling(SVS) + + Feature Ablatiom (FA) + + Occlusion (FO) +References
+[1] Ismail, Aya Abdelsalam, et al. +"Benchmarking deep learning interpretability in time series predictions." +Advances in neural information processing systems 33 (2020): 6441-6452. +[2] Kokhlikyan, Narine, et al. +"Captum: A unified and generic model interpretability library for pytorch." +arXiv preprint arXiv:2009.07896 (2020).
+Initialization
+Arguments:
+ model [torch.nn.Module]: model to be explained
+ NumTimeSteps int : Number of Time Step
+ NumFeatures int : Number Features
+ method str: Saliency Methode to be used
+ mode str: Second dimension 'time'->(1,time,feat) or 'feat'->(1,feat,time)
explain(item, labels, TSR=None, **kwargs)
+
+Method to explain the model based on the item.
+Arguments:
+ item np.array: item to get feature attribution for, if mode = time->(1,time,feat) or mode = feat->(1,feat,time)
+ labels int: label
+ TSR bool: if True time series rescaling according to [1] is used, else plain (scaled) weights are returened
+Returns:
+ np.array: feature attribution weights mode = time->(time,feat) or mode = feat->(feat,time)
Saliency_TF(model, NumTimeSteps, NumFeatures, method='GRAD', mode='time', tsr=True, device='cpu')
+
+
+ Bases: Saliency
Tensorflow Implementation for Saliency Calculation based on [1]. +The Saliency Methods are based on the library tf-explain [2] and shap [3]. +For Tensorflow the following saliency methods are available: + + Gradients (GRAD) + + Integrated Gradients (IG) + + Gradient Shap (GS)) + + DeepLiftShap (DLS) + + SmoothGrad (SG) + + Occlusion (FO)
+Attention: GS and DLS only work for Python < 3.10.
+[1] Ismail, Aya Abdelsalam, et al. +"Benchmarking deep learning interpretability in time series predictions." +Advances in neural information processing systems 33 (2020): 6441-6452.
+[2] Meudec, Raphael: , tf-explain. https://github.com/sicara/tf-explain
+[3] Lundberg, Scott M., and Su-In Lee. +"A unified approach to interpreting model predictions." +Advances in neural information processing systems 30 (2017). + https://shap.readthedocs.io/
+| PARAMETER | +DESCRIPTION | +
|---|---|
model |
+
+
+
+ model to be explained +
+
+ TYPE:
+ |
+
NumTimeSteps |
+
+
+
+ Number of Time Step +
+
+ TYPE:
+ |
+
NumFeatures |
+
+
+
+ Number Features +
+
+ TYPE:
+ |
+
method |
+
+
+
+ Saliency Methode to be used +
+
+ TYPE:
+ |
+
mode |
+
+
+
+ Second dimension 'time'->
+
+ TYPE:
+ |
+
explain(item, labels, TSR=None)
+
+Method to explain the model based on the item.
+Arguments:
+ item np.array: item to get feature attribution for, if mode = time->(1,time,feat) or mode = feat->(1,feat,time)
+ labels int: label
+ TSR bool: if True time series rescaling according to [1] is used, else plain (scaled) weights are returened
+Returns:
+ np.array: feature attribution weights mode = time->(time,feat) or mode = feat->(feat,time)
BaseModel(model=None, change=False, model_path='', backend='Func')
+
+
+ Bases: ABC
Initialize Base Model. +Arguments: + model: trained ML Model, either the model or the direct function call for returning the probability distribution. + change bool: True if dimension change is necessary. + model_path str: path to trained model. + backend str: ML framework. For frameworks other than TensorFlow (TF), Sklearn (SK) or PyTorch (PYT), + provide 'Func'.
+ + + + +load_model(path)
+
+
+ abstractmethod
+
+
+Loads the model provided at the given path. +Arguments: + path str: Path to the trained model- + Returns: + loaded model.
+ +predict(item)
+
+
+ abstractmethod
+
+
+Unified prediction function. +Arguments: + item np.array: item to be classified + Returns: + an array of output scores for a classifier, and a singleton array of predicted value for a regressor.
+ +TensorFlowModel(model, change=False)
+
+
+ Bases: BaseModel
Wrapper for Tensorflow Models that unifiy the prediction function for a classifier. +Arguments: + model : Trained TF Model. + change bool: if swapping of dimension is necessary = True
+ + + + +predict(item)
+
+Unified prediction function. +Arguments: + item np.array: item to be classified. + Returns: + an array of output scores for a classifier.
+ +Module containing an interface to trained PyTorch model.
+ + + +PyTorchModel(model, change=False)
+
+
+ Bases: BaseModel
Wrapper for PyTorch Models that unifiy the prediction function for a classifier. +Arguments: + model : Trained PYT Model. + change bool: if swapping of dimension is necessary = True
+ + + + +predict(item)
+
+Unified prediction function. +Arguments: + item np.array: item to be classified. + Returns: + an array of output scores for a classifier.
+ +SklearnModel(model, change=False)
+
+
+ Bases: BaseModel
Wrapper for Sklearn Models that unifiy the prediction function for a classifier. +Arguments: + model : Trained Sklearn Model. + change bool: if swapping of dimension is necessary = True
+ + + + +predict(item)
+
+Unified prediction function. +Arguments: + item np.array: item to be classified. + Returns: + an array of output scores for a classifier.
+ +import numpy as np
+from tslearn.datasets import UCR_UEA_datasets
+from tslearn.svm import TimeSeriesSVC
+from tslearn.preprocessing import TimeSeriesScalerMinMax
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='BasicMotions'
+
+
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x = TimeSeriesScalerMinMax().fit_transform(train_x)
+test_x = TimeSeriesScalerMinMax().fit_transform(test_x)
+
+
+
+model = TimeSeriesSVC(kernel="gak", gamma="auto", probability=True)
+model.fit(train_x, train_y)
+print("Correct classification rate:", model.score(test_x, test_y))
+
+
+item=test_x[10].reshape(1,test_x.shape[1],test_x.shape[2])
+shape=item.shape
+y_target= model.predict_proba(item)
+pred_y= model.predict_proba(train_x)
+Correct classification rate: 0.925 ++
from TSInterpret.InterpretabilityModels.counterfactual.COMTECF import COMTECF
+exp_model= COMTECF(model,(train_x,pred_y),mode='time', backend='SK', method= 'opt')
+exp = exp_model.explain(item)
+array, label=exp
+
+
+%matplotlib inline
+org_label=np.argmax(y_target)
+cf_label=label[0]
+exp=array
+
+exp_model.plot_in_one(item,org_label,exp,cf_label,figsize=(15,15))
+2023-08-08 14:05:08.336311: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-08-08 14:05:09.740330: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
<class 'numpy.ndarray'> +<class 'numpy.ndarray'> +(6, 100) +(6, 100) ++
+
+import sklearn
+import pickle
+import numpy as np
+from tslearn.datasets import UCR_UEA_datasets
+import tslearn
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='BasicMotions'
+tslearn.datasets.CachedDatasets().list_datasets()
+['Trace']+
train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+print(train_x.shape)
+print(test_x.shape)
+(40, 100, 6) +(40, 100, 6) ++
enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder_tf.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
import tensorflow as tf
+model = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/BasicMotionsbest_model.hdf5')
+2023-09-18 14:42:58.962386: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 14:42:59.585681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT +2023-09-18 14:43:00.261652: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 +2023-09-18 14:43:00.262132: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. +Skipping registering GPU devices... ++
label_pred=np.argmax(model.predict(test_x),axis=1)
+label_pred
+2/2 [==============================] - 0s 2ms/step ++
array([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 3, 3, + 3, 3, 3, 3, 3, 3, 3, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])+
item=test_x[0].reshape(1,test_x.shape[1],test_x.shape[2])
+shape=item.shape
+y_target= model.predict(item)
+1/1 [==============================] - 0s 16ms/step ++
y_target
+array([[2.8015620e-05, 1.2882984e-03, 9.9654043e-01, 6.2079956e-03]], + dtype=float32)+
from TSInterpret.InterpretabilityModels.counterfactual.COMTECF import COMTECF
+
+exp_model= COMTECF(model,(train_x,train_y),mode='time', backend='TF', method= 'brute')
+exp = exp_model.explain(item)
+1/1 [==============================] - 0s 21ms/step +1/1 [==============================] - 0s 14ms/step +2/2 [==============================] - 0s 2ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 16ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step ++
array, label=exp
+%matplotlib inline
+org_label=np.argmax(y_target)
+cf_label=label[0]
+exp=array
+print(exp.shape)
+exp_model.plot_in_one(np.array(item[0]),org_label,np.array(exp[0]),cf_label)
+(1, 100, 6) ++
+A counterfactual explanation, originally introduced to machine learning by [2], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+Counterfactual Explanations for Machine Learning on Multivariate Time Series Data (COMTE) developed by Ates et al. [1] builds counterfactuals in multivariate setting by perturbing the features of a time series with the help of an heuristic algorithm. They adapted the original formualtion of Wachter et al. [2] $L= \lambda (f(x^{cf}-y^{cf}))+d(x,x^{cf})$ by replacing the point wise distance function $d$ with a function on feature basis.
+
The code in TSInterpret is based on the authors implementation .
+[1] Emre Ates, Burak Aksar, Vitus J. Leung, and Ayse K. Coskun. Counterfactual Explanations for Machine Learning on Multivariate Time Series Data. 2021 International Conference on Applied Artificial Intelligence (ICAPAI), pages 1–8, May 2021. arXiv: 2008.10781. URL: http://arxiv.org/abs/2008.10781 (visited on 2022-03-25), doi:10.1109/ICAPAI49758.2021.9462056.
+[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset,fit
+import pandas as pd
+from tslearn.datasets import UCR_UEA_datasets
+import torch
+import warnings
+warnings.filterwarnings("ignore")
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +dataset='BasicMotions'
+X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,-1,-2)#X_train.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=np.swapaxes(X_test,-1,-2)#X_test.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(train_y.reshape(-1,1))
+f=open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb')
+pickle.dump(enc1,f)
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels= X_train.shape[-1], num_pred_classes=n_pred_classes)
+#fit(model, train_loader,test_loader)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+<All keys matched successfully>+
#torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/ResNet')
+Using a interpretability algorithm consists of 4 steps:
+COMTE works on all models returning a probability function. The Initialization takes the following arguments:
+ +`model`: The model to be explaines.
+`data`: Tuple of Data and Labels.
+`backend`: `PYT`, `SK`, or `TF`.
+`mode`: second dimension is either `feat` or `time`.
+`method`: Optimization Method either `brut` or `opt`.model.eval()
+item=test_x[10].reshape(1,test_x.shape[1],-1)
+shape=item.shape
+_item= torch.from_numpy(item).float()
+model.eval()
+y_pred = torch.nn.functional.softmax(model(_item)).detach().numpy()
+y_label= np.argmax(y_pred)
+%load_ext autoreload
+%autoreload 2
+
+from TSInterpret.InterpretabilityModels.counterfactual.COMTECF import COMTECF
+
+exp_model= COMTECF(model,(train_x,train_y),backend='PYT',mode='feat', method= 'opt')
+Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedexp = exp_model.explain(item)
+array, label=exp
+print(label)
+[0] ++
item.shape
+(1, 6, 100)+
array.shape
+(1, 6, 100)+
All plot function take as input the item to be explained and the returned explanation. As as additonal option a figsize can be given.
+ +i=0
+org_label=y_label
+cf_label=label[0]
+exp=array
+
+exp_model.plot_in_one(np.array(item),org_label,np.array(exp),cf_label,figsize=(15,15))
+
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +Counterfactual Explanations for Machine Learning on Multivariate Time Series Data(COMTE) developed by Ates et al. [1] builds counterfactuals in multivariate setting by perturbing the features of a time series with the help of an heuristic algorithm. +The code in TSInterpret is based on the authors implementation .
+[1] Emre Ates, Burak Aksar, Vitus J. Leung, and Ayse K. Coskun. Counterfactual Explanations for Machine Learning on Multivariate Time Series Data. 2021 International Conference on Applied Artificial Intelligence (ICAPAI), pages 1–8, May 2021. arXiv: 2008.10781. URL: http://arxiv.org/abs/2008.10781 (visited on 2022-03-25), doi:10.1109/ICAPAI49758.2021.9462056.
+ +import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset
+import pandas as pd
+from tslearn.datasets import UCR_UEA_datasets
+import torch
+import warnings
+warnings.filterwarnings("ignore")
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +dataset='Epilepsy'
+X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,1,2)#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=np.swapaxes(X_test,1,2)#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+train_x=np.nan_to_num(train_x)
+test_x=np.nan_to_num(test_x)
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(train_y.reshape(-1,1))
+f=open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb')
+pickle.dump(enc1,f)
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+enc1.categories_
+[array(['EPILEPSY', 'RUNNING', 'SAWING', 'WALKING'], dtype='<U8')]+
Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels= X_train.shape[-1], num_pred_classes=n_pred_classes)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+model.eval()
+ResNetBaseline( + (layers): Sequential( + (0): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(3, 64, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(3, 64, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (2): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + ) + ) + (final): Linear(in_features=128, out_features=4, bias=True) +)+
Using a interpretability algorithm consists of 4 steps:
+COMTE works on all models returning a probability function. The Initialization takes the following arguments:
+ +`model`: The model to be explaines.
+`data`: Tuple of Data and Labels.
+`backend`: `PYT`, `SK`, or `TF`.
+`mode`: second dimension is either `feat` or `time`.
+`method`: Optimization Method either `brut` or `opt`.item=test_x[1].reshape(1,test_x.shape[1],-1)
+shape=item.shape
+_item= torch.from_numpy(item).float()
+model.eval()
+y_pred = torch.nn.functional.softmax(model(_item)).detach().numpy()
+y_label= np.argmax(y_pred)
+y_label
+0+
enc1.inverse_transform(np.array([[1,0,0,0]]))
+array([['EPILEPSY']], dtype='<U8')+
from TSInterpret.InterpretabilityModels.counterfactual.COMTECF import COMTECF
+
+
+exp_model= COMTECF(model,(train_x,train_y),backend='PYT',mode='feat', method= 'brute')
+2023-09-18 14:45:51.081533: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 14:45:51.757656: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedexp = exp_model.explain(item)
+array, label=exp
+label
+array([3])+
enc1.inverse_transform(np.array([[0,0,1,0]]))
+array([['SAWING']], dtype='<U8')+
item.shape
+(1, 3, 206)+
All plot function take as input the item to be explained and the returned explanation. As as additonal option a figsize can be given.
+ +i=0
+org_label=y_label
+cf_label=label[0]
+exp=array
+
+exp_model.plot_in_one(item[0],'EPILEPSY',exp[0],'SAWING',save_fig='Ates.png')
+
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +import pickle
+import numpy as np
+from tslearn.datasets import UCR_UEA_datasets
+dataset='GunPoint'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+print(train_x.shape)
+print(test_x.shape)
+(50, 150, 1) +(150, 150, 1) ++
train_x.shape
+(50, 150, 1)+
train_y.shape
+(50,)+
model= pickle.load(open(f'../../ClassificationModels/models/GunPoint/GunPoint_SVM.sav','rb')).fit(train_x,train_y)
+#/media/jacqueline/Data/InterpretabiltyTimeSeries/ClassificationModels/models/GunPoint/GunPoint_SVM.sav
+explained_instance = train_x[0]
+nb_interpretable_feature = 10
+explanation_size = 5
+nb_neighbors = 1000
+learning_process_name = 'Lime'
+transform_name = 'straight_line'
+from TSInterpret.InterpretabilityModels.leftist.leftist import LEFTIST
+2022-10-27 11:59:39.218998: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory +2022-10-27 11:59:39.219023: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. ++
model_to_explain = model
+random_state=0
+leftist = LEFTIST(model_to_explain,(test_x,None),mode='time', backend='SK')
+explanations = leftist.explain(np.array(explained_instance),1000,explanation_size=explanation_size)
+%matplotlib inline
+leftist.plot(np.array([explained_instance]).reshape(1,-1,1), np.array([explanations[0]]))
+time mode +(1, 150) +(1, 150) +-1 +1 +0 ++
+import numpy as np
+from tslearn.datasets import UCR_UEA_datasets
+dataset='ElectricDevices'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+print(train_x.shape)
+print(test_x.shape)
+(8926, 96, 1) +(7711, 96, 1) ++
train_x.shape
+(8926, 96, 1)+
train_y.shape
+(8926,)+
import tensorflow as tf
+model_to_explain = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/best_model.hdf5')
+2023-06-15 11:04:50.284047: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA +To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-06-15 11:04:50.505469: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory +2023-06-15 11:04:50.505494: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. +2023-06-15 11:04:51.418759: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory +2023-06-15 11:04:51.418867: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory +2023-06-15 11:04:51.418879: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. +2023-06-15 11:04:52.395804: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory +2023-06-15 11:04:52.395839: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) +2023-06-15 11:04:52.395861: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (jacqueline-ThinkPad-P53): /proc/driver/nvidia/version does not exist +2023-06-15 11:04:52.396059: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA +To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. ++
explained_instance = train_x[0]
+nb_interpretable_feature = 10
+explanation_size = 5
+nb_neighbors = 1000
+learning_process_name = 'Lime'
+transform_name = 'straight_line'
+from TSInterpret.InterpretabilityModels.leftist.leftist import LEFTIST
+leftist = LEFTIST(model_to_explain,(test_x, test_y),mode='time',backend='TF',transform_name='straight_line',learning_process_name='Lime')
+#,'straight_line',segmentator,,learning_process_name
+explanations = leftist.explain(np.array(explained_instance))
+32/32 [==============================] - 0s 5ms/step ++
leftist.plot(np.array([explained_instance]), np.array([explanations[1]]))
+time mode +(1, 96) +(1, 96) +-1 +1 +0 ++
+Agnostic Local Explanation for Time Series Classification (LEFTIST) by Guilleme et al. (2019) [1] adapted LIME for time series classification and proposed to use prefixed (both the +length and the position) shapelets as the interpretable components and provide the feature importance of each shapelet.
+The code in TSInterpret is based on the authors implementation .
+[1] Guillemé, Maël, et al. "Agnostic local explanation for time series classification." 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2019.
+ +import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset
+from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+dataset='ECG5000'
+Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x = train_x.reshape(-1,1, train_x.shape[-2])
+test_x = test_x.reshape(-1,1, test_x.shape[-2])
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+Loads a pretrained 1D-Conv ResNet.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels=1, num_pred_classes=n_pred_classes)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+model.eval()
+ResNetBaseline( + (layers): Sequential( + (0): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(1, 64, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(1, 64, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (2): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + ) + ) + (final): Linear(in_features=128, out_features=5, bias=True) +)+
Using a interpretability algorithm consists of 4 steps:
+ +1. Load the Interpretability Method
+2. Instaniate the Method with the desired Parameters
+3. Call the explain Method
+4. Plot the results
+
+LEFTIST works on all type of black-box classificators and can therefore be used with PyTorch (PYT), TensorFlow (TF), Sklearn (SK) and predict function, as long as the predict function returns a probability distribution. The Initialization takes the following arguments:
+model_to_explain: The model to be explaines. data: Reference Set of type (X_train, y_train).mode: Second dimension 'time' or 'feat'.backend: TF, PYT, SK or func.transform_name: Name of transformer.
* `uniform`: Mean Transform
+ * `straight_line`: StraightlineTransform
+ * `background`: RandBackgroundTransformlearning_process_name: 'Lime' or 'Shap'.nb_interpretable_feature: number of desired features.explained_instance = train_x[1]
+nb_interpretable_feature = 10
+explanation_size = 2
+nb_neighbors = 1000
+learning_process_name = 'Lime'
+transform_name = 'straight_line'
+model_to_explain=model
+from TSInterpret.InterpretabilityModels.leftist.leftist import LEFTIST
+2023-06-15 10:59:13.237750: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA +To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-06-15 10:59:15.142855: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory +2023-06-15 10:59:15.142970: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory +2023-06-15 10:59:15.142981: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. ++
leftist = LEFTIST(model_to_explain,(test_x,None),mode='feat', backend='PYT', \
+learning_process_name='Lime',transform_name='straight_line')
+Prepare the instance and the predicted label of the instance as parameters for the explain methods.
+model_to_explain: The model to be explaines. nb_neighbors: Number if neighbors used in the calculatuions.idx_label: index of label to explain. If None, return an explanation for each label.explanation_size: Size of desired explanation.explanations = leftist.explain(np.array(explained_instance))
+Visualize the explanation by calling .plot(). +Parameters are the original instance to be explained and the explanation.
+ +leftist.plot(np.array([explained_instance]),np.array([explanations[1]]),save='LEFTIST.png')
+feat +NOT Time mode +-1 +1 +0 ++
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +Agnostic Local Explanation for Time Series Classification (LEFTIST) by Guilleme et al. (2019) [1] adapted LIME for time series classification and proposed to use prefixed (both the length and the position) shapelets as the interpretable components and provide the feature importance of each shapelet.
+
The code in TSInterpret is based on the authors implementation . The image is reproduced from [1].
+[1] Guillemé, Maël, et al. "Agnostic local explanation for time series classification." 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2019.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset
+from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='ElectricDevices'
+Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x = train_x.reshape(-1,1, train_x.shape[-2])
+test_x = test_x.reshape(-1,1, test_x.shape[-2])
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
Loads a pretrained 1D-Conv ResNet.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels=1, num_pred_classes=n_pred_classes)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+<All keys matched successfully>+
Using a interpretability algorithm consists of 4 steps:
+ +1. Load the Interpretability Method
+2. Instaniate the Method with the desired Parameters
+3. Call the explain Method
+4. Plot the results
+
+LEFTIST works on all type of black-box classificators and can therefore be used with PyTorch (PYT), TensorFlow (TF), Sklearn (SK) and predict function, as long as the predict function returns a probability distribution. The Initialization takes the following arguments:
+model_to_explain: The model to be explaines. data: Reference Set of type (X_train, y_train).mode: Second dimension 'time' or 'feat'.backend: TF, PYT, SK or func.transform_name: Name of transformer.
* `uniform`: Mean Transform
+ * `straight_line`: StraightlineTransform
+ * `background`: RandBackgroundTransformlearning_process_name: 'Lime' or 'Shap'.nb_interpretable_feature: number of desired features.model.eval()
+explained_instance = train_x[10]
+nb_interpretable_feature = 10
+explanation_size = 5
+nb_neighbors = 1000
+learning_process_name = 'Lime'
+transform_name = 'straight_line'
+model_to_explain=model
+from TSInterpret.InterpretabilityModels.leftist.leftist import LEFTIST
+leftist = LEFTIST(model_to_explain,(test_x,None),mode='feat', backend='PYT', \
+learning_process_name='Lime',transform_name='straight_line')
+Prepare the instance and the predicted label of the instance as parameters for the explain methods.
+model_to_explain: The model to be explaines. nb_neighbors: Number if neighbors used in the calculatuions.idx_label: index of label to explain. If None, return an explanation for each label.explanation_size: Size of desired explanation.explanations = leftist.explain(np.array(explained_instance))
+Visualize the explanation by calling .plot(). +Parameters are the original instance to be explained and the explanation.
+ +leftist.plot(np.array([explained_instance]),np.array([explanations[1]]))
+feat +NOT Time mode +-1 +1 +0 ++
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+import matplotlib.pyplot as plt
+import seaborn as sns
+from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
+import pandas as pd
+import os
+from ClassificationModels.ResNet import Classifier_RESNET
+from ClassificationModels.CNN import Classifier_CNN
+import tensorflow as tf
+2023-06-14 09:10:04.860948: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA +To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-06-14 09:10:06.200602: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory +2023-06-14 09:10:06.200708: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory +2023-06-14 09:10:06.200719: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. ++
dataset='ElectricDevices'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+import tensorflow as tf
+model = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/best_model.hdf5')
+2023-06-14 09:10:10.787128: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory +2023-06-14 09:10:10.787156: W tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:265] failed call to cuInit: UNKNOWN ERROR (303) +2023-06-14 09:10:10.787179: I tensorflow/compiler/xla/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (jacqueline-ThinkPad-P53): /proc/driver/nvidia/version does not exist +2023-06-14 09:10:10.787364: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA +To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. ++
item=test_x[10].reshape(1,-1,1)
+shape=item.shape
+y_target= model.predict(item)
+1/1 [==============================] - 0s 189ms/step ++
print(y_target)
+[[6.5272084e-06 1.0916962e-18 2.0263034e-11 1.5986618e-07 8.7627843e-03 + 1.0000000e+00 1.8584505e-08]] ++
from TSInterpret.InterpretabilityModels.counterfactual.NativeGuideCF import NativeGuideCF
+exp_model=NativeGuideCF(model,(train_x,train_y), backend='TF', mode = 'time',method='NUN_CF')
+(96, 1) +279/279 [==============================] - 1s 3ms/step ++
exp,label=exp_model.explain(item, np.argmax(y_target,axis=1))
+(1, 1, 96) +[5] +1/1 [==============================] - 0s 50ms/step +average_pooling1d_1 +1/1 [==============================] - 0s 39ms/step +1/1 [==============================] - 0s 66ms/step +1/1 [==============================] - 0s 46ms/step +1/1 [==============================] - 0s 59ms/step +1/1 [==============================] - 0s 57ms/step +1/1 [==============================] - 0s 21ms/step +1/1 [==============================] - 0s 83ms/step +1/1 [==============================] - 0s 74ms/step +1/1 [==============================] - 0s 71ms/step +1/1 [==============================] - 0s 42ms/step +1/1 [==============================] - 0s 69ms/step +1/1 [==============================] - 0s 43ms/step +1/1 [==============================] - 0s 39ms/step +1/1 [==============================] - 0s 45ms/step +1/1 [==============================] - 0s 87ms/step +1/1 [==============================] - 0s 88ms/step +1/1 [==============================] - 0s 52ms/step +1/1 [==============================] - 0s 45ms/step +1/1 [==============================] - 0s 51ms/step +1/1 [==============================] - 0s 68ms/step +1/1 [==============================] - 0s 55ms/step +1/1 [==============================] - 0s 51ms/step +1/1 [==============================] - 0s 47ms/step +1/1 [==============================] - 0s 45ms/step +1/1 [==============================] - 0s 28ms/step +1/1 [==============================] - 0s 47ms/step +1/1 [==============================] - 0s 59ms/step +1/1 [==============================] - 0s 44ms/step +1/1 [==============================] - 0s 29ms/step +1/1 [==============================] - 0s 46ms/step +1/1 [==============================] - 0s 69ms/step +1/1 [==============================] - 0s 51ms/step +1/1 [==============================] - 0s 25ms/step +1/1 [==============================] - 0s 27ms/step +1/1 [==============================] - 0s 30ms/step +1/1 [==============================] - 0s 61ms/step +1/1 [==============================] - 0s 52ms/step +1/1 [==============================] - 0s 73ms/step +1/1 [==============================] - 0s 48ms/step +1/1 [==============================] - 0s 56ms/step +1/1 [==============================] - 0s 87ms/step ++
%matplotlib inline
+exp_model.plot(item.reshape(-1),np.argmax(y_target,axis=1)[0],exp.reshape(-1),label)
+
+%matplotlib inline
+exp_model.plot_in_one(item,np.argmax(y_target,axis=1)[0],exp,label)
+
+A counterfactual explanation, originally introduced to machine learning by [2], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+Delaney et al.[1] proposed using the K-nearest neighbors from the dataset belonging to a different class as native guide to generate counterfactuals. They propose three options for transforming the original time series with this native guide: the plain native guide, the native guide with bary centering, and transformation based on the native guide and class activation mapping. The desired method can be selected by providing the method parameter during interpretability instantiation.
+
The code in TSInterpret is based on the authors implementation .
+[1] E. Delaney, D. Greene, and M. T. Keane. Instance-Based Counterfactual Explanations for Time Series Classification. In A. A. S ́anchez-Ruiz and M. W.Floyd, editors, Case-Based Reasoning Research and Development, volume 12877, pages 32–47. Springer International Publishing, Cham, 2021. ISBN 978-3-030-86956-4 978-3-030-86957-1. doi: 10.1007/978-3-030-86957-1 3. Series Title: Lecture Notes in Computer Science.
+[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset
+import warnings
+warnings.filterwarnings("ignore")
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +dataset='ElectricDevices'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x = train_x.reshape(-1,1, train_x.shape[-2])
+test_x = test_x.reshape(-1,1, test_x.shape[-2])
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+f= open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb')
+pickle.dump(enc1,f)
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels=1, num_pred_classes=n_pred_classes)
+state_dict= torch.load(f'../../ClassificationModels/models/{dataset}/ResNet')
+model.load_state_dict(state_dict)
+<All keys matched successfully>+
Using a interpretability algorithm consists of 4 steps:
+Native Guide counterfactuals works on all models tensorflow and pyTorch models returning a probability function. The Initialization takes the following arguments:
+ +`model`: The model to be explaines.
+`shape`: The data shape in form of (features, timesteps).
+`data`: Tuple of Data and Labels used to find and build CF.
+`backend`: `PYT`, `SK`, or `TF`.
+`mode`: second dimension is either `feat` or `time`.
+`method`: ['NUN_CF', "dtw_bary_center", 'NG'].model.eval()
+item=test_x[20].reshape(1,1,-1)
+shape=item.shape
+_item= torch.from_numpy(item).float()
+model.eval()
+y_target = torch.nn.functional.softmax(model(_item)).detach().numpy()
+from TSInterpret.InterpretabilityModels.counterfactual.NativeGuideCF \
+ import NativeGuideCF
+exp_model=NativeGuideCF(model,(train_x,train_y), \
+ backend='PYT', mode='feat',method='NUN_CF')
+WARNING:root:no value was provided for `target_layer`, thus set to 'layers'. +WARNING:root:no value was provided for `fc_layer`, thus set to 'final'. ++
Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explained.target: desired target class of CF.exp,label=exp_model.explain(item, np.argmax(y_target,axis=1)[0])
+Plot function takes as input the item to be explained and the returned explanation, as well as the assigned labels.
+ +exp_model.plot(item,np.argmax(y_target,axis=1)[0],exp,label)
+
+exp_model.plot_in_one(item,np.argmax(y_target,axis=1)[0],exp,label)
+
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +Delaney et al.[1] proposed using the K-nearest neighbors from the dataset belonging to a different class as native guide to generate counterfactuals. They propose three options for transforming the original time series with this native guide: the plain native guide, the native guide with bary centering, and transformation based on the native guide and class activation mapping. The desired method can be selected by providing the method parameter during interpretability instantiation.
+The code in TSInterpret is based on the authors implementation .
+[1] E. Delaney, D. Greene, and M. T. Keane. Instance-Based Counterfactual Explanations for Time Series Classification. In A. A. S ́anchez-Ruiz and M. W. Floyd, editors, Case- +Based Reasoning Research and Development, volume 12877, pages 32–47. Springer International Publishing, Cham, 2021. ISBN 978-3-030-86956-4 978-3-030-86957-1. doi: +10.1007/978-3-030-86957-1 3. Series Title: Lecture Notes in Computer Science.
+ +from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+import matplotlib.pyplot as plt
+import seaborn as sns
+from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
+import pandas as pd
+from tslearn.datasets import UCR_UEA_datasets
+import os
+import warnings
+warnings.filterwarnings("ignore")
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
Load Data and reshape the data to fit a 1D-Conv ResNet. Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +dataset='ECG5000'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x = train_x.reshape(-1,1, train_x.shape[-2])
+test_x = test_x.reshape(-1,1, test_x.shape[-2])
+np.unique(test_y)
+array([1, 2, 3, 4, 5])+
enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+f= open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb')
+pickle.dump(enc1,f)
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels= train_x.shape[-2], num_pred_classes=n_pred_classes)
+fit(model,train_loader,test_loader)
+if dataset in os.listdir('../../ClassificationModels/models/'):
+ print('Folder exists')
+else:
+ os.mkdir(f'../../ClassificationModels/models/{dataset}')
+torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/ResNet')
+
+test_preds, ground_truth = get_all_preds(model, test_loader)
+ground_truth=np.argmax(ground_truth,axis=1)
+#test_preds=np.argmax(test_preds,axis=1)
+sns.set(rc={'figure.figsize':(5,4)})
+heatmap=confusion_matrix(ground_truth, test_preds)
+sns.heatmap(heatmap, annot=True)
+plt.savefig(f'../../ClassificationModels/models/{dataset}/ResNet_confusion_matrix.png')
+plt.close()
+acc= accuracy_score(ground_truth, test_preds)
+a = classification_report(ground_truth, test_preds, output_dict=True)
+dataframe = pd.DataFrame.from_dict(a)
+dataframe.to_csv(f'../../ClassificationModels/models/{dataset}/classification_report.csv', index = False)
+Epoch: 1, Train loss: 0.569, Val loss: 0.436 +Epoch: 2, Train loss: 0.486, Val loss: 0.296 +Epoch: 3, Train loss: 0.305, Val loss: 0.296 +Epoch: 4, Train loss: 0.345, Val loss: 0.393 +Epoch: 5, Train loss: 0.269, Val loss: 0.273 +Epoch: 6, Train loss: 0.342, Val loss: 0.368 +Epoch: 7, Train loss: 0.273, Val loss: 0.297 +Epoch: 8, Train loss: 0.231, Val loss: 0.292 +Epoch: 9, Train loss: 0.214, Val loss: 0.559 +Epoch: 10, Train loss: 0.247, Val loss: 0.258 +Epoch: 11, Train loss: 0.231, Val loss: 0.25 +Epoch: 12, Train loss: 0.201, Val loss: 0.303 +Epoch: 13, Train loss: 0.172, Val loss: 0.31 +Epoch: 14, Train loss: 0.222, Val loss: 0.28 +Epoch: 15, Train loss: 0.185, Val loss: 0.278 +Epoch: 16, Train loss: 0.227, Val loss: 0.254 +Epoch: 17, Train loss: 0.192, Val loss: 0.227 +Epoch: 18, Train loss: 0.156, Val loss: 0.238 +Epoch: 19, Train loss: 0.149, Val loss: 0.233 +Epoch: 20, Train loss: 0.197, Val loss: 0.299 +Epoch: 21, Train loss: 0.225, Val loss: 0.29 +Epoch: 22, Train loss: 0.193, Val loss: 0.281 +Epoch: 23, Train loss: 0.135, Val loss: 0.254 +Epoch: 24, Train loss: 0.136, Val loss: 0.293 +Epoch: 25, Train loss: 0.128, Val loss: 0.234 +Epoch: 26, Train loss: 0.153, Val loss: 0.444 +Epoch: 27, Train loss: 0.213, Val loss: 0.299 +Epoch: 28, Train loss: 0.147, Val loss: 0.321 +Epoch: 29, Train loss: 0.169, Val loss: 0.264 +Epoch: 30, Train loss: 0.151, Val loss: 0.254 +Epoch: 31, Train loss: 0.146, Val loss: 0.254 +Epoch: 32, Train loss: 0.143, Val loss: 0.314 +Epoch: 33, Train loss: 0.154, Val loss: 0.395 +Epoch: 34, Train loss: 0.176, Val loss: 0.417 +Epoch: 35, Train loss: 0.151, Val loss: 0.266 +Epoch: 36, Train loss: 0.186, Val loss: 0.36 +Epoch: 37, Train loss: 0.12, Val loss: 0.325 +Epoch: 38, Train loss: 0.135, Val loss: 0.292 +Epoch: 39, Train loss: 0.102, Val loss: 0.272 +Epoch: 40, Train loss: 0.108, Val loss: 0.28 +Epoch: 41, Train loss: 0.114, Val loss: 0.324 +Epoch: 42, Train loss: 0.107, Val loss: 0.319 +Epoch: 43, Train loss: 0.099, Val loss: 0.29 +Epoch: 44, Train loss: 0.103, Val loss: 0.469 +Epoch: 45, Train loss: 0.137, Val loss: 0.556 +Epoch: 46, Train loss: 0.149, Val loss: 0.326 +Epoch: 47, Train loss: 0.187, Val loss: 0.427 +Epoch: 48, Train loss: 0.135, Val loss: 0.309 +Epoch: 49, Train loss: 0.148, Val loss: 0.259 +Epoch: 50, Train loss: 0.105, Val loss: 0.325 +Epoch: 51, Train loss: 0.103, Val loss: 0.398 +Epoch: 52, Train loss: 0.1, Val loss: 0.345 +Epoch: 53, Train loss: 0.076, Val loss: 0.368 +Epoch: 54, Train loss: 0.102, Val loss: 0.317 +Epoch: 55, Train loss: 0.093, Val loss: 0.387 +Epoch: 56, Train loss: 0.107, Val loss: 0.7 +Epoch: 57, Train loss: 0.174, Val loss: 0.29 +Epoch: 58, Train loss: 0.116, Val loss: 0.314 +Epoch: 59, Train loss: 0.079, Val loss: 0.366 +Epoch: 60, Train loss: 0.068, Val loss: 0.355 +Epoch: 61, Train loss: 0.084, Val loss: 0.326 +Epoch: 62, Train loss: 0.082, Val loss: 0.39 +Epoch: 63, Train loss: 0.073, Val loss: 0.338 +Epoch: 64, Train loss: 0.091, Val loss: 0.367 +Epoch: 65, Train loss: 0.081, Val loss: 0.368 +Epoch: 66, Train loss: 0.074, Val loss: 0.453 +Epoch: 67, Train loss: 0.079, Val loss: 0.557 +Epoch: 68, Train loss: 0.096, Val loss: 0.328 +Epoch: 69, Train loss: 0.074, Val loss: 0.427 +Epoch: 70, Train loss: 0.061, Val loss: 0.416 +Epoch: 71, Train loss: 0.071, Val loss: 0.413 +Epoch: 72, Train loss: 0.057, Val loss: 0.419 +Epoch: 73, Train loss: 0.047, Val loss: 0.437 +Epoch: 74, Train loss: 0.041, Val loss: 0.447 +Epoch: 75, Train loss: 0.042, Val loss: 0.475 +Epoch: 76, Train loss: 0.046, Val loss: 0.457 +Epoch: 77, Train loss: 0.038, Val loss: 0.738 +Epoch: 78, Train loss: 0.09, Val loss: 0.609 +Epoch: 79, Train loss: 0.102, Val loss: 0.48 +Epoch: 80, Train loss: 0.051, Val loss: 0.609 +Epoch: 81, Train loss: 0.088, Val loss: 0.663 +Epoch: 82, Train loss: 0.15, Val loss: 0.62 +Epoch: 83, Train loss: 0.194, Val loss: 0.343 +Epoch: 84, Train loss: 0.114, Val loss: 0.338 +Epoch: 85, Train loss: 0.078, Val loss: 0.405 +Epoch: 86, Train loss: 0.066, Val loss: 0.469 +Epoch: 87, Train loss: 0.054, Val loss: 0.442 +Epoch: 88, Train loss: 0.044, Val loss: 0.483 +Epoch: 89, Train loss: 0.035, Val loss: 0.519 +Epoch: 90, Train loss: 0.031, Val loss: 0.625 +Epoch: 91, Train loss: 0.029, Val loss: 0.613 +Epoch: 92, Train loss: 0.046, Val loss: 0.421 +Epoch: 93, Train loss: 0.061, Val loss: 0.505 +Epoch: 94, Train loss: 0.07, Val loss: 0.707 +Epoch: 95, Train loss: 0.19, Val loss: 0.36 +Epoch: 96, Train loss: 0.104, Val loss: 0.497 +Epoch: 97, Train loss: 0.144, Val loss: 0.356 +Epoch: 98, Train loss: 0.135, Val loss: 0.391 +Epoch: 99, Train loss: 0.086, Val loss: 0.376 +Epoch: 100, Train loss: 0.055, Val loss: 0.463 +Epoch: 101, Train loss: 0.06, Val loss: 0.49 +Epoch: 102, Train loss: 0.071, Val loss: 0.51 +Epoch: 103, Train loss: 0.055, Val loss: 0.449 +Epoch: 104, Train loss: 0.107, Val loss: 0.405 +Epoch: 105, Train loss: 0.071, Val loss: 0.497 +Epoch: 106, Train loss: 0.102, Val loss: 0.652 +Epoch: 107, Train loss: 0.097, Val loss: 0.533 +Epoch: 108, Train loss: 0.057, Val loss: 0.431 +Epoch: 109, Train loss: 0.04, Val loss: 0.521 +Epoch: 110, Train loss: 0.039, Val loss: 0.576 +Epoch: 111, Train loss: 0.034, Val loss: 0.52 +Epoch: 112, Train loss: 0.026, Val loss: 0.697 +Epoch: 113, Train loss: 0.024, Val loss: 0.638 +Epoch: 114, Train loss: 0.053, Val loss: 0.872 +Epoch: 115, Train loss: 0.075, Val loss: 0.674 +Epoch: 116, Train loss: 0.108, Val loss: 0.584 +Epoch: 117, Train loss: 0.046, Val loss: 0.707 +Early stopping! +Folder exists ++
Using a interpretability algorithm consists of 4 steps:
+Native Guide counterfactuals works on all models tensorflow and pyTorch models returning a probability function. The Initialization takes the following arguments:
+ +`model`: The model to be explaines.
+`shape`: The data shape in form of (features, timesteps).
+`data`: Tuple of Data and Labels used to find and build CF.
+`backend`: `PYT`, `SK`, or `TF`.
+`mode`: second dimension is either `feat` or `time`.
+`method`: Optimization Method either `brut` or `opt`.item=test_x[10].reshape(1,1,-1)
+shape=item.shape
+_item= torch.from_numpy(item).float()
+model.eval()
+y_target = torch.nn.functional.softmax(model(_item)).detach().numpy()
+from TSInterpret.InterpretabilityModels.counterfactual.NativeGuideCF \
+ import NativeGuideCF
+exp_model=NativeGuideCF(model,(train_x,train_y), \
+ backend='PYT', mode='feat',method="NUN_CF")
+2023-08-04 16:53:38.340266: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-08-04 16:53:39.151477: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT +WARNING:root:no value was provided for `target_layer`, thus set to 'layers'. +WARNING:root:no value was provided for `fc_layer`, thus set to 'final'. ++
(1, 140) ++
Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explained.target: desired target class of CF.exp,label=exp_model.explain(item, np.argmax(y_target,axis=1) )
+Plot function takes as input the item to be explained and the returned explanation, as well as the assigned labels.
+ +exp_model.plot(item,np.argmax(y_target,axis=1)[0],exp,label)
+
+exp_model.plot_in_one(item,'Normal',exp,'Abnormal','NUN_CF')
+
+def plot_in_one(self,item,org_label,exp,cf_label, save_fig= None,figsize=(15,15)):
+ """
+ Plot Function for Counterfactuals in uni-and multivariate setting. In the multivariate setting only the changed features are visualized.
+ Arguments:
+ item np.array: original instance.
+ org_label int: originally predicted label.
+ exp np.array: returned explanation.
+ cf_label int: lebel of returned instance.
+ figsize Tuple: Size of Figure (x,y).
+ save_fig str: Path to Save the figure.
+ """
+ if self.mode == 'time':
+ item = item.reshape(item.shape[0],item.shape[2],item.shape[1])
+ #TODO This is new and needs to be testes
+ if item.shape[-2]>1:
+ res = (item != exp).any(-1)
+ ind=np.where(res[0])
+ if len(ind[0]) == 0:
+ print('Items are identical')
+ return
+ elif len(ind[0]) > 1:
+ self.plot_multi(item,org_label,exp,cf_label,figsize=figsize, save_fig=save_fig)
+ return
+ else:
+ item =item[ind]
+
+ plt.style.use("classic")
+ colors = [
+ '#08F7FE', # teal/cyan
+ '#FE53BB', # pink
+ '#F5D300', # yellow
+ '#00ff41', # matrix green
+ ]
+ indices= np.where(exp[0] != item)
+ df = pd.DataFrame({f'Predicted: {org_label}': list(item.flatten()),
+ f'Counterfactual: {cf_label}': list(exp.flatten())})
+ fig, ax = plt.subplots(figsize=(10,5))
+ df.plot(marker='.', color=colors, ax=ax)
+ # Redraw the data with low alpha and slighty increased linewidth:
+ n_shades = 10
+ diff_linewidth = 1.05
+ alpha_value = 0.3 / n_shades
+ for n in range(1, n_shades+1):
+ df.plot(marker='.',
+ linewidth=2+(diff_linewidth*n),
+ alpha=alpha_value,
+ legend=False,
+ ax=ax,
+ color=colors)
+
+ ax.grid(color='#2A3459')
+ plt.xlabel('Time', fontweight = 'bold', fontsize='large')
+ plt.ylabel('Value', fontweight = 'bold', fontsize='large')
+ if save_fig is None:
+ plt.show()
+ else:
+ plt.savefig(save_fig)
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +A counterfactual explanation, originally introduced to machine learning by [1], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+Shapelet-based Temporal Association Rule Mining for Multivariate Time Series Classification developed by Bahri et al. [1] builds counterfactuals in multivariate setting by perturbing the features of a time series with the help of a shapelet algorithm. Hereby they extract the most prominent class shapelets using an adaption of the shapelet transform algorithm of [2] implemented in sktime. The algorithm is adapted to multivariate timeseries analysis, treating each dimension as a univariate timeseries and information gain. The algorithm imputes the shapelets based on prior observed occcurences and shapelet quality. If a single shapelet does not cause a counterfactual prediction, more perturbations in other dimensions or other parts of the timeseries are performed.
+
Visualization of SETS from the original paper [1]; The code in TSInterpret is based on the authors implementation .
+[1] Omar Bahri and Soukaina Filali Boubrahimi and Shah Muhammad Hamdi.Shapelet-Based Counterfactual Explanations for Multivariate Time Series. 2022 ACM SIGKDD Workshop on Mining and Learning from Time Series. arXiv:2208.10462 URL:https://arxiv.org/abs/2208.10462 (visited on 2023-12-04)
+[2] Hills, Jon and Lines, Jason and Baranauskas, Edgaras and Mapp, James and Bagnall, Anthony. Classification of time series by shapelet transformation. Data Mining and Knowledge Discovery. Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
+import numpy as np
+from tslearn.datasets import UCR_UEA_datasets
+import warnings
+import tensorflow.keras as keras
+
+warnings.filterwarnings("ignore")
+2023-12-15 17:29:49.635370: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered +2023-12-15 17:29:49.635399: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered +2023-12-15 17:29:49.636427: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered +2023-12-15 17:29:49.641956: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-12-15 17:29:50.349624: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
data = UCR_UEA_datasets().load_dataset("ECG200")
+model = keras.models.load_model("../../ClassificationModels/models/ECG200/best_model_ecg200_fcn_wang_ep200.hdf5")
+2023-12-15 17:29:51.626851: E external/local_xla/xla/stream_executor/cuda/cuda_driver.cc:274] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected +2023-12-15 17:29:51.626885: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:129] retrieving CUDA diagnostic information for host: jacqueline-ThinkPad-P53 +2023-12-15 17:29:51.626893: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:136] hostname: jacqueline-ThinkPad-P53 +2023-12-15 17:29:51.626959: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:159] libcuda reported version is: 525.147.5 +2023-12-15 17:29:51.626980: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:163] kernel reported version is: 525.147.5 +2023-12-15 17:29:51.626985: I external/local_xla/xla/stream_executor/cuda/cuda_diagnostics.cc:241] kernel version seems to match DSO: 525.147.5 ++
Using a interpretability algorithm consists of 4 steps:
+SETS works on all models returning a probability function. The Initialization takes the following arguments:
+ +`model`: The model to be explaines.
+`data`: Tuple of Data and Labels.
+`backend`: `PYT`, `SK`, or `TF`.
+`mode`: second dimension is either `feat` or `time`.
+`method`: Optimization Method either `brut` or `opt`.
+`min_shapelet_len` : Value for min length of extracted shapelets
+`max_shapelet_len`: Value for max length of extracted shapelets
+`time_contract_in_mins_per_dim` : Max time for shapelet extraction per dimension
+`initial_num_shapelets_per_case` : Initial number of shapelets per case.%load_ext autoreload
+%autoreload 2
+
+from TSInterpret.InterpretabilityModels.counterfactual.SETSCF import SETSCF
+
+train_x, train_y, test_x, test_y = data
+
+# Note: this is for demonstration purposes, time contract per minutes should be adjusted for better results!
+exp_model= SETSCF(model,
+ (train_x, train_y),
+ backend='TF',
+ mode='time',
+ min_shapelet_len=3,
+ max_shapelet_len=20,
+ time_contract_in_mins_per_dim=1,
+ #initial_num_shapelets_per_case=10,
+ fit_shapelets = False)
+Extract Shapelets with information gain rejection lvl 0.001 and shapelets per class of 30 ++
This method is optional! If shapelets are already exctracted, please specify here the occlusion threshhold and if shapelets belonging to multiple classes should be retained. +This method is called by default if the explain_model is fitted.
+ +exp_model.fit(occlusion_threshhold=1e-1,remove_multiclass_shapelets=True)
+Fit function to prune shapelets with occlusion threshhold of 0.1 and remove shapelets belonging to more than one class set to True
+Shapelet by index per class and dimension: {0: [[1, 4, 5, 7, 8, 9, 11, 13, 14, 15, 20, 21, 23, 25, 30, 31, 34]], 1: [[0, 2, 3, 6, 10, 12, 16, 17, 18, 19, 22, 24, 26, 27, 28, 29, 32, 33]]}
+
+Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedts = 5
+test_y[ts]
+cf_explanation, label = exp_model.explain(np.swapaxes(test_x[ts],0,1), target = None)
+1/1 [==============================] - 0s 108ms/step +1/1 [==============================] - 0s 15ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +1/1 [==============================] - 0s 13ms/step +Counterfactual has been found ++
All plot function take as input the item to be explained and the returned explanation. As as additonal option a figsize can be given.
+ +exp_model.plot(
+ cf_explanation,
+ np.argmax(model(data[2])[ts]),
+ test_x[ts],
+ label,
+ vis_change=True,
+ all_in_one=False,
+ save_fig=None,
+ figsize=(6.4, 4.8))
+
+data = UCR_UEA_datasets().load_dataset("BasicMotions")
+model = keras.models.load_model("../../ClassificationModels/models/BasicMotions/best_model_bm_fcn_wang_ep200.hdf5")
+train_x,train_y,test_x,test_y = data
+exp_model= SETSCF(model,
+ (train_x,train_y),
+ backend='TF',
+ mode='time',
+ min_shapelet_len=3,
+ max_shapelet_len=20,
+ time_contract_in_mins_per_dim=1,
+ #initial_num_shapelets_per_case=10,
+ fit_shapelets = False)
++--------------------------------------------------------------------------- +TypeError Traceback (most recent call last) +/tmp/ipykernel_15697/2091522095.py in <cell line: 1>() +----> 1 exp_model= SETSCF(model, + 2 (train_x,train_y), + 3 backend='TF', + 4 mode='time', + 5 min_shapelet_len=3, + +TypeError: SETSCF.__init__() got an unexpected keyword argument 'initial_num_shapelets_per_case'+
exp_model.fit(occlusion_threshhold=1e-1,remove_multiclass_shapelets=True)
+Fit function to prune shapelets with occlusion threshhold of 0.1 and remove shapelets belonging to more than one class set to True
+Shapelet by index per class and dimension: {0: [[0, 14, 15, 20, 26, 27, 28, 29, 32, 33, 34, 36, 41, 43, 46, 53, 54, 59, 67, 73, 81], [0, 1, 29, 30, 36, 37, 41, 42, 43, 45, 46, 50, 53, 57, 59, 65, 67, 68, 72, 73, 74, 84, 85, 88, 90, 96, 97, 98, 100], [0, 1, 10, 11, 20, 24, 26, 35, 36, 41, 47, 48, 56, 58, 64, 65, 67, 72, 74, 81], [11, 12, 26, 27, 29, 31, 33, 36, 38, 41, 49, 53, 55, 56, 57, 61, 70, 73, 81, 82, 83, 85, 86, 90], [2, 9, 10, 13, 14, 20, 21, 23, 24, 26, 30, 31, 37, 43, 44, 49, 52, 54, 56, 57, 69, 81, 84, 91, 92], [0, 1, 2, 3, 4, 5, 16, 17, 18, 30, 31, 32, 36, 37, 38, 39, 40, 45, 49, 51, 59, 60, 67, 68, 69, 81, 100]], 1: [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 16, 17, 18, 21, 22, 37, 42, 45, 60, 61, 64, 74], [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 31, 32, 38, 44, 51, 54, 71, 75, 101, 102, 104], [4, 16, 23, 25, 31, 33, 39, 44, 45, 46, 49, 54, 55, 59, 66, 68, 71, 75, 76, 77, 82, 84, 86], [0, 3, 5, 6, 7, 8, 15, 16, 18, 19, 22, 28, 34, 39, 40, 42, 51, 52, 71, 74, 75, 80, 87, 91], [3, 5, 6, 7, 11, 12, 15, 16, 17, 27, 28, 33, 41, 46, 51, 58, 66, 72, 73, 82, 86, 87, 93, 95, 96], [6, 7, 8, 19, 20, 21, 22, 23, 33, 41, 42, 46, 47, 50, 52, 53, 54, 57, 58, 62, 63, 64, 65, 66, 70, 73, 99, 104, 109]], 2: [[44, 47, 48, 49, 50, 51, 55, 56, 57, 58, 62, 63, 66, 68, 69, 76, 77, 78, 79, 80, 82], [49, 52, 58, 60, 62, 63, 66, 69, 70, 76, 78, 79, 80, 82, 83, 86, 87, 89, 91, 92, 93, 94, 95, 99, 103, 105, 106, 107], [12, 15, 17, 27, 28, 30, 32, 37, 38, 40, 42, 50, 51, 52, 57, 60, 73, 78, 79, 80, 85], [35, 37, 44, 45, 47, 48, 50, 58, 60, 62, 63, 64, 65, 66, 67, 68, 69, 76, 77, 78, 84, 88, 89], [18, 19, 34, 35, 36, 38, 42, 45, 47, 50, 53, 59, 61, 63, 65, 67, 74, 75, 80, 83, 85, 88, 89, 90, 94, 97, 98, 99], [55, 71, 72, 74, 77, 78, 79, 80, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 94, 95, 97, 98, 101, 102, 103, 105, 106, 107, 108]], 3: [[13, 19, 23, 24, 25, 30, 31, 35, 38, 39, 40, 52, 65, 70, 71, 72, 75], [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 33, 34, 35, 39, 40, 47, 48, 55, 56, 61, 64, 77, 81], [2, 3, 5, 6, 7, 8, 9, 13, 14, 18, 19, 21, 22, 29, 34, 43, 53, 61, 62, 63, 69, 70, 83], [1, 2, 4, 9, 10, 13, 14, 17, 20, 21, 23, 24, 25, 30, 32, 43, 46, 54, 59, 72, 79], [0, 1, 4, 8, 22, 25, 29, 32, 39, 40, 48, 55, 60, 62, 64, 68, 70, 71, 76, 77, 78, 79], [9, 10, 11, 12, 13, 14, 15, 24, 25, 26, 27, 28, 29, 34, 35, 43, 44, 48, 56, 61, 75, 76, 93, 96]]}
+
+ts = 12
+cf_explanation, label = exp_model.explain(np.swapaxes(test_x[ts],0,1), target = None)
+1/1 [==============================] - 0s 37ms/step +1/1 [==============================] - 0s 38ms/step +1/1 [==============================] - 0s 43ms/step +1/1 [==============================] - 0s 70ms/step +1/1 [==============================] - 0s 17ms/step +1/1 [==============================] - 0s 20ms/step +1/1 [==============================] - 0s 19ms/step +1/1 [==============================] - 0s 30ms/step +1/1 [==============================] - 0s 24ms/step +1/1 [==============================] - 0s 26ms/step +1/1 [==============================] - 0s 39ms/step +1/1 [==============================] - 0s 25ms/step +1/1 [==============================] - 0s 22ms/step +1/1 [==============================] - 0s 25ms/step +1/1 [==============================] - 0s 31ms/step +1/1 [==============================] - 0s 27ms/step +1/1 [==============================] - 0s 31ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 33ms/step +1/1 [==============================] - 0s 14ms/step +1/1 [==============================] - 0s 31ms/step +1/1 [==============================] - 0s 29ms/step +1/1 [==============================] - 0s 26ms/step +Counterfactual has been found ++
#item, org_label, exp, cf_label, save_fig=None, figsize=(6.4, 4.8)
+exp_model.plot_in_one(test_x[10],0,np.swapaxes(cf_explanation,0,1),1,figsize=(15,15))
+
+import pandas as pd
+import numpy as np
+import torch
+import gc
+import torch.nn as nn
+from tqdm import tqdm_notebook as tqdm
+from torch.utils.data import Dataset, DataLoader
+import seaborn as sns
+import matplotlib.pyplot as plt
+from sklearn.metrics import confusion_matrix, accuracy_score
+import random
+from sklearn import tree
+from sklearn.model_selection import cross_val_score
+from pymop import Problem
+import os
+from tslearn.datasets import UCR_UEA_datasets
+import warnings
+warnings.filterwarnings('ignore')
+import pickle
+import sklearn
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='ElectricDevices'
+X_train,train_y,X_test,test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x=X_train
+test_x=X_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder_tf.pkl','wb'))
+#enc1=pickle.load(open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','rb'))
+test_y=enc1.transform(test_y.reshape(-1,1))
+n_classes = test_y.shape[1]
+import tensorflow as tf
+model = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/best_model.hdf5')
+2023-09-19 07:32:02.634236: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-19 07:32:03.306787: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT +2023-09-19 07:32:03.908472: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 +2023-09-19 07:32:03.908909: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. +Skipping registering GPU devices... ++
observation_01= X_test[0]
+label_01 =test_y[0]
+
+print(observation_01.shape)
+print(label_01.shape)
+(96, 1) +(7,) ++
from TSInterpret.InterpretabilityModels.counterfactual.TSEvoCF import TSEvo
+exp =TSEvo(model= model,data=(test_x,test_y), backend='TF',mode='time',epochs =100)
+y was one Hot Encoded ++
pop,label_cf=exp.explain(observation_01.reshape(1,-1,1),np.array([np.argmax(label_01)]))
+No Target ++
np.array(pop).shape
+(1, 96)+
exp.plot_in_one(observation_01.reshape(1,-1,1),label_01[0],pop,np.argmax(label_cf),save_fig=None)
+
+A counterfactual explanation, originally introduced to machine learning by [2], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+
TSEvo [1] adapts the notion of counterfactual to the domain of uni- and multivariate time series classification by building and optimizing a Problem consisting of notions for proximity ($R_{1}$), sparsity ($R_{2}$) and the output distance ($R_{4}$). The optimization problem is solved by applying an evolutionary algorithm based on NSGA-II with mutation operators ($R_{3}$) adapted to the time series domain.
+[1] Jacqueline Höllig, Cedric Kulbach and Steffen Thoma. TSEvo: Evolutionary Counterfactual Explanations for Time Series Classification. 2022 International Conference on Machine Learning and Applications (ICMLA).
+[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import pandas as pd
+import numpy as np
+import torch
+import gc
+import torch.nn as nn
+from tqdm import tqdm_notebook as tqdm
+from torch.utils.data import Dataset, DataLoader
+import seaborn as sns
+import matplotlib.pyplot as plt
+from sklearn.metrics import confusion_matrix, accuracy_score
+import random
+from sklearn import tree
+from sklearn.model_selection import cross_val_score
+from pymop import Problem
+import os
+from tslearn.datasets import UCR_UEA_datasets
+import warnings
+warnings.filterwarnings('ignore')
+import pickle
+import sklearn
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='GunPoint'
+X_train,train_y,X_test,test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,1,2)#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=np.swapaxes(X_test,1,2)#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+#enc1=pickle.load(open(f'../../ClassificationModels//models/{dataset}/OneHotEncoder.pkl','rb'))
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+test_y=enc1.transform(test_y.reshape(-1,1))
+train_y=enc1.transform(train_y.reshape(-1,1))
+n_classes = test_y.shape[1]
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset, get_all_preds, fit
+stride = 1
+kernel_size=10
+padding = kernel_size - 1
+input_size= train_x.shape[-1]
+device = torch.device( "cpu")#"cuda:0" if torch.cuda.is_available() else
+model = ResNetBaseline(in_channels= 1, num_pred_classes=n_classes)
+#train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+#test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+#train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+#test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+#fit(model, train_loader, test_loader)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+#torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/ResNet')
+test_dataset = UCRDataset(test_x,test_y)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=True)
+model.eval()
+y_pred,labels= get_all_preds(model,test_loader)
+observation_01, label_01 = test_dataset[0]
+label_01=np.array([y_pred[0]])
+print(observation_01.shape)
+print(label_01.shape)
+(1, 150) +(1,) ++
Using a interpretability algorithm consists of 4 steps:
+Plot the results +### 1. & 2. Loading & Initialization
+model: The model to be explaines.
+ data: Tuple of Data and Labels.
+ backend: PYT, SK, or TF.
+ mode: second dimension is either feat or time.
from TSInterpret.InterpretabilityModels.counterfactual.TSEvoCF import TSEvo
+exp =TSEvo(model= model,data=(test_x,np.array(y_pred)), mode = 'feat',backend='PYT',epochs =500)
+Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedmethod: ['authentic_opposing','mutate_both','gaussian','frequency']target_y: targeted CF class, can be Noneepochs: # iterationspop,label_cf=exp.explain(observation_01.reshape(1,1,-1),np.array(label_01))
+No Target ++
exp.plot_in_one(observation_01,label_01[0],pop,np.argmax(label_cf),save_fig=None)
+
+A counterfactual explanation, originally introduced to machine learning by [2], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+
TSEvo [1] adapts the notion of counterfactual to the domain of uni- and multivariate time series classification by building and optimizing a Problem consisting of notions for proximity ($R_{1}$), sparsity ($R_{2}$) and the output distance ($R_{4}$). The optimization problem is solved by applying an evolutionary algorithm based on NSGA-II with mutation operators ($R_{3}$) adapted to the time series domain.
+[1] Jacqueline Höllig, Cedric Kulbach and Steffen Thoma. TSEvo: Evolutionary Counterfactual Explanations for Time Series Classification. 2022 International Conference on Machine Learning and Applications (ICMLA).
+[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import pandas as pd
+import numpy as np
+import torch
+import gc
+import torch.nn as nn
+from tqdm import tqdm_notebook as tqdm
+from torch.utils.data import Dataset, DataLoader
+import seaborn as sns
+import matplotlib.pyplot as plt
+from sklearn.metrics import confusion_matrix, accuracy_score
+import random
+from sklearn import tree
+from sklearn.model_selection import cross_val_score
+from pymop import Problem
+import os
+from tslearn.datasets import UCR_UEA_datasets
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+from ClassificationModels.LSTM_T import LSTM
+import warnings
+warnings.filterwarnings('ignore')
+import pickle
+dataset='BasicMotions'
+X_train,train_y,X_test,test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x=X_train#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=X_test#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+enc1=pickle.load(open(f'../../ClassificationModels//models/{dataset}/OneHotEncoder.pkl','rb'))
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+#test_y=enc1.transform(test_y.reshape(-1,1))
+n_classes = test_y.shape[1]
+#n_pred_classes =train_y.shape[0]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+device='cpu'
+hidden_size=10
+rnn=0.1
+model = LSTM(6, hidden_size ,n_classes,rnndropout=rnn).to(device)
+#fit(model,train_loader,test_loader)
+if dataset in os.listdir('../../ClassificationModels/models/'):
+ print('Folder exists')
+else:
+ os.mkdir(f'../../ClassificationModels/models/{dataset}')
+#torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/LSTM')
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/LSTM'))
+
+model.eval()
+Folder exists ++
LSTM( + (drop): Dropout(p=0.1, inplace=False) + (fc): Linear(in_features=10, out_features=4, bias=True) + (rnn): LSTM(6, 10, batch_first=True) +)+
y_pred, ground_truth = get_all_preds(model, test_loader)
+observation_01, label_01 = test_dataset[0]
+label_01=np.array([y_pred[0]])
+print(observation_01.shape)
+print(label_01.shape)
+(100, 6) +(1,) ++
Using a interpretability algorithm consists of 4 steps:
+Plot the results +### 1. & 2. Loading & Initialization
+model: The model to be explaines.
+ data: Tuple of Data and Labels.
+ backend: PYT, SK, or TF.
+ mode: second dimension is either feat or time.
from TSInterpret.InterpretabilityModels.counterfactual.TSEvoCF import TSEvo
+exp =TSEvo(model= model,data=(test_x,np.array(y_pred)), mode = 'time',backend='PYT',epochs =10)
+<class 'numpy.int64'> ++
Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedmethod: ['authentic_opposing','mutate_both','gaussian','frequency']target_y: targeted CF class, can be Noneepochs: # iterationspop,label_cf=exp.explain(observation_01.reshape(1,-1,6),np.array(label_01))
+(1, 6, 100) +No Target ++
exp.plot_in_one(observation_01.reshape(observation_01.shape[0],observation_01.shape[1]),label_01[0],pop,np.argmax(label_cf),save_fig=None)
+
+A counterfactual explanation, originally introduced to machine learning by [2], answers the question "what if" by building counterexamples. Based on an input instance $x$, the goal is to find a counterfactual $x^{cf}$ close to the original instance $x$ but differently classified $y \neq y^{cf}$ by a predictor $f$. The intention is to visualize boundary cases. Further research has shown that counterfactual explanations are easy to understand for humans because they are intuitive to human thinking by showing counterexamples.
+
TSEvo [1] adapts the notion of counterfactual to the domain of uni- and multivariate time series classification by building and optimizing a Problem consisting of notions for proximity ($R_{1}$), sparsity ($R_{2}$) and the output distance ($R_{4}$). The optimization problem is solved by applying an evolutionary algorithm based on NSGA-II with mutation operators ($R_{3}$) adapted to the time series domain.
+[1] Jacqueline Höllig, Cedric Kulbach and Steffen Thoma. TSEvo: Evolutionary Counterfactual Explanations for Time Series Classification. 2022 International Conference on Machine Learning and Applications (ICMLA).
+[2] Wachter, Sandra, Brent Mittelstadt, and Chris Russell. "Counterfactual explanations without opening the black box: Automated decisions and the GDPR." Harv. JL & Tech. 31 (2017): 841.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import pandas as pd
+import numpy as np
+import torch
+import gc
+import torch.nn as nn
+from tqdm import tqdm_notebook as tqdm
+from torch.utils.data import Dataset, DataLoader
+import seaborn as sns
+import matplotlib.pyplot as plt
+from sklearn.metrics import confusion_matrix, accuracy_score
+import random
+from sklearn import tree
+from sklearn.model_selection import cross_val_score
+from pymop import Problem
+import os
+from tslearn.datasets import UCR_UEA_datasets
+import warnings
+warnings.filterwarnings('ignore')
+import pickle
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
import sklearn
+dataset='BasicMotions'
+X_train,train_y,X_test,test_y=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,1,2)
+test_x=np.swapaxes(X_test,1,2)
+
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(test_y.reshape(-1, 1))#pickle.load(open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','rb'))
+test_y=enc1.transform(test_y.reshape(-1,1))
+n_classes = test_y.shape[1]
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset, get_all_preds
+
+device = torch.device( "cpu")#"cuda:0" if torch.cuda.is_available() else
+model = ResNetBaseline(in_channels=6, num_pred_classes=4)
+model.load_state_dict(torch.load(f'../../ClassificationModels//models/{dataset}/ResNet'))
+test_dataset = UCRDataset(test_x,test_y)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=True)
+model.eval()
+y_pred,labels= get_all_preds(model,test_loader)
+observation_01, label_01 = test_dataset[0]
+label_01=np.array([y_pred[0]])
+print(observation_01.shape)
+print(label_01.shape)
+(6, 100) +(1,) ++
Using a interpretability algorithm consists of 4 steps:
+Plot the results +### 1. & 2. Loading & Initialization
+model: The model to be explaines.
+ data: Tuple of Data and Labels.
+ backend: PYT, SK, or TF.
+ mode: second dimension is either feat or time.
from TSInterpret.InterpretabilityModels.counterfactual.TSEvoCF import TSEvo
+exp =TSEvo(model= model,data=(test_x,np.array(y_pred)), mode = 'feat',backend='PYT',epochs =10)
+Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedmethod: ['authentic_opposing','mutate_both','gaussian','frequency']target_y: targeted CF class, can be Noneepochs: # iterationspop,label_cf=exp.explain(observation_01.reshape(1,6,-1),np.array(label_01))
+No Target ++
exp.plot_in_one(observation_01,label_01[0],pop,np.argmax(label_cf),save_fig=None)
+
+from tslearn.datasets import UCR_UEA_datasets
+import numpy as np
+import torch
+import seaborn as sns
+import sklearn
+from ClassificationModels.LSTM_T import LSTM
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
#Load Data
+dataset='GunPoint'
+X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=X_train.reshape(-1,X_train.shape[-2],X_train.shape[-1])
+test_x=X_test.reshape(-1,X_train.shape[-2],X_train.shape[-1])
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+#pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+
+print(train_y)
+#Load Model
+n_pred_classes =len(np.unique(train_y))
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+[[0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.]] ++
/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
model = LSTM(1, 10 ,2 ,0.1)#ResNetBaseline(in_channels=1, num_pred_classes=n_pred_classes)
+#fit(model,train_loader,test_loader)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/LSTM'))
+
+model.eval()
+LSTM( + (drop): Dropout(p=0.1, inplace=False) + (fc): Linear(in_features=10, out_features=2, bias=True) + (rnn): LSTM(1, 10, batch_first=True) +)+
item = np.array([test_x[0,:,:]],dtype=np.float64)
+label =0
+#TODO SLIDING WINDOW
+from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model, train_x.shape[-2],train_x.shape[-1], method='FO', mode='time')
+print(item.shape)
+exp= int_mod.explain(item,labels=label,TSR =True)
+2023-09-18 12:55:40.118252: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 12:55:41.436679: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
(1, 150, 1) ++
print(exp.shape)
+print(np.array([test_x[0,:,:]]).shape)
+int_mod.plot(np.array([test_x[0,:,:]]),exp, figsize=(30,30))
+(150, 1) +(1, 150, 1) +time mode ++
+from tslearn.datasets import UCR_UEA_datasets
+import numpy as np
+import torch
+import seaborn as sns
+import sklearn
+from ClassificationModels.LSTM_T import LSTM
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
#Load Data
+dataset='GunPoint'
+X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=X_train.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=X_test.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+#pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+
+print(train_y)
+#Load Model
+n_pred_classes =len(np.unique(train_y))
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+[[0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [0. 1.] + [1. 0.] + [1. 0.] + [1. 0.] + [0. 1.] + [0. 1.] + [1. 0.] + [0. 1.] + [1. 0.] + [0. 1.]] ++
/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
model =ResNetBaseline(in_channels=1, num_pred_classes=n_pred_classes)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+
+model.eval()
+ResNetBaseline( + (layers): Sequential( + (0): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(1, 64, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(1, 64, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (2): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + ) + ) + (final): Linear(in_features=128, out_features=2, bias=True) +)+
item = np.array([test_x[0,:,:]],dtype=np.float64)
+label =0
+from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model, train_x.shape[-1],train_x.shape[-2], method='FO', \
+ mode='feat')
+exp= int_mod.explain(item,labels=label,TSR =True)
+2023-09-18 12:47:33.775890: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 12:47:34.578751: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
print(exp.shape)
+print(np.array([test_x[0,:,:]]).shape)
+int_mod.plot(np.array([test_x[0,:,:]]),exp, figsize=(30,30))
+(1, 150) +(1, 1, 150) +NOT Time mode ++
+import pickle
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as snst
+from tslearn.datasets import UCR_UEA_datasets
+import tensorflow as tf
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm +2023-09-18 14:08:21.084927: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 14:08:21.806931: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
dataset='BasicMotions'#'GunPoint'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+import sklearn
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(train_y.reshape(-1,1))#pickle.load(open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','rb'))
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
import tensorflow as tf
+
+
+#model = Classifier_CNN(f'./ClassificationModels/models/{dataset}', (train_x.shape[1], train_x.shape[2]), train_y.shape[-1], True)
+#y_true = np.argmax(test_y, axis=1)
+#model.fit(train_x,train_y,test_x,test_y)
+
+model_to_explain = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/{dataset}best_model.hdf5')
+2023-09-18 14:08:22.758550: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 +2023-09-18 14:08:22.759033: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. +Skipping registering GPU devices... ++
from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model_to_explain, train_x.shape[-2],train_x.shape[-1], method='IG',mode='time')
+Mode in TF Saliency time ++
print(np.array([test_x[0,:,:]]).shape)
+(1, 100, 6) ++
item= np.array([test_x[0,:,:]])
+label=int(np.argmax(test_y[0]))
+label
+2+
exp=int_mod.explain(item,labels=label,TSR =True)
+%matplotlib inline
+int_mod.plot(np.array([test_x[0,:,:]]),exp)
+time mode ++
+%matplotlib inline
+int_mod.plot(np.array([test_x[0,:,:]]),exp, heatmap = True)
+time mode ++
+import pickle
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as snst
+from tslearn.datasets import UCR_UEA_datasets
+import tensorflow as tf
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm +2023-09-18 12:06:18.348758: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 12:06:19.269948: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
dataset='Coffee'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+import tensorflow as tf
+
+
+#model = Classifier_CNN(f'./ClassificationModels/models/{dataset}', (train_x.shape[1], train_x.shape[2]), train_y.shape[-1], True)
+#y_true = np.argmax(test_y, axis=1)
+#model.fit(train_x,train_y,test_x,test_y)
+
+model_to_explain = tf.keras.models.load_model(f'../../ClassificationModels/models/{dataset}/cnn/last_model.hdf5')
+2023-09-18 12:06:20.361423: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355 +2023-09-18 12:06:20.362034: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1960] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. +Skipping registering GPU devices... ++
from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model_to_explain, train_x.shape[-2],train_x.shape[-1], method='IG',mode='time')
+Mode in TF Saliency time ++
print(np.array([test_x[0,:,:]]).shape)
+(1, 286, 1) ++
item= np.array([test_x[0,:,:]])
+label=int(np.argmax(test_y[0]))
+exp=int_mod.explain(item,labels=label,TSR =True)
+%matplotlib inline
+int_mod.plot(np.array([test_x[0,:,:]]),exp)
+time mode ++
+%matplotlib inline
+int_mod.plot(np.array([test_x[0,:,:]]),exp, heatmap = True)
+time mode ++
+Temporal Saliency Rescaling (TSR), developed by Ismail et al. (2020) [1], is built upon known saliency methods such as GradCam [2] or Shap [3]. Their benchmark study shows that traditional saliency methods fail to reliably and accurately identify feature importances due to the time and feature domain. TSR is proposed on top of the traditional saliency methods as a two-step approach for improving the quality of saliency maps. The importance of each time step is calculated, followed by the feature importance.
+The code in TSInterpret is based on the authors implementation .
+[1] Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Deep Learning Interpretability in Time Series Predictions. arXiv:2010.13924 [cs, stat], October 2020. arXiv: 2010.13924.
+[2] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Int J Comput Vis, 128(2):336–359, Feb. 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007/s11263-019-01228-7. arXiv: 1610.02391.
+[3] S. M. Lundberg and S.-I. Lee. A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R.Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
+ +import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+import pandas as pd
+import os
+from tslearn.datasets import UCR_UEA_datasets
+from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
+import sklearn
+import matplotlib as plt
+import seaborn as sns
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='Epilepsy'
+UCR_UEA_datasets().list_multivariate_datasets()
+['ArticularyWordRecognition', + 'AtrialFibrillation', + 'BasicMotions', + 'CharacterTrajectories', + 'Cricket', + 'DuckDuckGeese', + 'EigenWorms', + 'Epilepsy', + 'EthanolConcentration', + 'ERing', + 'FaceDetection', + 'FingerMovements', + 'HandMovementDirection', + 'Handwriting', + 'Heartbeat', + 'InsectWingbeat', + 'JapaneseVowels', + 'Libras', + 'LSST', + 'MotorImagery', + 'NATOPS', + 'PenDigits', + 'PEMS-SF', + 'Phoneme', + 'RacketSports', + 'SelfRegulationSCP1', + 'SelfRegulationSCP2', + 'SpokenArabicDigits', + 'StandWalkJump', + 'UWaveGestureLibrary']+
Load Data and reshape the data to fit a 1D-Conv ResNet. +Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,1,2)#.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+train_x=np.nan_to_num(train_x)
+test_x=np.swapaxes(X_test,1,2)#X_test.reshape(-1,X_train.shape[-1],X_train.shape[-2])
+test_x=np.nan_to_num(test_x)
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels=X_train.shape[-1], num_pred_classes=n_pred_classes)
+#model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+fit(model,train_loader,test_loader)
+if dataset in os.listdir('../../ClassificationModels/models/'):
+ print('Folder exists')
+else:
+ os.mkdir(f'../../ClassificationModels/models/{dataset}')
+torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/ResNet')
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+test_preds, ground_truth = get_all_preds(model, test_loader)
+
+model.eval()
+Epoch: 1, Train loss: 0.762, Val loss: 1.043 +Epoch: 2, Train loss: 0.552, Val loss: 0.392 +Epoch: 3, Train loss: 0.442, Val loss: 0.275 +Epoch: 4, Train loss: 0.627, Val loss: 0.399 +Epoch: 5, Train loss: 0.361, Val loss: 0.34 +Epoch: 6, Train loss: 0.188, Val loss: 0.222 +Epoch: 7, Train loss: 0.143, Val loss: 0.148 +Epoch: 8, Train loss: 0.054, Val loss: 0.197 +Epoch: 9, Train loss: 0.039, Val loss: 0.161 +Epoch: 10, Train loss: 0.017, Val loss: 0.145 +Epoch: 11, Train loss: 0.023, Val loss: 0.123 +Epoch: 12, Train loss: 0.019, Val loss: 0.155 +Epoch: 13, Train loss: 0.006, Val loss: 0.177 +Epoch: 14, Train loss: 0.004, Val loss: 0.181 +Epoch: 15, Train loss: 0.003, Val loss: 0.176 +Epoch: 16, Train loss: 0.003, Val loss: 0.171 +Epoch: 17, Train loss: 0.002, Val loss: 0.163 +Epoch: 18, Train loss: 0.002, Val loss: 0.163 +Epoch: 19, Train loss: 0.001, Val loss: 0.164 +Epoch: 20, Train loss: 0.001, Val loss: 0.166 +Epoch: 21, Train loss: 0.001, Val loss: 0.171 +Epoch: 22, Train loss: 0.001, Val loss: 0.173 +Epoch: 23, Train loss: 0.001, Val loss: 0.172 +Epoch: 24, Train loss: 0.001, Val loss: 0.177 +Epoch: 25, Train loss: 0.001, Val loss: 0.178 +Epoch: 26, Train loss: 0.001, Val loss: 0.179 +Epoch: 27, Train loss: 0.001, Val loss: 0.183 +Epoch: 28, Train loss: 0.001, Val loss: 0.185 +Epoch: 29, Train loss: 0.0, Val loss: 0.186 +Epoch: 30, Train loss: 0.0, Val loss: 0.188 +Epoch: 31, Train loss: 0.0, Val loss: 0.188 +Epoch: 32, Train loss: 0.0, Val loss: 0.19 +Epoch: 33, Train loss: 0.0, Val loss: 0.191 +Epoch: 34, Train loss: 0.0, Val loss: 0.193 +Epoch: 35, Train loss: 0.0, Val loss: 0.195 +Epoch: 36, Train loss: 0.0, Val loss: 0.196 +Epoch: 37, Train loss: 0.0, Val loss: 0.198 +Epoch: 38, Train loss: 0.0, Val loss: 0.199 +Epoch: 39, Train loss: 0.0, Val loss: 0.2 +Epoch: 40, Train loss: 0.0, Val loss: 0.201 +Epoch: 41, Train loss: 0.0, Val loss: 0.201 +Epoch: 42, Train loss: 0.0, Val loss: 0.203 +Epoch: 43, Train loss: 0.0, Val loss: 0.204 +Epoch: 44, Train loss: 0.0, Val loss: 0.205 +Epoch: 45, Train loss: 0.0, Val loss: 0.206 +Epoch: 46, Train loss: 0.0, Val loss: 0.208 +Epoch: 47, Train loss: 0.0, Val loss: 0.209 +Epoch: 48, Train loss: 0.0, Val loss: 0.211 +Epoch: 49, Train loss: 0.0, Val loss: 0.209 +Epoch: 50, Train loss: 0.0, Val loss: 0.209 +Epoch: 51, Train loss: 0.0, Val loss: 0.211 +Epoch: 52, Train loss: 0.0, Val loss: 0.214 +Epoch: 53, Train loss: 0.0, Val loss: 0.215 +Epoch: 54, Train loss: 0.0, Val loss: 0.217 +Epoch: 55, Train loss: 0.0, Val loss: 0.216 +Epoch: 56, Train loss: 0.0, Val loss: 0.217 +Epoch: 57, Train loss: 0.0, Val loss: 0.217 +Epoch: 58, Train loss: 0.0, Val loss: 0.217 +Epoch: 59, Train loss: 0.0, Val loss: 0.219 +Epoch: 60, Train loss: 0.0, Val loss: 0.218 +Epoch: 61, Train loss: 0.0, Val loss: 0.219 +Epoch: 62, Train loss: 0.0, Val loss: 0.222 +Epoch: 63, Train loss: 0.0, Val loss: 0.223 +Epoch: 64, Train loss: 0.0, Val loss: 0.222 +Epoch: 65, Train loss: 0.0, Val loss: 0.223 +Epoch: 66, Train loss: 0.0, Val loss: 0.223 +Epoch: 67, Train loss: 0.0, Val loss: 0.223 +Epoch: 68, Train loss: 0.0, Val loss: 0.224 +Epoch: 69, Train loss: 0.0, Val loss: 0.226 +Epoch: 70, Train loss: 0.0, Val loss: 0.228 +Epoch: 71, Train loss: 0.0, Val loss: 0.227 +Epoch: 72, Train loss: 0.0, Val loss: 0.228 +Epoch: 73, Train loss: 0.0, Val loss: 0.229 +Epoch: 74, Train loss: 0.0, Val loss: 0.23 +Epoch: 75, Train loss: 0.0, Val loss: 0.231 +Epoch: 76, Train loss: 0.0, Val loss: 0.232 +Epoch: 77, Train loss: 0.0, Val loss: 0.233 +Epoch: 78, Train loss: 0.0, Val loss: 0.232 +Epoch: 79, Train loss: 0.0, Val loss: 0.232 +Epoch: 80, Train loss: 0.0, Val loss: 0.233 +Epoch: 81, Train loss: 0.0, Val loss: 0.233 +Epoch: 82, Train loss: 0.0, Val loss: 0.233 +Epoch: 83, Train loss: 0.0, Val loss: 0.235 +Epoch: 84, Train loss: 0.0, Val loss: 0.235 +Epoch: 85, Train loss: 0.0, Val loss: 0.236 +Epoch: 86, Train loss: 0.0, Val loss: 0.236 +Epoch: 87, Train loss: 0.0, Val loss: 0.237 +Epoch: 88, Train loss: 0.0, Val loss: 0.237 +Epoch: 89, Train loss: 0.0, Val loss: 0.238 +Epoch: 90, Train loss: 0.0, Val loss: 0.239 +Epoch: 91, Train loss: 0.0, Val loss: 0.24 +Epoch: 92, Train loss: 0.0, Val loss: 0.24 +Epoch: 93, Train loss: 0.0, Val loss: 0.241 +Epoch: 94, Train loss: 0.0, Val loss: 0.241 +Epoch: 95, Train loss: 0.0, Val loss: 0.242 +Epoch: 96, Train loss: 0.0, Val loss: 0.242 +Epoch: 97, Train loss: 0.0, Val loss: 0.243 +Epoch: 98, Train loss: 0.0, Val loss: 0.244 +Epoch: 99, Train loss: 0.0, Val loss: 0.244 +Epoch: 100, Train loss: 0.0, Val loss: 0.246 +Epoch: 101, Train loss: 0.0, Val loss: 0.245 +Epoch: 102, Train loss: 0.0, Val loss: 0.245 +Epoch: 103, Train loss: 0.0, Val loss: 0.246 +Epoch: 104, Train loss: 0.0, Val loss: 0.247 +Epoch: 105, Train loss: 0.0, Val loss: 0.247 +Epoch: 106, Train loss: 0.0, Val loss: 0.247 +Epoch: 107, Train loss: 0.0, Val loss: 0.249 +Epoch: 108, Train loss: 0.0, Val loss: 0.25 +Epoch: 109, Train loss: 0.0, Val loss: 0.25 +Epoch: 110, Train loss: 0.0, Val loss: 0.25 +Epoch: 111, Train loss: 0.0, Val loss: 0.25 +Early stopping! +Folder exists ++
ResNetBaseline( + (layers): Sequential( + (0): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(3, 64, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 64, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(3, 64, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (1): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + (residual): Sequential( + (0): Conv1dSamePadding(64, 128, kernel_size=(1,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + ) + ) + (2): ResNetBlock( + (layers): Sequential( + (0): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(8,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (1): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(5,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + (2): ConvBlock( + (layers): Sequential( + (0): Conv1dSamePadding(128, 128, kernel_size=(3,), stride=(1,)) + (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) + (2): ReLU() + ) + ) + ) + ) + ) + (final): Linear(in_features=128, out_features=4, bias=True) +)+
Using a interpretability algorithm consists of 4 steps:
+The TSR works on gradient-based models. Currently support for TensorFlow (TF) and PYTorch (PYT) is available. For PYT the Subclass Saliency_PTY is used, while TF expects the use of Saliency_TF.
+The Initialization takes the following arguments:
model: The model to be explaines. NumTimeStep: Number of Time Step.NumFetaures: Number Features.method: Saliency Methode to be used.
* Gradients (GRAD)
+* Integrated Gradients (IG)
+* Gradient Shap (GS)
+* DeepLift (DL)
+* DeepLiftShap (DLS)
+* SmoothGrad (SG)
+* Shapley Value Sampling(SVS)
+* Feature Permutation (FP)
+* Feature Sampling (FS)
+* Occlusion (FO)mode: Second dimension 'time' or 'feat'.from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model, train_x.shape[-1],train_x.shape[-2], method='GS', \
+ mode='feat')
+2023-09-18 14:02:29.017433: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 14:02:29.790998: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedlabels: predicted label for the item as class. TSR: Turns temporal rescaling on / off.item = np.array([train_x[1,:,:]])
+label =test_preds[1]
+exp=int_mod.explain(item,labels=label,TSR =False)
+exp.shape
+(3, 206)+
All plot function take as input the item to be explained and the returned explanation. As as additonal option a figsize can be given. +For visualizing saliency there are two visualization options provided:
+int_mod.plot(np.array(item),exp, figsize=(30,30), save='Ismail_Ep.png')
+NOT Time mode ++
+int_mod.plot(np.array([test_x[0,:,:]]),exp, heatmap = True)
+NOT Time mode ++
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +Temporal Saliency Rescaling (TSR), developed by Ismail et al. (2020) [1], is built upon known saliency methods such as GradCam [2] or Shap [3]. Their benchmark study shows that traditional saliency methods fail to reliably and accurately identify feature importances due to the time and feature domain. TSR is proposed on top of the traditional saliency methods as a two-step approach for improving the quality of saliency maps. The importance of each time step is calculated, followed by the feature importance.
+The code in TSInterpret is based on the authors implementation .
+[1] Aya Abdelsalam Ismail, Mohamed Gunady, Héctor Corrada Bravo, and Soheil Feizi. Benchmarking Deep Learning Interpretability in Time Series Predictions. arXiv:2010.13924 [cs, stat], October 2020. arXiv: 2010.13924.
+[2] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. Int J Comput Vis, 128(2):336–359, Feb. 2020. ISSN 0920-5691, 1573-1405. doi: 10.1007/s11263-019-01228-7. arXiv: 1610.02391.
+[3] S. M. Lundberg and S.-I. Lee. A Unified Approach to Interpreting Model Predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R.Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
+ +import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, UCRDataset,fit
+import pandas as pd
+import os
+from tslearn.datasets import UCR_UEA_datasets
+import sklearn
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
dataset='BasicMotions'
+Load Data and reshape the data to fit a 1D-Conv ResNet. +Note that the input for a 1D-Conv Resnet hat the shape (batch, features, timesteps).
+ +X_train,y_train, X_test, y_test=UCR_UEA_datasets().load_dataset(dataset)
+train_x=np.swapaxes(X_train,1,2)
+test_x=np.swapaxes(X_test,1,2)
+train_y = y_train
+test_y=y_test
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
Trains a ResNet and saves the results.
+ +n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+model = ResNetBaseline(in_channels=6, num_pred_classes=n_pred_classes)
+fit(model, train_loader, test_loader)
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/ResNet'))
+Epoch: 1, Train loss: 1.186, Val loss: 1.212 +Epoch: 2, Train loss: 1.126, Val loss: 0.892 +Epoch: 3, Train loss: 0.814, Val loss: 0.763 +Epoch: 4, Train loss: 0.674, Val loss: 0.608 +Epoch: 5, Train loss: 0.552, Val loss: 0.423 +Epoch: 6, Train loss: 0.348, Val loss: 0.259 +Epoch: 7, Train loss: 0.192, Val loss: 0.132 +Epoch: 8, Train loss: 0.099, Val loss: 0.072 +Epoch: 9, Train loss: 0.021, Val loss: 0.007 +Epoch: 10, Train loss: 0.001, Val loss: 0.008 +Epoch: 11, Train loss: 0.004, Val loss: 0.009 +Epoch: 12, Train loss: 0.001, Val loss: 0.002 +Epoch: 13, Train loss: 0.0, Val loss: 0.0 +Epoch: 14, Train loss: 0.0, Val loss: 0.0 +Epoch: 15, Train loss: 0.0, Val loss: 0.0 +Epoch: 16, Train loss: 0.0, Val loss: 0.0 +Epoch: 17, Train loss: 0.0, Val loss: 0.001 +Epoch: 18, Train loss: 0.0, Val loss: 0.001 +Epoch: 19, Train loss: 0.0, Val loss: 0.002 +Epoch: 20, Train loss: 0.0, Val loss: 0.003 +Epoch: 21, Train loss: 0.0, Val loss: 0.003 +Epoch: 22, Train loss: 0.0, Val loss: 0.004 +Epoch: 23, Train loss: 0.0, Val loss: 0.004 +Epoch: 24, Train loss: 0.0, Val loss: 0.004 +Epoch: 25, Train loss: 0.0, Val loss: 0.005 +Epoch: 26, Train loss: 0.0, Val loss: 0.005 +Epoch: 27, Train loss: 0.0, Val loss: 0.005 +Epoch: 28, Train loss: 0.0, Val loss: 0.005 +Epoch: 29, Train loss: 0.0, Val loss: 0.005 +Epoch: 30, Train loss: 0.0, Val loss: 0.005 +Epoch: 31, Train loss: 0.0, Val loss: 0.005 +Epoch: 32, Train loss: 0.0, Val loss: 0.004 +Epoch: 33, Train loss: 0.0, Val loss: 0.004 +Epoch: 34, Train loss: 0.0, Val loss: 0.004 +Epoch: 35, Train loss: 0.0, Val loss: 0.004 +Epoch: 36, Train loss: 0.0, Val loss: 0.004 +Epoch: 37, Train loss: 0.0, Val loss: 0.004 +Epoch: 38, Train loss: 0.0, Val loss: 0.004 +Epoch: 39, Train loss: 0.0, Val loss: 0.004 +Epoch: 40, Train loss: 0.0, Val loss: 0.004 +Epoch: 41, Train loss: 0.0, Val loss: 0.003 +Epoch: 42, Train loss: 0.0, Val loss: 0.003 +Epoch: 43, Train loss: 0.0, Val loss: 0.003 +Epoch: 44, Train loss: 0.0, Val loss: 0.003 +Epoch: 45, Train loss: 0.0, Val loss: 0.003 +Epoch: 46, Train loss: 0.0, Val loss: 0.003 +Epoch: 47, Train loss: 0.0, Val loss: 0.003 +Epoch: 48, Train loss: 0.0, Val loss: 0.003 +Epoch: 49, Train loss: 0.0, Val loss: 0.003 +Epoch: 50, Train loss: 0.0, Val loss: 0.003 +Epoch: 51, Train loss: 0.0, Val loss: 0.003 +Epoch: 52, Train loss: 0.0, Val loss: 0.003 +Epoch: 53, Train loss: 0.0, Val loss: 0.002 +Epoch: 54, Train loss: 0.0, Val loss: 0.002 +Epoch: 55, Train loss: 0.0, Val loss: 0.002 +Epoch: 56, Train loss: 0.0, Val loss: 0.002 +Epoch: 57, Train loss: 0.0, Val loss: 0.002 +Epoch: 58, Train loss: 0.0, Val loss: 0.002 +Epoch: 59, Train loss: 0.0, Val loss: 0.002 +Epoch: 60, Train loss: 0.0, Val loss: 0.002 +Epoch: 61, Train loss: 0.0, Val loss: 0.002 +Epoch: 62, Train loss: 0.0, Val loss: 0.002 +Epoch: 63, Train loss: 0.0, Val loss: 0.002 +Epoch: 64, Train loss: 0.0, Val loss: 0.002 +Epoch: 65, Train loss: 0.0, Val loss: 0.002 +Epoch: 66, Train loss: 0.0, Val loss: 0.002 +Epoch: 67, Train loss: 0.0, Val loss: 0.002 +Epoch: 68, Train loss: 0.0, Val loss: 0.002 +Epoch: 69, Train loss: 0.0, Val loss: 0.002 +Epoch: 70, Train loss: 0.0, Val loss: 0.002 +Epoch: 71, Train loss: 0.0, Val loss: 0.002 +Epoch: 72, Train loss: 0.0, Val loss: 0.002 +Epoch: 73, Train loss: 0.0, Val loss: 0.002 +Epoch: 74, Train loss: 0.0, Val loss: 0.002 +Epoch: 75, Train loss: 0.0, Val loss: 0.001 +Epoch: 76, Train loss: 0.0, Val loss: 0.001 +Epoch: 77, Train loss: 0.0, Val loss: 0.001 +Epoch: 78, Train loss: 0.0, Val loss: 0.001 +Epoch: 79, Train loss: 0.0, Val loss: 0.001 +Epoch: 80, Train loss: 0.0, Val loss: 0.001 +Epoch: 81, Train loss: 0.0, Val loss: 0.001 +Epoch: 82, Train loss: 0.0, Val loss: 0.001 +Epoch: 83, Train loss: 0.0, Val loss: 0.001 +Epoch: 84, Train loss: 0.0, Val loss: 0.001 +Epoch: 85, Train loss: 0.0, Val loss: 0.001 +Epoch: 86, Train loss: 0.0, Val loss: 0.001 +Epoch: 87, Train loss: 0.0, Val loss: 0.001 +Epoch: 88, Train loss: 0.0, Val loss: 0.001 +Epoch: 89, Train loss: 0.0, Val loss: 0.001 +Epoch: 90, Train loss: 0.0, Val loss: 0.001 +Epoch: 91, Train loss: 0.0, Val loss: 0.001 +Epoch: 92, Train loss: 0.0, Val loss: 0.001 +Epoch: 93, Train loss: 0.0, Val loss: 0.001 +Epoch: 94, Train loss: 0.0, Val loss: 0.001 +Epoch: 95, Train loss: 0.0, Val loss: 0.001 +Epoch: 96, Train loss: 0.0, Val loss: 0.001 +Epoch: 97, Train loss: 0.0, Val loss: 0.001 +Epoch: 98, Train loss: 0.0, Val loss: 0.001 +Epoch: 99, Train loss: 0.0, Val loss: 0.001 +Epoch: 100, Train loss: 0.0, Val loss: 0.001 +Epoch: 101, Train loss: 0.0, Val loss: 0.001 +Epoch: 102, Train loss: 0.0, Val loss: 0.001 +Epoch: 103, Train loss: 0.0, Val loss: 0.001 +Epoch: 104, Train loss: 0.0, Val loss: 0.001 +Epoch: 105, Train loss: 0.0, Val loss: 0.001 +Epoch: 106, Train loss: 0.0, Val loss: 0.001 +Epoch: 107, Train loss: 0.0, Val loss: 0.001 +Epoch: 108, Train loss: 0.0, Val loss: 0.001 +Epoch: 109, Train loss: 0.0, Val loss: 0.001 +Epoch: 110, Train loss: 0.0, Val loss: 0.001 +Epoch: 111, Train loss: 0.0, Val loss: 0.001 +Epoch: 112, Train loss: 0.0, Val loss: 0.001 +Epoch: 113, Train loss: 0.0, Val loss: 0.001 +Epoch: 114, Train loss: 0.0, Val loss: 0.001 +Early stopping! ++
<All keys matched successfully>+
Using a interpretability algorithm consists of 4 steps:
+The TSR works on gradient-based models. Currently support for TensorFlow (TF) and PYTorch (PYT) is available. For PYT the Subclass Saliency_PTY is used, while TF expects the use of Saliency_TF.
+The Initialization takes the following arguments:
model: The model to be explaines. NumTimeStep: Number of Time Step.NumFetaures: Number Features.method: Saliency Methode to be used.
* Gradients (GRAD)
+* Integrated Gradients (IG)
+* Gradient Shap (GS)
+* DeepLift (DL)
+* DeepLiftShap (DLS)
+* SmoothGrad (SG)
+* Shapley Value Sampling(SVS)
+* Feature Ablation (FA)
+* Occlusion (FO)mode: Second dimension 'time' or 'feat'.model.eval()
+from TSInterpret.InterpretabilityModels.Saliency.TSR import TSR
+int_mod=TSR(model, train_x.shape[-1],train_x.shape[-2], method='GRAD', \
+ mode='feat')
+Prepeare the instance and the predicted label of the instance as parameters for the explain methods.
+item: item to be explainedlabels: predicted label for the item as class. TSR: Turns temporal rescaling on / off.item = np.array([test_x[0,:,:]])
+label = int(np.argmax(test_y[0]))
+exp=int_mod.explain(item,labels=label,TSR = True, attribution=0.0)
+All plot function take as input the item to be explained and the returned explanation. As as additonal option a figsize can be given. +For visualizing saliency there are two visualization options provided:
+int_mod.plot(np.array([test_x[0,:,:]]),exp,figsize=(15,15))
+NOT Time mode ++
+int_mod.plot(np.array([test_x[0,:,:]]),exp, heatmap = True)
+NOT Time mode ++
+Additional Examples, e.g. for the use with LSTM or TF can be found here.
+ +import sklearn
+import pickle
+import numpy as np
+import torch
+from ClassificationModels.CNN_T import ResNetBaseline, get_all_preds, fit, UCRDataset
+from ClassificationModels.LSTM_T import LSTM
+import matplotlib.pyplot as plt
+import seaborn as sns
+from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
+import pandas as pd
+import os
+from tslearn.datasets import UCR_UEA_datasets
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html + from .autonotebook import tqdm as notebook_tqdm ++
#dataset='GunPoint'f
+dataset='BasicMotions'
+train_x,train_y, test_x, test_y=UCR_UEA_datasets().load_dataset(dataset)
+enc1=sklearn.preprocessing.OneHotEncoder(sparse=False).fit(np.vstack((train_y.reshape(-1,1),test_y.reshape(-1,1))))
+pickle.dump(enc1,open(f'../../ClassificationModels/models/{dataset}/OneHotEncoder.pkl','wb'))
+
+train_y=enc1.transform(train_y.reshape(-1,1))
+test_y=enc1.transform(test_y.reshape(-1,1))
+/home/jacqueline/.local/share/virtualenvs/TSInterpret-x4eqnPOt/lib/python3.9/site-packages/sklearn/preprocessing/_encoders.py:972: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value. + warnings.warn( ++
n_pred_classes =train_y.shape[1]
+train_dataset = UCRDataset(train_x.astype(np.float64),train_y.astype(np.int64))
+test_dataset = UCRDataset(test_x.astype(np.float64),test_y.astype(np.int64))
+train_loader = torch.utils.data.DataLoader(train_dataset,batch_size=16,shuffle=True)
+test_loader = torch.utils.data.DataLoader(test_dataset,batch_size=1,shuffle=False)
+device='cpu'
+hidden_size=10
+rnn=0.1
+model = LSTM(6, hidden_size ,n_pred_classes,rnndropout=rnn).to(device)
+#fit(model,train_loader,test_loader)
+if dataset in os.listdir('../../ClassificationModels/models/'):
+ print('Folder exists')
+else:
+ os.mkdir(f'../../ClassificationModels/models/{dataset}')
+#torch.save(model.state_dict(), f'../../ClassificationModels/models/{dataset}/LSTM')
+model.load_state_dict(torch.load(f'../../ClassificationModels/models/{dataset}/LSTM'))
+
+model.eval()
+test_preds, ground_truth = get_all_preds(model, test_loader)
+ground_truth=np.argmax(ground_truth,axis=1)
+
+sns.set(rc={'figure.figsize':(5,4)})
+heatmap=confusion_matrix(ground_truth, test_preds)
+sns.heatmap(heatmap, annot=True)
+plt.savefig(f'../../ClassificationModels/models/{dataset}/LSTM_confusion_matrix.png')
+plt.close()
+acc= accuracy_score(ground_truth, test_preds)
+a = classification_report(ground_truth, test_preds, output_dict=True)
+dataframe = pd.DataFrame.from_dict(a)
+dataframe.to_csv(f'../../ClassificationModels/models/{dataset}/LSTMclassification_report.csv', index = False)
+Folder exists ++
item=test_x[0].reshape(1,-1,6)
+shape=item.shape
+_item= torch.from_numpy(item).float()
+model.eval()
+y_target= model(_item).detach().numpy()
+#y_target = torch.nn.functional.softmax(model(_item)).detach().numpy()
+from TSInterpret.InterpretabilityModels.Saliency.SaliencyMethods_PTY import Saliency_PTY
+int_mod=Saliency_PTY(model,NumTimeSteps=train_x.shape[-2],NumFeatures=train_x.shape[-1], method='FA', mode ='time')
+2023-09-18 13:59:41.345000: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. +To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. +2023-09-18 13:59:42.368222: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT ++
print(np.array([test_x[0,:,:]]).shape)
+(1, 100, 6) ++
exp=int_mod.explain(np.array([test_x[0,:,:]]),labels =1 ,TSR = True)
+int_mod.plot(np.array([test_x[0,:,:]]),exp)
+time mode ++
+int_mod.plot(np.array([test_x[0,:,:]]),exp, heatmap = True)
+time mode ++
+\n {translation(\"search.result.term.missing\")}: {...missing}\n
\n }\n \n