Explainable AI for Flair

[krzysztoffiok]

Hi,

is there anyone who tried/did anyone post an idea of integrating an Explainable AI (XAI) tool with Flair?

There are more or less universal solutions that work also for DL and NLP like LIME or better SHAP.

It would really be great to be able to explain why a given model predicts what it predicts and do that either separately for each text-instance or model-wise.

Homepage of SHAP: https://github.com/slundberg/shap

[alanakbik]

Would be interested in this as well. For character language models, there are some visualizations of hidden states that give some indication on whats happening (see for instance this blog post by Andrej Karpathy).

[Hellisotherpeople]

I've demonstrated that something like this is feasible with word embeddings in a repo on my GitHub called "active explainable classification" which uses Flair for embeddings and ELI5 for Lime

[djstrong]

I would be interested too (preferring SHAP).

[kdk2612]

Is the similar work being done for NER? Or if something is done, can you point me in the right direction?

[krzysztoffiok]

Hi again. @alanakbik, I've done a quick review of literature (i wouldn't call it systematic) but what i found about XAI in NLP classification is:

If you analyze text and use features that are understandable by humans e.g. provided by lexicon-based methods like LIWC, SEANCE, Term Frequency, and feed them into an ML model, than it is easy and possible to use out of the box packages like LIME or SHAP. With this packages you can achieve either instance level or model level explanations.

For text representation created by LSTMs based on LMs that provide simple static word embeddings (i.e. not changing with context of the token in a sentence) it is possible to create instance-level visualizations of rationale for model predictions as shown in [Li, J., Chen, X., Hovy, E., & Jurafsky, D. (2015). Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066.] and [Arras, L., Montavon, G., Müller, K. R., & Samek, W. (2017). Explaining recurrent neural network predictions in sentiment analysis. arXiv preprint arXiv:1706.07206.]. Also, [Karpathy, A. (2015). The unreasonable effectiveness of recurrent neural networks. Andrej Karpathy blog, 21, 23.] showed that this is possible for character-level LMs with Recurrent Neural Networks. Unfortunately these prediction models do not provide state of the art performance and there are no ready to use packages to try them on your model. In all these cases instance level explanations are presented.

Also, i found that if a more complex context-aware method creating token representations is used (like a transformer model), there are no methods that allow presentation of models rationale for predictions. The features that they produce are not interpretable and i haven't found any methods to map those embeddings back to tokens.

Do you think what i wrote is true? Did i miss something obvious?

[alanakbik]

Thanks for sharing the overview! I think there are a few tools for visualizing attention in transformers, such as https://github.com/jessevig/bertviz - maybe they are also used for visualizing attention in transformers that have been fine-tuned to certain tasks?

[krzysztoffiok]

Hmm thanks a lot for this link, i'll check it out definetely.

[krzysztoffiok]

@alanakbik thanks again, i see the tool is nice because first of all it works out of the box. It allows a very detailed inspection of what is going on in the model, which is nice. At the same time it doesn't offer any aggregated view, it is rather impossible to get an answer from this tool if one posed a question: "why did my model label this sentence as class x?" Actually i don't see any relation to classification task here, it's only what the attention mechanism focuses on. So maybe future will bring some sort of aggregation/reasoning on top of this extracted knowledge...

[krzysztoffiok]

Or is it that the tokens that are linked to [CLS] token can be considered as strongly influencing the classification output? For instance in the figure below the more distance tokens "wonderful" and "Dad" seem to be strongly connected with CLS. Do you think this might be the right way to interpret this?

[alanakbik]

I think so - if the CLS token is used for classification and the model is fine-tuned then maybe it could be interpreted this way. Of course normally there are many layers of self-attention, so I am not sure how this visualization deals with that.

[krzysztoffiok]

This figure was the output of head 0 layer 0, if i select differently then nothing reasonable is outputed (see below). Also its funny why did the tokenizer divide eulogy into e ul ogy...

[HiyaToki]

Hello,

I am new to using FLAIR. Is still an active endeavor for the contributors and developers of FLAIR?

Other NLP toolkits already have simple gradient visualization and other interpretation methods implemented (e.g. https://allennlp.org/interpret, and a demo of these https://demo.allennlp.org/sentiment-analysis/) Links to specific literature can be found through the second link. I think these methods could be a valuable asset if integrated in FLAIR.

If these are already implemented in FLAIR, could you please explain how I could use them?

Thanks!

[alanakbik]

It would be cool to have this in Flair - we ourselves don't currently have the capacity to integrate visualization options, but maybe someone in the community is interested to do this?

[robinvanschaik]

I think that explainable AI would be great!

Recently a Google offshoot called LIT was released.

While this repository looks nice, it is still in its infancy.

There is some documentation on how to add models to the LIT framework here.

However, I don't really have a grasp whether the implementation of adding new models will be scaleable, nor whether this progess will differentiate greatly between all the available (fine-tuned) models in FLAIR.

[robinvanschaik]

Adding to the discussion that CAPTUM has been used with FLAIR. I have not yet achieved this, but it should be possible.

meta-pytorch/captum#414 (comment)