Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Explainability Evaluation] - Characterization score, and other ways to combine Fid+/- scores #5960

Closed
BlazStojanovic opened this issue Nov 12, 2022 · 0 comments

Comments

@BlazStojanovic
Copy link
Contributor

BlazStojanovic commented Nov 12, 2022

🚀 The feature, motivation and pitch

Implement functionality to combine Fidelity +/- metrics (#5958) features for the purpose of evaluation explainability. This feature request is a part of broader explainability evaluation metrics (parent issue #5628) which will be crucial for the new revamp of explainability (see roadmap #5520).

Brief overview

Fidelity measures check the explanations for their faithfulness to the model. In [1] there are several ways proposed for combining both fidelity scores, like

The area under the $\mathrm{Fid}+/(1-\text { Fid-) }$ curve - AUC

The characterization score: $\operatorname{charact} = \frac{w_{+}+w_{-}}{\frac{w_{+}}{f i d_{+}}+\frac{w_{-}}{1-f i d_{-}}}$

Implementation checklist

  • Implement the characterization score
  • Implement the AUC

References

  1. GraphFramEx: Towards Systematic Evaluation of Explainability Methods for Graph Neural Networks
rusty1s added a commit that referenced this issue Dec 15, 2022
…to combine Fid+/- scores (#6188)

This PR implements the features from #5960 

The tests and the math part is done, I'm somewhat lost with the sphinx
documentation and very much open to suggestions for the docstrings.

As #5962 also requires calculating an AUC and probably a weighted
harmonic mean for F_1 type scores these parts have been implemented as
separate helper functions.

Co-authored-by: Blaž Stojanovič <Blaz@stojanovic.si>
Co-authored-by: rusty1s <matthias.fey@tu-dortmund.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants