Getting Feature Importances of Individual Features from Interaction Terms

[jinlow]

I know that I can get feature importance for each term, with the term_importances, however this includes importances of single terms, as well as interaction terms. Is there a way to distribute the importance of the interaction terms back to their individual features?

Basically what I want is a single vector, N number of features long, where N is the number of input features, to use in variable reduction techniques, or for model interpretation. It's hard to know the importance of a feature in a model precisely, when you can't easily distribute it's importance from when it's used in interactions.

It's my understanding that these interaction terms are essentially a tree structure? Perhaps this could be traversed to determine how the weighted average of bin weights could be attributed to the individual feature of an interaction?

I may be missing a method, or misunderstanding how this is working as well :)

[paulbkoch]

@jinlow -- One principled way that you could do this would be to calculate the SHAP values for the EBM with interactions. For EBMs without interactions, the EBM importances are exactly the same as the SHAP values. For EBMs with interactions, there is definitely a way to do this using TreeSHAP, since every EBM can be losslessly converted into an ensemble of decision trees. We don't have any code that does this translation though, so it would be a bit of an undertaking if you wanted to use TreeSHAP. A simpler solution would be to use KernelSHAP.

[jinlow]

Thanks @paulbkoch this makes sense. I will explore these methods.

Similarly, to using treeshap to calculate the overall feature importance's, this could be used then I assume to calculate record level contribution values using treeshap?

Is there general appetite for this that you have seen? I would possibly be interested in working on the code to convert the model into a decision tree ensemble.

[paulbkoch]

Yes, conversion of an EBM into a tree model would allow calculating the SHAP values in a more efficient and precise way than using kernel SHAP.

Having a tree converter would definitely be a valuable contribution @jinlow. There are some deployment environments where installing newer packages like interpret are forbidden. In those cases being able to convert an EBM into another model could allow for deployment. Secondly, since trees are so ubiquitous, I imagine there could be many oddball scenarios where having a tree representation would allow a user to perform an analysis on an EBM in a way that wouldn't otherwise be possible, like in this SHAP example.

There are obviously a few possible model formats that could be used. I'd be happy with any of them, but if you're interested in a recommendation, I think HistGradientBoostingRegressor or GradientBoostingRegressor might be a good place to start since they are represented as pure python objects, unlike the more complicated XGBoost/LightGBM/catboost classes. SHAP handles them here:

https://github.com/shap/shap/blob/8e9fddd96ccdae6ca74b0de33f6abf386409bd10/shap/explainers/_tree.py#L1057-L1199

Some other resources that you might find useful:
EBM internal representation: https://interpret.ml/docs/ebm-internals-regression.html

You might also find some useful ideas in the following packages. I think they might internally convert EBMs into something tree-like:
https://github.com/interpretml/ebm2onnx
https://github.com/kaspersgit/ml_2_sql
https://github.com/jpmml/sklearn2pmml