Adding LOO PIT vignette

[ecoronado92]

Opening issue to discuss adding the following tutorial

https://tinyurl.com/loo-pit-tutorial

[jgabry]

Thanks @ecoronado92! I've wanted to have some more intuitive demonstrations of the LOO PIT stuff for a while, so this would be great to have as a vignette. I'll try to take a look at the source at https://github.com/ecoronado92/towards_data_science/blob/master/hierarchical_models/bayes_lmm/loo_pit_example/loo_pit.Rmd soon and give you some comments about anything that might need to be changed for this to work as a package vignette for CRAN (and any other comments I have if that's ok). Oh and just a heads up: if I don't get to it before Saturday it might not happen until the week after next (I'm moving next week).

Thanks again for making this tutorial and I'll get back to you soon!

[ecoronado92]

@jgabry no problem! I really enjoyed your paper, it has taught me so much about Bayesian models evaluation outside of PPCs. (I enjoyed it so much I wrote a couple Towards Data Science posts about it haha)

Thanks again, good luck with the move and Happy 4th of July!

[ecoronado92]

Hey @jgabry - hope the move went well! Just wanted to follow-up to continue our conversation :)

[avehtari]

Thanks for the proposal. The reasons for me why I haven't worked on vignette are

1. we should fix first the density estimate as it now is dropping near edges while it should look uniform for actually uniform data, see isssue Fixing the density estimation for ppc_loo_pit_overlay #171
2. there should be a note saying that discrete case requires some additional work Predictive Model Assessment for Count Data (doesn't seem to have open issue yet)

For 1. we are going to soon have also something that doesn't have the problems of kernel density estimate and histogram, but quick fix could also be useful. 2. has been delayed as we've been working on better approach for 1.

@ecoronado92 are you interested in a) fixing the density estimate using a kernel density estimate with boundary conditions, or b) adding nonrandomized uniform version of the PIT histogram as described in the reference mentioned above? I can make a separate issue.

[ecoronado92]

@avehtari No worries, thanks for getting back to me.
Ah I see, so these two things would need to be updated in the demo I made so it can be included as a package vignette?

I'm happy to address 1. with a quick fix using the packages mentioned in #171 and then we can move forward on the other points. Since my demo doesn't use the ppc_loo_pit_overlay() function and instead builds the empirical LOO PIT from scratch, should I add these changes to my demo or open a pull request?

Let me know if that works.

[avehtari]

Hmmm... There's a problem that we wouldn't like to add more dependencies to other packages (even that I suggested couple packages), so we should carefully consider what to do with this. @jgabry what do you think?

There is also an alternative ECDF and ECDF-diff based approach which would be better than density estimates and we should soon have code public for that.

Since my demo doesn't use the ppc_loo_pit_overlay() function and instead builds the empirical LOO PIT from scratch, should I add these changes to my demo or open a pull request?

For a vignette to be included in bayesplot package, it would be good to have ppc_loo_pit_overlay() used.

[ecoronado92]

@avehtari Ok, that makes sense...I can add ppc_loo_pit_overlay() but will wait to see what we decide on the ECDF part to start refactoring the current demo

[jgabry]

Sorry I haven't chimed in yet! After I moved we also ended up having some issues with the latest RStan release and I've been spending a lot of time responding to people with installation issues.

There's a problem that we wouldn't like to add more dependencies to other packages (even that I suggested couple packages), so we should carefully consider what to do with this. @jgabry what do you think?

Yeah it'd be better to have our own implementation that fixes the problem with the density plots. I'd rather not add another dependency just to call one function. However, we could add it to Suggests instead of Imports and then it wouldn't be required to have it when loading bayesplot, just for using the relevant function.

@ecoronado92 Do you want to take a look at those other packages and check if (a) it would be difficult to do our own implementation of the better density estimates or, if that seems like too much effort, then (b) if you can use one of those packages to fix the problem, in which case we'll add it as a suggested package?

There is also an alternative ECDF and ECDF-diff based approach which would be better than density estimates and we should soon have code public for that.

@avehtari It would be great to have this. Is one of your students working on this?

If we can fix the problem with the density estimation then I think it's probably worth adding this vignette and then updating it when the ECDF and ECDF-diff stuff is ready. Otherwise we can wait until we have that.

What do you both think?

[avehtari]

@avehtari It would be great to have this. Is one of your students working on this?

Yes. It takes some time to have it in vignette as it's not just a new plot, so we need to have proper description of the new parts and experiments to demonstrate that the behavior.

[ecoronado92]

@ecoronado92 Do you want to take a look at those other packages and check if (a) it would be difficult to do our own implementation of the better density estimates or, if that seems like too much effort, then (b) if you can use one of those packages to fix the problem, in which case we'll add it as a suggested package?

@jgabry Yes, I can look into those implementations in the packages suggested in #171 and see how difficult it could be to implement it from scratch. Hopefully it's straightforward.

How should I proceed in either case - a pull request in the script for ppc_loo_pit_overlay or would you prefer I start tinkering my demo linked above?

For a vignette to be included in bayesplot package, it would be good to have ppc_loo_pit_overlay() used.

@avehtari Apologies, I mispoke. I do use the ppc_loo_pit_overlay() in the demo as the ground truth.
The demo approximates the ppc_loo... plot by computing the same empirical CDF pvals using a brute-force loo refitting instead of PSIS, and uses animations so show this process to build some intuition.

[jgabry]

Ok great thanks!

How should I proceed in either case - a pull request in the script for ppc_loo_pit_overlay or would you prefer I start tinkering my demo linked above?

Whichever you prefer is fine by me. Presumably if you do it by tinkering with your demo and you get it working then we could apply the same approach in the ppc_loo_pit_overlay() code.

[ecoronado92]

@jgabry I ran some quick simulations and tested one of the packages. I wanted to run these by you before diving into more coding. Is this what we're trying to achieve?

Below is the code I used to generate a small random unif sample and some histograms with KDEs in blue (stats density()) and red (boundary-corrected)

library(evmix)
set.seed(15)
n=100 
x = runif(n)
xx = seq(-0.1, 1.1, 0.001)
hist(x, freq=FALSE, col = rgb(0.2,0.2,0.2,0.2))
lines(xx, dbckden(xx, x, lambda=bw.nrd0(x)/2,  xmax=1, bcmethod = "beta2", kernel = "gaussian"), 
      lwd = 2, col = "red")
lines(density(x), lty = 2, lwd = 2, col = "blue")

[jgabry]

Cool thanks! Yeah I think the red curve looks much better. @avehtari is this what you had in mind for the corrected density estimation?

[avehtari]

Can you try with larger n? Preferably the same n a used in bayesplot to generate the bunch of reference density lines