Is it normal that the estimated effective sample size goes down as more samples are added?

[JGameCreation]

I am using blackjax.diagnostics.effective_sample_size to keep sampling batches of 256 samples on 16 chains until I have achieved a certain effective sample size. However, I often noticed that the estimate of effective sample size goes down as more samples are added. As far as I understand, it is not possible that the actual effective sample size decreases as more samples are added. Is the estimator just biased to overestimate the effective sample size when there are relatively few samples?

[AdrienCorenflos]

That's interesting. Can you provide a MVP for this? In practice the ESS diagnostic is a statistic based on the empirical autocorrelation. It may be that the autocorrelation is worse at stationarity than at initialisation (that's usual) and that's what you are picking up.

Hard to say without more details.

[JGameCreation]

It will take me some time to create an MVP but what happens is that the chains are first concentrated on the main mode and then discover a lower probability area as more samples are generated:

[AdrienCorenflos]

Am I understanding from the second plot that the samples first start top left and then reach the bottom right? If that's the case, then my comment was correct: before stationary you have large moves with high probability, so the ESS would look large, but after that, it would decrease as the autocorrelation increases.

[JGameCreation]

No, the samples start bottom right and then reach the top left (look at the coordinates on the x and y axes)

[AdrienCorenflos]

Well, it's a bit weird but hard to tell what is going on without the problem at hand. Some hand-wavy explanations:

1. You calibrated the chain on the first mode, so it's now not compatible anymore with the other region
2. Your model is misspecified (the posterior is not proper) so the chain diverges
3. The chain is transitioning between two modes which may take a while

[JGameCreation]

Thanks for your advice! My existing hand-wavy theory is that there is this region with a relatively small probability mass that only gets sampled if there are a lot of samples drawn and that messes up the ESS calculations.

[AdrienCorenflos]

Also, not that it's my place to tell you, but in general, I would be very wary of stopping a Markov chain based on a stochastic criterion such as "predefined ESS" achieved. When you are doing so, you are by construction biasing your sample average.
I remember either Betancourt or Gelman advocating specifically against this type of practice with reason.

[JGameCreation]

I understand that this is a problem, but it raises the question of the value of a diagnostic tool that you are not allowed to use for making decisions, even if it may be in a less explicit way.