Tests are failing for `VecCorrBijector` in _very_ rare scenarios

See for example https://github.com/TuringLang/Bijectors.jl/actions/runs/9000497684/job/24724777705?pr=304#step:5:396

This was first encountered in #529, which does not touch that part of the code at all, i.e. it seems random. _But_ I haven't been able to reproduce this (even sampling thousands of realizations).