towards first paper results

[ismael-mendoza]

0.1 (artificial ellipticities, low noise)

First, we need to setup the infrastructure to produce and evaluate shear posteriors. We do this following the procedure in Bernstein+ 2014 to produce artificial ellipticity samples, and use these samples to infer shear posteriors with the Schneider+2014 methodology

• (0.1) check calibration as a function of priors and n_gals #19
• (0.1) compare jackknife errors with Bayesian uncertainty on multiplicative bias #18
• implement set of benchmarks for CI #4
• setup README tagging latest benchmark plots with date #24
• setup reproducible script to easily diagnose shear posteriors
• look through examples of calibration in literature #25
• examine dependence on calibration + bias on (effective) number of interim posterior ellipticity samples
• setup script to obtain shear posterior by combining galaxy batches assuming a Gaussian approximation
• setup script to cache artificial interim posterior ellipticities #14
• create script to obtain shear posterior samples given interim posterior ellipticities #15
• investigate we can run standard MCMC for non-galaxy image chains #7
• simplify jacobian calculation in likelihood #17

0.2 (artificial ellipticities, realistic noise)

The goal here is to evaluate the feasibility of running the shear posteriors chain with a much larger number of ellipticities (~10^7). Next, we want to check that these posteriors are correct at least in mean, and to the extent possible, in terms of the uncertainty.

• (0.2) investigate shape noise cancellation in this context
• decide whether we should clip noisy, observed elliptcities
• (0.2) benchmark time and memory for full noise test
• (0.2) consider running and comparing full noise test with sub-posterior gaussian combination
• (0.2) check posterior bias is within requirements
• (0.2) assess posterior uncertainty
• (0.2) compare results with Bernstein+2014, reproduce setup

0.3 (end-to-end test, low noise)

Here the goal is to check that the ellipticity interim posteriors from actual NUTS chains can be used to infer shear accurately. We will start with a low-noise regime. If we want to thoroughly asses the posterior calibration, speeding up the NUTS chains by avoiding the warmup will be fruitful so we also investigate this idea more in this section.

I think we want to perform this tests first on the same galaxy, different noise realizations. Need to think about what the prior should be.

• test whether we can run NUTS without warmup on different noise realizations of same galaxy #3
• (0.3) think of changes to shear likelihood when using interim posterior samples from images
• (0.3) what prior and interim prior should we use for galaxy properties?
• (0.3) setup script to cache interim posterior samples from NUTS chains
• (0.3) setup pipeline script to ingest interim posteriors from NUTS and return shear posterior
• (0.3) asses shear posterior bias
• (0.3) assess shear posterior calibration

0.4 (Galaxies with realistic priors)

In this part, we proceed with running NUTS on galaxies with the realistic prior distribution of properties that we will use for the paper. We first benchmark and evaluate the convergence of these NUTS chains in this more realistic setting. We investigate small optimizations we can make by perhaps looking at the tuned parameters as a fnc. of SNR which I think will be the most impact property on the final values of these parameters. Next, we run a small test on high SNR galaxies to infer their shear posterior and compare with the previous two parts.

• setup prior for properties of Gaussian galaxies
• (0.4) select interim prior to be use in experiments
• (0.4) investigate SBI as an alternative to initialization NUTS (step size + mass matrix)
• (0.4) benchmark NUTS chains on Gaussian galaxy images with realistic prior
• (0.4) investigate if we need full rank mass matrix for good chain convergence
• (0.4) make plots of tuned parameters as a fnc of SNR
• (0.4) run and evaluate shear posterior on low # of galaxies with high SNR

0.5 (full test on Gaussian galaxies)

• (0.5) explore ways to optimize sampling galaxy properties #26
• obtain one full shear posterior on 10^7 gaussian galaxies with realistic prior and noise level

0.6 (Extensions)

• more complicatd parametric models
• evaluate model bias (in posterior mean and uncertainty)
• investigate ways to mitigate model bias

Code improvements

• add split-gelman-rubin as metric in diagnostics #23
• use #SBATCH --qos=regular in slurm scripts #20
• add more automatic logging framework #16
• create simple io function to save data to h5py #12
• create simple function to create contour plot (boiler plate) #13