Re-running each MCMC saved sample to get a report file in Hake assessment

[k-doering-NOAA]

Imported from redmine, Issue #75754
Opened by @k-doering-NOAA on 2020-03-03
Status when imported: New

This first came up in #75174, but is a separate topic from the original issue being discussed there.

Kelli noted:

Thanks Ian for thinking of the hake assessment. We do in fact re-run each saved sample with bias adjustment 
ramp = 1.0 to generate a report file that can later be read in. Does anyone know how other Bayesian assessments 
written in ADMB get around this issue? Also, what other stocks turn on MCMC within an SS model?

and Rick responded:

Kelli and Ian,
I think we need a new issue for this topic of re-running posteriors to get
full report. I had no idea that was being done. Is there more that can be
added to posteriors.sso to obviate that need?

[k-doering-NOAA]

comment from @kellijohnson-NOAA on 2020-03-03:
I am uncertain the exact values that we are after but currently unavailable in the posteriors file. I will look and document here.

And, to answer Kathryn's question in the previous thread ... we re-run the model for each MCMC draw to mimic the results that would have been found but a full report file was not saved, which is why we turn off bias adjustment because we are trying to match MCMC conditions but using no estimation in the MLE framework. Basically, we just want a copy of the Report.sso file for each MCMC draw that was saved.

[k-doering-NOAA]

comment from @iantaylor-NOAA on 2020-03-03:
Here's a long illustrated answer to the question of what quantities are
used in the hake assessment that aren't in the standard MCMC output.

My memory is that the approach of re-running the model to get a full
Report.sso file (and later CompReport.sso as well) for each of the MCMC
samples started 4 or 5 years ago as a way to get expected values from the
survey to make this figure:
[image: image.png]
However, once we had all those report files, that allowed calculation of
the median and uncertainty intervals for the expected age comp values for
each year and data source that went into this figure (where the intervals
are too tiny to see in almost all cases, except for instance the 1980
cohort in 1983 and 1986, but the medians are more representative of the
MCMC results than the MLE proportions).
[image: image.png]

And also the selectivity by age and year for both fishery and survey which
allows representation of the uncertainty in the time-varying fishery
selectivity (which is much larger than the uncertainty in the baseline
selectivity shown in 1990 in the figure below):
[image: image.png]

The index values are already now available under derived quantities, but
those were only 25 values per MCMC sample anyway. The expected proportions
at age by year and fleet is about 900 values per MCMC sample. Selectivity
by age and year is about 200 unique values given that selectivity is
assumed constant from age 6 onward.

There may be other values I'm forgetting or that have been added since my
time on the hake team because once you have access to everything, it's easy
to think of additional diagnostics that would be good to convert from MLE
to MCMC.

As with the other issue about MCMC output (#75174), solving this problem
for hake, which is a relatively simple model, would be easier than
generalizing to the needs of other assessments.

One strategy that I've been pondering the is option to re-run these models
simultaneously in the cloud, extract the quantities of interest also in the
cloud and download an additional summary file containing just what's
needed. Teresa A'mar also looked into calling the write_bigoutput()
function within the mceval phase to create all these files automatically
without an external R script to facilitate that which would be more
efficient (though still leading to lots of big files).

On Tue, Mar 3, 2020 at 9:40 AM vlab.redmine@noaa.gov wrote:

[k-doering-NOAA]

comment from @iantaylor-NOAA on 2020-03-03:
I tried including images by replying by email rather than filling in text on the VLab redmine site, but one got lost and the other two don't seem to have appeared in the email notification, so now I'm trying again by using the redmine form and attaching them as PNG files and referencing within the text (by including filename within two exclamation marks).

Figure showing index uncertainty from MCMC results:
!hake_survey_uncertainty.png!

Figure showing age comp uncertainty from MCMC:
!hake_age_comp_uncertainty.png!

Figure showing selectivity uncertainty from MCMC:
!hake_selectivity_uncertainty.png!

[k-doering-NOAA]

comment from @kellijohnson-NOAA on 2020-03-03:
Thanks Ian for the list.
I am not super keen on having the only available option be running something in the cloud. I think that it should be accessible on individual desktops with the added benefit of it having the potential that someone could run it on the cloud if they want to set that up.

[k-doering-NOAA]

comment from @iantaylor-NOAA on 2020-03-03:
Kelli,
Could you speak to roughly how long it takes to get the extra Report files
and extract the info from them? My memory is that it's less than an hour,
certainly much faster than running the MCMC in the first place, although
that obviously depends on how many samples you're using. It seems possible
that even if the samples used to calculate reference points, quotas, etc.
are based on a larger MCMC sample, the figures could just use fewer than
1000 samples as a way to speed things up.
-Ian

On Tue, Mar 3, 2020 at 11:30 AM vlab.redmine@noaa.gov wrote:

[k-doering-NOAA]

comment from @kellijohnson-NOAA on 2020-03-03:
I just had Chris check and it was 35 minutes for 2000 samples.

[k-doering-NOAA]

comment from @RickMethot on 2020-03-05:
Good exploration so far. Here are some other ideas:

1. Issue #46175 would provide for custom control on which parts of report.sso are generated. This could allow for quicker reporting of just what you need.
2. Turning on the writing of those reports during MCEVAL should be easy.
3. Writing those reports in append mode (like the data.ss_new file) would also be easy. Or creating a way to write each as a named file.
4. An alternative could be to augment the cumreport.sso which writes in append mode for each run.