MPI warmup auto #2886
MPI warmup auto #2886
Conversation
…imental/warmup-auto
…ests for automatically picked metric
so that cmdstan has access to the parameter
Don't use first samples for warmup
Changed auto adaptation to specific ignore init_buffer samples
| logger.info(message); | ||
| } | ||
|
|
||
| /* |
There was a problem hiding this comment.
can we keep the old v1? It helps when I go back do specific sampler testing. All I need to do then is to switch the sampler call.
| ~adapt_auto_e_nuts() {} | ||
|
|
||
| sample | ||
| transition(sample& init_sample, callbacks::logger& logger) { |
There was a problem hiding this comment.
I've updated the call to mpi adapter, see
. Its logic is easier to understand. It also makes single chain default back to regular warmup.
|
|
||
| if (update) { | ||
| //std::cout << this->z_.inv_e_metric_ << std::endl; | ||
| this->z_.is_diagonal_ = reinterpret_cast<mpi_auto_adaptation<Model> *>(this->var_adapt)->is_diagonal_; |
There was a problem hiding this comment.
I use CRTP to access things like z_. If this is only for MPI you can put it inside var_adapt's learn_metric virtual function. Just want to explain how it's done in other samplers, not something you have to do at this point, since the code is still evolving.
| template <typename Model> | ||
| class mpi_auto_adaptation : public mpi_metric_adaptation { | ||
| #ifdef STAN_LANG_MPI | ||
| using est_t = stan::math::mpi::mpi_covar_estimator; |
There was a problem hiding this comment.
I've updated the dense_e 's covar calculation and no longer use the naive version of mpi_covar_estimator. It's intended to improve communication performance. It's ok for you to use the old version.
| }; | ||
|
|
||
| template <class Model, class RNG> | ||
| struct has_cross_chain_warmup<mcmc::adapt_auto_e_nuts<Model, RNG>> { |
| using stan::math::mpi::Communicator; | ||
|
|
||
| if (Session::is_in_inter_chain_comm(num_chains)) { | ||
| if (file_name.size() > 0 && num_chains > 1 && Session::is_in_inter_chain_comm(num_chains)) { |
| if (file_name.size() > 0 && num_chains > 1 && Session::is_in_inter_chain_comm(num_chains)) { | ||
| const Communicator& comm = Session::inter_chain_comm(num_chains); | ||
| file_name = "mpi." + std::to_string(comm.rank()) + "." + file_name; | ||
| file_name = file_name + "." + "mpi." + std::to_string(comm.rank()); |
There was a problem hiding this comment.
file_name could be output.csv so you'll be looking at output.csv.mpi.1.
yizhang-yiz
left a comment
yizhang-yiz
left a comment
There was a problem hiding this comment.
See the comments. Since we were doing it the same time some things here are no longer needed. In particular, v2 already supports degenerating to regular chain when only one MPI proc is supplied.
yizhang-yiz
force-pushed
the
mpi_warmup_v2
branch
3 times, most recently
from
a7c61ac to
405266d
Compare
Got auto adaptation working with merged mpi_warmup_v2
|
This can close. @yizhang-yiz probably has a more up to date implementation of this elsewhere. |
Submission Checklist
./runTests.py src/test/unitmake cpplintSummary
This pulls in the auto metric from: #2815
How to Verify
This is part of: stan-dev/cmdstan#821
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses: