[FR] Use new num_chains arg in cmdstan for hmc, fixed params, and generated quantities

[SteveBronder]

Is your feature request related to a problem? Please describe.

So once stan-dev/cmdstan#987 goes into cmdstan, as long as the user compiled with threading on then we should be able to run multiple chains within one Stan program by specifying num_chains inside of the sample argument!

Describe the solution you'd like

Is there a reasonable line for cmdstanr to check if the method is hmc with dense or diag_e inverse metric and then when the user specifies parallel chains, internally cmdstanr passes those as command args to cmdstan instead of using R level parallelism? This should save folks a lot of memory and likewise give a nice little speed bump.

This would also have to wait / check that cmdstanr is at least greater than version 2.27 and has compiled with threading on

Describe alternatives you've considered

I'm not familiar enough with the internals of cmdstanr to know other alts, though I'm happy to discuss whatever can work for yinz!

Additional context
Add any other context about the feature request here.

[rok-cesnovar]

@SteveBronder Thanks! I set the milestone for after the 2.27++ release.

Yeah, I think this will just be a special case if version > 2.27, STAN_THREADS is on and the appropriate methods are used then just run a single executable instead of running multiple and waiting for feedback (https://github.com/stan-dev/cmdstanr/blob/master/R/run.R#L344).

So this will work with sampling for both hmc and nuts and for both dense and diag inverse metric?

[SteveBronder]

So this will work with sampling for both hmc and nuts and for both dense and diag inverse metric?

It's just nuts with the dense and diag metric. Once I have all the cmdstan stuff in for nuts then I'm going to go back to the Stan repo and add it for hmc and all the other stuff in sampling

[rok-cesnovar]

Yeah, no rush. Was just trying to double check. These 2 are definitely the most used so no rush for the rest.

[SteveBronder]

Just to update this is in now! Should we wait for cmdstan 2.27+? I think if folks are using an older version of cmdstan with multiple threads we would have to do this check anyway?

[rok-cesnovar]

I am going to start playing with this and add it soon-ish, just because I am excited for the feature.

For these kind of new features we always check for versions anyway, so we will definitely only use this with 2.28+
The only problem will be checking the version with a cloned cmdstan (before the next release), because we leave the Github version 2.x.0 and its hard to distinguish. But there will be a release in a few weeks anyway.

[SteveBronder]

^wonder if once we do a release we should bump devel up to like 2.x.x.1 just to be like "this is not 2.x.x"