Integrate standalone function compilation with RStan #422

martinmodrak · 2017-06-16T07:16:51Z

Summary:

The development version of Stan supports standalone compilation of the functions block (stan-dev/stan#2267), this should be accessible from R.

Description:

Design and implement an interface in RStan to take advantage of standalone function compilation. I am currently unsure about the best way to do this, but will update in discussion here. I am also willing to implement this (on the horizon of few weeks), but cannot assign myself as I am not part of this project.

bgoodri · 2017-06-25T05:19:19Z

I tried to put this in, but there seem to be several issues.

The using stan::math::lgamma;in the global namespace (I have never understood why lgamma has to be handled differently from every other function in Stan Math anyway):

file61be757ce77f.cpp:11:19: error: target of using declaration conflicts with declaration already in scope
using stan::math::lgamma;
                  ^
/usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/scal/fun/lgamma.hpp:36:19: note: target of using declaration
    inline double lgamma(double x) {
                  ^
/usr/include/x86_64-linux-gnu/bits/mathcalls.h:261:13: note: conflicting declaration
__MATHCALL (lgamma,, (_Mdouble_));
            ^

Putting // [[Rcpp::export]] before templated functions. Rcpp can only export the versions that specialize to doubles

file61be757ce77f.cpp:175:42: error: unknown type name 'T0__'
    Rcpp::traits::input_parameter< const T0__& >::type a(aSEXP);

I am still not sure how a R user would call a function ending in _rng without having to manually set up a Boost random-number generator (and be responsible for manually destroying it at the end).
Is it correct that if you pass a length 0 std::vector<std::string> to the namespaces argument, then everything gets put into the global namespace? That is the only way Rcpp can export functions.

bgoodri · 2017-06-25T05:22:25Z

What I have so far is in commit 4d10329 but I had to comment out the version of expose_stan_functions that calls it.

bob-carpenter · 2017-06-25T05:36:50Z

I don't think there's any difference with lgamma now. Historically, there's been a huge mess with different compilers throwing the TR1/C++11/C functions into the top-level :: namespace and into the std:: namespace. This should be cleaned up now so that lgamma works the same way as everythnig else.
I don't know where that's coming from---there should only be double and int types---is int OK?
How would you like the _rng functions to work? Take a seed as an argument and generate an RNG internally? They take an RNG reference as is because everything needs to be replicable on a chain.
I hope that's how it would behave.

bgoodri · 2017-06-25T07:38:23Z

int types are fine. What is not fine is stuff like

// [[Rcpp::export]]
template <typename T0__>
typename boost::math::tools::promote_args<T0__>::type
...

I think the vast majority of people calling a user-defined function that ends in _rng from R would expect it to work pretty much like rnorm; you just call it with a mean and standard deviation and don't worry about messing with the seed. We achieved that in the way expose_stan_functions worked before, at the cost that you cannot reset the seed after it has been passed once without recompiling the thing, which a small number of people are really bothered by.

martinmodrak · 2017-06-26T08:49:06Z

Just wanted to let you know, that I'll be happy to work to resolve the issues, but won't be able to dedicate time before Friday.

sakrejda · 2017-06-26T14:53:50Z

@bgoodri re: #3 above: 1) we add an rstan function that generates an RNG and returns an R object referring to it (so that it is not deleted, this is what Rcpp::XPtr is for); 2) the _rng functions require this RNG pointer as one of their arguments; 3) we can create a default RNG in the rstan namespace and have the _rng functions default to using that one. I've done similar setups before and I'm happy to help with this when it's time.

bob-carpenter · 2017-06-26T17:25:39Z

Should we piggyback on R's PRNG to initialize that so that the behavior is replicable with the same R seed?

We could expose two additional functions for each foo_rng defined in a Stan program:

template <class Rng>
foo_rng(..., Rng& rng);

Namely

foo_rng(...);  // built-in RNG or generate seed from R?
foo_rng(..., int seed);  // use seed to construct PRNG

@bgoodri suggested the first of these is what R users will expect. I can see two alternative implementations:

@sakrejda's suggestion: use a global PRNG from RStan's namespace
Generate a seed from R's standard PRNG

martinmodrak · 2017-07-04T10:12:48Z

So I tried to fix the RCpp export stuff and lgamma on a branch: stan-dev/stan#2347 Hope that helps.

bgoodri · 2017-09-09T18:53:57Z

@sakrejda As of commit 09cefa3 we might be down to RNG issues with standalone functions. You first have to install StanHeaders (make sure to change the two subrepos under StanHeaders/inst/include to point to the master branches locally) and rstan from GitHub. Then try to call expose_stan_functions on a Stan program with user-defined functions. Even though there are no user-defined _rng functions, I get from clang++-6.0:

In file included from file3e336d973cf.cpp:148:
In file included from /usr/lib/R/site-library/Rcpp/include/Rcpp.h:27:
In file included from /usr/lib/R/site-library/Rcpp/include/RcppCommon.h:179:
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap.h:523:22: error: no viable conversion from 'const boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> >' to 'SEXP' (aka 'SEXPREC *')
                SEXP x = object;
                     ^   ~~~~~~
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap.h:729:13: note: in instantiation of function template specialization 'Rcpp::internal::wrap_dispatch_unknown_iterable<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
            return wrap_dispatch_unknown_iterable(object, typename ::Rcpp::traits::has_iterator<T>::type());
                   ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap.h:767:20: note: in instantiation of function template specialization 'Rcpp::internal::wrap_dispatch_unknown<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
            return wrap_dispatch_unknown(object, typename ::Rcpp::traits::is_convertible<T,SEXP>::type());
                   ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap.h:784:20: note: in instantiation of function template specialization 'Rcpp::internal::wrap_dispatch_eigen<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
            return wrap_dispatch_eigen(object, typename traits::is_eigen_base<T>::type());
                   ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap.h:804:20: note: in instantiation of function template specialization 'Rcpp::internal::wrap_dispatch_unknown_importable<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
            return wrap_dispatch_unknown_importable(object, typename ::Rcpp::traits::is_importer<T>::type());
                   ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/wrap_end.h:30:25: note: in instantiation of function template specialization 'Rcpp::internal::wrap_dispatch<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
          return internal::wrap_dispatch( object, typename ::Rcpp::traits::wrap_type_traits<T>::wrap_category() ) ;
                           ^
file3e336d973cf.cpp:156:29: note: in instantiation of function template specialization 'Rcpp::wrap<boost::random::additive_combine_engine<boost::random::linear_congruential_engine<unsigned int, 40014, 0, 2147483563>, boost::random::linear_congruential_engine<unsigned int, 40692, 0, 2147483399> > >' requested here
    rcpp_result_gen = Rcpp::wrap(__create_rng(seed));
                            ^
In file included from file3e336d973cf.cpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/src/stan/model/standalone_functions_header.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core.hpp:14:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/core/matrix_vari.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/rev/mat/fun/Eigen_NumTraits.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:4:
In file included from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/Dense:1:
In file included from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/Core:437:
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/Matrix.h:296:22: error: no matching member function for call to '_init1'
      Base::template _init1<T>(x);
      ~~~~~~~~~~~~~~~^~~~~~~~~
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/Exporter.h:31:28: note: in instantiation of function template specialization 'Eigen::Matrix<double, -1, 1, 0, -1, 1>::Matrix<SEXPREC *>' requested here
                    Exporter( SEXP x ) : t(x){}
                                         ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/as.h:87:41: note: in instantiation of member function 'Rcpp::traits::Exporter<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::Exporter' requested here
            ::Rcpp::traits::Exporter<T> exporter(x);
                                        ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/as.h:152:26: note: in instantiation of function template specialization 'Rcpp::internal::as<Eigen::Matrix<double, -1, 1, 0, -1, 1> >' requested here
        return internal::as<T>(x, typename traits::r_type_traits<T>::r_category());
                         ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/InputParameter.h:72:54: note: in instantiation of function template specialization 'Rcpp::as<Eigen::Matrix<double, -1, 1, 0, -1, 1> >' requested here
        ConstReferenceInputParameter(SEXP x_) : obj( as<T>(x_) ){}
                                                     ^
file3e336d973cf.cpp:166:91: note: in instantiation of member function 'Rcpp::ConstReferenceInputParameter<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::ConstReferenceInputParameter' requested here
    Rcpp::traits::input_parameter< const Eigen::Matrix<double, Eigen::Dynamic,1>& >::type x(xSEXP);
                                                                                          ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:768:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'Eigen::Index' (aka 'long') for 1st argument
    EIGEN_STRONG_INLINE void _init1(Index size, typename internal::enable_if<    (Base::SizeAtCompileTime!=1 || !internal::is_convertible<T, Scalar>::value)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:804:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'const Eigen::PlainObjectBase<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::Scalar *' (aka 'const double *') for 1st argument
    EIGEN_STRONG_INLINE void _init1(const Scalar* data){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:818:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'const Eigen::Matrix<double, -1, 1, 0, -1, 1>' for 1st argument
    EIGEN_STRONG_INLINE void _init1(const Derived& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:782:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC *>'
    EIGEN_STRONG_INLINE void _init1(const Scalar& val0, typename internal::enable_if<Base::SizeAtCompileTime==1 && internal::is_convertible<T, Scalar>::value,T>::type* = 0)
                             ^                                             ~~~~~~~~~
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:791:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC **>'
    EIGEN_STRONG_INLINE void _init1(const Index& val0,
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:811:30: note: candidate template ignored: could not match 'DenseBase<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const DenseBase<OtherDerived>& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:825:30: note: candidate template ignored: could not match 'EigenBase<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const EigenBase<OtherDerived>& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:831:30: note: candidate template ignored: could not match 'ReturnByValue<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const ReturnByValue<OtherDerived>& other)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:839:30: note: candidate template ignored: could not match 'RotationBase<type-parameter-0-1, ColsAtCompileTime>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const RotationBase<OtherDerived,ColsAtCompileTime>& r)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:847:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC *>'
    EIGEN_STRONG_INLINE void _init1(const Scalar& val0,
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:859:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC **>'
    EIGEN_STRONG_INLINE void _init1(const Index& val0,
                             ^
In file included from file3e336d973cf.cpp:148:
In file included from /usr/lib/R/site-library/Rcpp/include/Rcpp.h:27:
In file included from /usr/lib/R/site-library/Rcpp/include/RcppCommon.h:160:
In file included from /usr/lib/R/site-library/Rcpp/include/Rcpp/as.h:25:
/usr/lib/R/site-library/Rcpp/include/Rcpp/internal/Exporter.h:31:28: error: cannot initialize a member subobject of type 'std::basic_ostream<char> *' with an lvalue of type 'SEXP' (aka 'SEXPREC *')
                    Exporter( SEXP x ) : t(x){}
                                         ^ ~
/usr/lib/R/site-library/Rcpp/include/Rcpp/as.h:87:41: note: in instantiation of member function 'Rcpp::traits::Exporter<std::basic_ostream<char> *>::Exporter' requested here
            ::Rcpp::traits::Exporter<T> exporter(x);
                                        ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/as.h:152:26: note: in instantiation of function template specialization 'Rcpp::internal::as<std::basic_ostream<char> *>' requested here
        return internal::as<T>(x, typename traits::r_type_traits<T>::r_category());
                         ^
/usr/lib/R/site-library/Rcpp/include/Rcpp/InputParameter.h:34:38: note: in instantiation of function template specialization 'Rcpp::as<std::basic_ostream<char> *>' requested here
        inline operator T() { return as<T>(x) ; }
                                     ^
file3e336d973cf.cpp:168:44: note: in instantiation of member function 'Rcpp::InputParameter<std::basic_ostream<char> *>::operator std::basic_ostream<char> *' requested here
    rcpp_result_gen = Rcpp::wrap(Soules(x, pstream__));
                                           ^
1 warning and 3 errors generated.

bgoodri · 2017-09-10T21:02:19Z

Upon further review, it appears as if the problem is not merely with the RNGs or with the pstream__. After hacking around all that, I still get instantiation errors from Eigen:

In file included from wtf.cpp:4:
In file included from /tmp/foo.hpp:5:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat.hpp:9:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/meta/get.hpp:4:
In file included from /usr/local/lib/R/site-library/StanHeaders/include/stan/math/prim/mat/fun/Eigen.hpp:4:
In file included from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/Dense:1:
In file included from /usr/local/lib/R/site-library/RcppEigen/include/Eigen/Core:437:
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/Matrix.h:296:22: error: no matching member function for call to '_init1'
      Base::template _init1<T>(x);
      ~~~~~~~~~~~~~~~^~~~~~~~~
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/internal/Exporter.h:31:28: note: in instantiation of function template specialization 'Eigen::Matrix<double, -1, 1, 0, -1, 1>::Matrix<SEXPREC *>' requested here
                    Exporter( SEXP x ) : t(x){}
                                         ^
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/as.h:87:41: note: in instantiation of member function 'Rcpp::traits::Exporter<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::Exporter' requested here
            ::Rcpp::traits::Exporter<T> exporter(x);
                                        ^
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/as.h:152:26: note: in instantiation of function template specialization 'Rcpp::internal::as<Eigen::Matrix<double, -1, 1, 0, -1, 1> >' requested here
        return internal::as<T>(x, typename traits::r_type_traits<T>::r_category());
                         ^
/usr/local/lib/R/site-library/Rcpp/include/Rcpp/InputParameter.h:72:54: note: in instantiation of function template specialization 'Rcpp::as<Eigen::Matrix<double, -1, 1, 0, -1, 1> >' requested here
        ConstReferenceInputParameter(SEXP x_) : obj( as<T>(x_) ){}
                                                     ^
wtf.cpp:154:91: note: in instantiation of member function 'Rcpp::ConstReferenceInputParameter<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::ConstReferenceInputParameter' requested here
    Rcpp::traits::input_parameter< const Eigen::Matrix<double, Eigen::Dynamic,1>& >::type x(xSEXP);
                                                                                          ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:768:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'Eigen::Index' (aka 'int') for 1st argument
    EIGEN_STRONG_INLINE void _init1(Index size, typename internal::enable_if<    (Base::SizeAtCompileTime!=1 || !internal::is_convertible<T, Scalar>::value)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:804:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'const Eigen::PlainObjectBase<Eigen::Matrix<double, -1, 1, 0, -1, 1> >::Scalar *' (aka 'const double *') for 1st argument
    EIGEN_STRONG_INLINE void _init1(const Scalar* data){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:818:30: note: candidate function not viable: cannot convert argument of incomplete type 'SEXPREC *const' to 'const Eigen::Matrix<double, -1, 1, 0, -1, 1>' for 1st argument
    EIGEN_STRONG_INLINE void _init1(const Derived& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:782:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC *>'
    EIGEN_STRONG_INLINE void _init1(const Scalar& val0, typename internal::enable_if<Base::SizeAtCompileTime==1 && internal::is_convertible<T, Scalar>::value,T>::type* = 0)
                             ^                                             ~~~~~~~~~
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:791:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC **>'
    EIGEN_STRONG_INLINE void _init1(const Index& val0,
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:811:30: note: candidate template ignored: could not match 'DenseBase<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const DenseBase<OtherDerived>& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:825:30: note: candidate template ignored: could not match 'EigenBase<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const EigenBase<OtherDerived>& other){
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:831:30: note: candidate template ignored: could not match 'ReturnByValue<type-parameter-0-1>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const ReturnByValue<OtherDerived>& other)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:839:30: note: candidate template ignored: could not match 'RotationBase<type-parameter-0-1, ColsAtCompileTime>' against 'SEXPREC *const'
    EIGEN_STRONG_INLINE void _init1(const RotationBase<OtherDerived,ColsAtCompileTime>& r)
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:847:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC *>'
    EIGEN_STRONG_INLINE void _init1(const Scalar& val0,
                             ^
/usr/local/lib/R/site-library/RcppEigen/include/Eigen/src/Core/PlainObjectBase.h:859:30: note: candidate template ignored: substitution failure [with T = SEXPREC *]: implicit instantiation of undefined template 'Eigen::internal::enable_if<false, SEXPREC **>'
    EIGEN_STRONG_INLINE void _init1(const Index& val0,
                             ^
1 warning and 1 error generated.

bgoodri · 2017-09-10T21:40:19Z

Okay, as of b4ae642, which revives a few hacks from how expose_stan_functions worked before, it is working unless there is a user-defined _rng function. It fails with

file28737ad1fb81.cpp:218:36: error: use of undeclared identifier 'RNG'
    Rcpp::traits::input_parameter< RNG& >::type base_rng__(base_rng__SEXP);
                                   ^
file28737ad1fb81.cpp:218:41: error: expected expression
    Rcpp::traits::input_parameter< RNG& >::type base_rng__(base_rng__SEXP);
                                        ^
file28737ad1fb81.cpp:218:44: error: no type named 'type' in the global namespace
    Rcpp::traits::input_parameter< RNG& >::type base_rng__(base_rng__SEXP);
                                         ~~^
1 warning and 3 errors generated.

So, we need a way to deal with RNG before we can release RStan 2.17.x. @sakrejda ?

sakrejda · 2017-09-11T01:21:25Z

That's great! I'll look at it now but I can't code anything till Tuesday (I'm on a chrome book till then).

sakrejda · 2017-09-11T01:39:17Z

While I'm waiting on this cheapo laptop to install all stuff required for rstan, any chance you could dump the code here that generates that error? It's the c++ generated by Stan, right?

bgoodri · 2017-09-11T02:01:49Z

I'm on my phone but it should just be expose_stan_functions (foo.stan) where foo.stan has any function ending in _rng. On Sep 10, 2017 9:39 PM, "Krzysztof Sakrejda" <notifications@github.com> wrote: While I'm waiting on this cheapo laptop to install all stuff required for rstan, any chance you could dump the code here that generates that error? It's the c++ generated by Stan, right? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqoeOFdyz0Tsia2mYj_ISYuh1SV0Tks5shI9GgaJpZM4N8G-S> .

sakrejda · 2017-09-11T11:22:29Z

Yeah, I can't compile stanc on this thing so I'll try tomorrow on another computer.

sakrejda · 2017-09-13T00:32:29Z

I got rstan installed! Maybe for rstan3 we could reduce the number of dependencies 'cause it's horrid. I'll check this out now.

sakrejda · 2017-09-13T01:11:14Z

Ok, not sure exactly what the solution is since I haven't kept track of the PR we're working from but we're close. The issue is that the function doesn't meet this condition:

Have a return type that is either void or compatible with Rcpp::wrap and parameter
types that are compatible with Rcpp::as (see sections 3.1 and 3.2 of the ‘Rcppintroduction’
vignette for more details).

Here's the function:

template <typename T0__, class RNG>
typename boost::math::tools::promote_args<T0__>::type
f_rng(const T0__& mu, RNG& base_rng__, std::ostream* pstream__) {
    typedef typename boost::math::tools::promote_args<T0__>::type fun_scalar_t__;
    typedef fun_scalar_t__ fun_return_scalar_t__;
    const static bool propto__ = true;
    (void) propto__;
        fun_scalar_t__ DUMMY_VAR__(std::numeric_limits<double>::quiet_NaN());
        (void) DUMMY_VAR__;  // suppress unused var warning

    int current_statement_begin__ = -1;
    try {
        {
        current_statement_begin__ = 4;
        fun_scalar_t__ x;
        (void) x;  // dummy to suppress unused var warning

        stan::math::initialize(x, std::numeric_limits<double>::quiet_NaN());
        stan::math::fill(x,DUMMY_VAR__);
        stan::math::assign(x,normal_rng(mu,1, base_rng__));


        current_statement_begin__ = 5;
        return stan::math::promote_scalar<fun_return_scalar_t__>(x);
        }
    } catch (const std::exception& e) {
        stan::lang::rethrow_located(e, current_statement_begin__, prog_reader__());
        // Next line prevents compiler griping about no return
        throw std::runtime_error("*** IF YOU SEE THIS, PLEASE REPORT A BUG ***");
    }
}

The problem is that there's no Rcpp::as<boost::ecuyer1988>(...) so we could write one that transforms a SEXPR* to a boost::ecuyer1988. Another solution is to replace the references to the RNG's with pointers to RNG's since there is an Rcpp::as<XPtr<whatever>>(...). I confirmed that if I change the RNG references to RNG pointers in the generated .cpp file it compiles fine.

Is one of these preferable? The complexity is about the same and I guess on the Stan side we want references to RNG's rather than pointers so we should probably write the as method. @bob-carpenter am I right that we want these to stay as references in the generated code not pointers?

sakrejda · 2017-09-13T02:14:03Z

OK, a little more progress before I leave this till tomorrow. The generated file compiles as-is if it includes the following code before anything else (including Rcpp.h).

#include <boost/random/additive_combine.hpp>
#include <RcppCommon.h>

namespace Rcpp {
    template <typename T> SEXP wrap(const boost::ecuyer1988& RNG);
    template <> boost::ecuyer1988& as(SEXP ptr_RNG);
}


#include <Rcpp.h>

namespace Rcpp {
    template <typename T> SEXP wrap(const boost::ecuyer1988& RNG){
        return Rcpp::XPtr<boost::ecuyer1988>(RNG);
    }

    template<> boost::ecuyer1988& as(SEXP ptr_RNG) {
        Rcpp::XPtr<boost::ecuyer1988> ptr(ptr_RNG);
        boost::ecuyer1988& RNG = *ptr;
        return RNG;
    }
}

With this code we could create a boost::ecuyer1988 object on the C++ side (or with an R helper function) and guarantee its lifetime from there, then pass it to the R side as a pointer through the Rcpp::wrap method I just defined. Then it would be available in R (as an externalpointer object) s.t. it could be passed back into the standalone function. Where should the import/export code live? I think this is rstan specific so probably just in the src dir of rstan.

sakrejda · 2017-09-13T02:32:13Z

So I can find it later, here's what the C++ of the helper function would look like:


// [[Rcpp::export]]
Rcpp::XPtr<boost::ecuyer1988> get_rng(const int& seed=0) {
  boost::ecuyer1988* base_rng__= new boost::ecuyer1988(seed);
  Rcpp::XPtr<boost::ecuyer1988> ptr(base_rng__,true);
  return ptr;
}

Used like so:

> library(Rcpp)
> sourceCpp('get_rng.cpp')
> rng_ptr <- get_rng(0)
> rng_ptr
<pointer: 0x7fff52175860>
> class(rng_ptr)
[1] "externalptr"

sakrejda · 2017-09-14T17:52:38Z

@bgoodri What do you think about adding the get_rng function to the src directory (and the matching R function to import it). Then adding the as/wrap code to the src directory, and seeing if we can run this by hand. Once we do that I can suggest a package-global rng that can be refreshed with a seed through another R function. If that sounds like a plan to you I don't mind doing it, I just don't want to forge ahead if you have other plans.

bgoodri · 2017-09-14T18:00:19Z

That sounds better than anything I would come up with on my own.

…

On Thu, Sep 14, 2017 at 1:52 PM, Krzysztof Sakrejda < ***@***.***> wrote: @bgoodri <https://github.com/bgoodri> What do you think about adding the get_rng function to the src directory (and the matching R function to import it). Then adding the as/wrap code to the src directory, and seeing if we can run this by hand. Once we do that I can suggest a package-global rng that can be refreshed with a seed through another R function. If that sounds like a plan to you I don't mind doing it, I just don't want to forge ahead if you have other plans. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqsu-sQlucXt2kHu3YwAyuuNqtzp6ks5siWfogaJpZM4N8G-S> .

sakrejda · 2017-09-14T18:04:04Z

Great, I'm working on the makefiles at the moment so realistically I'll make some progress by Monday and then need to troubleshoot till it works...

sakrejda · 2017-09-19T00:17:49Z

I have a rough branch and I've gotten it to work via sourceCpp but the build cycle to debug the rstan build is really long b/c the grammar has to be rebuilt. Should I just update this branch or make a separate one?

bgoodri · 2017-09-19T02:55:47Z

You don't really need a branch. I do R CMD INSTALL --preclean rstan/ to install it once locally and then I can make changes to the R and hit Build and Reload in RStudio without having to rebuild the C++. If you do change the C++, then you can do R CMD INSTALL rstan/ and it will just rebuild the C++ files that have changed.

…

On Mon, Sep 18, 2017 at 8:17 PM, Krzysztof Sakrejda < ***@***.***> wrote: I have a rough branch and I've gotten it to work via sourceCpp but the build cycle to debug the rstan build is really long b/c the grammar has to be rebuilt. Should I just update this branch or make a separate one? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqiI9aZKqMw2wrIo2QG_Wbi1SwIufks5sjwgugaJpZM4N8G-S> .

sakrejda · 2017-09-19T12:00:57Z

Ok, that should speed things up, thanks!

sakrejda · 2017-09-19T12:25:00Z

I got almost all the way there...

There's this thing:

double f_rng(const double& mu, boost::ecuyer1988& base_rng__);
RcppExport SEXP sourceCpp_1_f_rng(SEXP muSEXP, SEXP base_rng__SEXP) {
BEGIN_RCPP
    Rcpp::RObject rcpp_result_gen;
    Rcpp::RNGScope rcpp_rngScope_gen;
    Rcpp::traits::input_parameter< const double& >::type mu(muSEXP);
    Rcpp::traits::input_parameter< boost::ecuyer1988& >::type base_rng__(base_rng__SEXP);
    rcpp_result_gen = Rcpp::wrap(f_rng(mu, base_rng__));
    return rcpp_result_gen;   
END_RCPP
}

That ultimately doesn't compile because Rcpp::traits::input_parameter< boost::ecuyer1988& >::type base_rng__(base_rng__SEXP); produces an SEXP. There's another easy way around this in hand-written code but it's not what sourceCpp does. All we need to do is specialize Rcpp::traits::input_parameter<T> for boost::ecuyer1988& but I'll ask a stackOverflow question for it b/c I'm sure you need to inject that in a way similar to how you inject as/wrap but it would be really time-consuming to work out the details without help.

sakrejda · 2017-09-19T15:11:24Z

I pushed what I have so far to a branch. I still don't have the types quite right in the exporter.h include file.

sakrejda · 2017-09-21T00:07:39Z

@bgoodri; victory!

> RNG <- get_rng()
> f_rng(mu=3, base_rng__=RNG)
[1] 2.047012

... there's still a bug somewhere b/c the RNG doesn't get advanced but at least I got the types straight. We could avoid finding how to not copy the RNG and just initialize it using R's RNG for the seed, or I could just figure out how to avoid the copies, I'm sure that's possible too.

It's on a branch: feature/issue-422-standalone-functions-k

sakrejda · 2017-09-21T00:08:10Z

Also is Stan generating that __create_rng function? That should go away b/c it breaks things.

bgoodri · 2017-09-21T00:12:40Z

Yes, at least with standalone functions. I don't think it gets created when you generate a model.

…

On Wed, Sep 20, 2017 at 8:08 PM, Krzysztof Sakrejda < ***@***.***> wrote: Also is Stan generating that __create_rng function? That should go away b/c it breaks things. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqkkqaSweAsm4i7p6m81Wk2peIlO2ks5skajqgaJpZM4N8G-S> .

sakrejda · 2017-09-21T00:25:36Z

I'll see if there's a change I can make to have it easily get wrapped by Rcpp and it would save us keeping get_rng in rstan.

martinmodrak · 2017-09-21T06:31:08Z

Glad to see you are making progress even without my help. Sorry for any mess in my code.
The __create_rng method is generated by Stan only in the standalone function compilation mode. See lang\generator\generate_rng_rcpp_helper.hpp and \generator\generate_standalone_functions.hpp which calls it. The reason I added it is that I thought that it would be helpful for the RCpp integration, so if it is making problems, just remove it, no other code should dependen on it.

sakrejda · 2017-09-24T17:49:40Z

@bgoodri I softlinked StanHeaders/inst/include/src to a local stan/src directory and now it can't find cvodes.h... I have nearly everything else working, I just need a branch of stan-dev/stan that's ahead of develop (doesn't create the __create_rng function) for testing. Any suggestions about how to do it?

bgoodri · 2017-09-24T17:52:31Z

https://raw.githubusercontent.com/stan-dev/rstan/develop/install_StanHeaders.R Just set the branch arguments to something else.

…

On Sun, Sep 24, 2017 at 1:49 PM, Krzysztof Sakrejda < ***@***.***> wrote: @bgoodri <https://github.com/bgoodri> I softlinked StanHeaders/inst/include/src to a local stan/src directory and now it can't find cvodes.h... I have nearly everything else working, I just need a branch of stan-dev/stan that's ahead of develop (doesn't create the __create_rng function) for testing. Any suggestions about how to do it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqm_VlL36ro0RHTHyiWwBMolkVn-Xks5slpY1gaJpZM4N8G-S> .

sakrejda · 2017-09-24T22:14:20Z

Ok, thanks.

…

On Sun, Sep 24, 2017, 1:52 PM bgoodri ***@***.***> wrote: https://raw.githubusercontent.com/stan-dev/rstan/develop/install_StanHeaders.R Just set the branch arguments to something else. On Sun, Sep 24, 2017 at 1:49 PM, Krzysztof Sakrejda < ***@***.***> wrote: > @bgoodri <https://github.com/bgoodri> I softlinked > StanHeaders/inst/include/src to a local stan/src directory and now it can't > find cvodes.h... I have nearly everything else working, I just need a > branch of stan-dev/stan that's ahead of develop (doesn't create the > __create_rng function) for testing. Any suggestions about how to do it? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#422 (comment)>, or mute > the thread > < https://github.com/notifications/unsubscribe-auth/ADOrqm_VlL36ro0RHTHyiWwBMolkVn-Xks5slpY1gaJpZM4N8G-S > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#422 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAfA6WMmDk9k1PsPjP278_51HoD3jiwyks5slpbggaJpZM4N8G-S> .

sakrejda · 2017-09-25T16:15:02Z

@martinmodrak This is much easier than doing the whole thing! 👍

sakrejda · 2017-11-03T16:30:29Z

@bgoodri can you check out my branch and let me know what else it needs? I don't have time to figure out what rstan standards are for getting code in but I do have time to make changes (one's much harder for me than the other).

Branch: https://github.com/stan-dev/rstan/tree/feature/issue-422-standalone-functions-k

bob-carpenter assigned martinmodrak Jun 16, 2017

bob-carpenter added the feature label Jun 16, 2017

bob-carpenter added this to the 3.0.0 milestone Jun 16, 2017

martinmodrak mentioned this issue Jul 4, 2017

Standalone compilation of functions is not compatible with RStan stan-dev/stan#2346

Closed

This was referenced Sep 15, 2017

Integrate standalone function compilation with PyStan stan-dev/pystan2#372

Closed

Feature request: expose_stan_functions for pystan? stan-dev/pystan2#360

Closed

dmenne mentioned this issue Oct 17, 2017

Multiple issues with expose_stan_functions. #458

Closed

alashworth mentioned this issue Mar 12, 2019

Standalone compilation of functions is not compatible with RStan alashworth/test-issue-import#151

Open

Integrate standalone function compilation with RStan #422

Integrate standalone function compilation with RStan #422

Comments

martinmodrak commented Jun 16, 2017

Summary:

Description:

bgoodri commented Jun 25, 2017

bgoodri commented Jun 25, 2017

bob-carpenter commented Jun 25, 2017

bgoodri commented Jun 25, 2017

martinmodrak commented Jun 26, 2017

sakrejda commented Jun 26, 2017

bob-carpenter commented Jun 26, 2017 • edited Loading

martinmodrak commented Jul 4, 2017

bgoodri commented Sep 9, 2017

bgoodri commented Sep 10, 2017

bgoodri commented Sep 10, 2017

sakrejda commented Sep 11, 2017

sakrejda commented Sep 11, 2017

bgoodri commented Sep 11, 2017 via email

sakrejda commented Sep 11, 2017

sakrejda commented Sep 13, 2017

sakrejda commented Sep 13, 2017

sakrejda commented Sep 13, 2017

sakrejda commented Sep 13, 2017

sakrejda commented Sep 14, 2017

bgoodri commented Sep 14, 2017 via email

sakrejda commented Sep 14, 2017

sakrejda commented Sep 19, 2017

bgoodri commented Sep 19, 2017 via email

sakrejda commented Sep 19, 2017

sakrejda commented Sep 19, 2017

sakrejda commented Sep 19, 2017

sakrejda commented Sep 21, 2017

sakrejda commented Sep 21, 2017

bgoodri commented Sep 21, 2017 via email

sakrejda commented Sep 21, 2017

martinmodrak commented Sep 21, 2017 • edited Loading

sakrejda commented Sep 24, 2017

bgoodri commented Sep 24, 2017 via email

sakrejda commented Sep 24, 2017 via email

sakrejda commented Sep 25, 2017

sakrejda commented Nov 3, 2017

bob-carpenter commented Jun 26, 2017 •

edited

Loading

martinmodrak commented Sep 21, 2017 •

edited

Loading