Feature/issue 2682 hcubature #2838

Franzi2114 · 2022-10-25T20:38:23Z

Summary

Here, we add the multidimensional integration routine hcubature. We need this feature for issue #2682. This is a split from PR #2822.

We add the following files:
stan/math/prim/functor/hcubature.hpp

Tests

We add the following files:
test/unit/math/prim/functor/hcubature_test.cpp

Side Effects

No.

Release notes

New multidimensional integration routine.

Checklist

Math issue Adding 7-Parameter Drift Diffusion Model (DDM) PDF with partial derivatives to Stan Math #2682
Copyright holder:
Franziska Henrich (franziska.henrich@psychologie.uni-freiburg.de)
Valentin Pratz (pratz@stud.uni-heidelberg.de)
Christoph Klauer (christoph.klauer@psychologie.uni-freiburg.de)
Raphael Hartmann (raphael.hartmann@staff.uni-marburg.de)

The copyright holder is typically you or your assignee, such as a university or company. By submitting this pull request, the copyright holder is agreeing to the license the submitted work under the following licenses:
- Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
- Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
the basic tests are passing
- unit tests pass (to run, use: ./runTests.py test/unit)
- header checks pass, (make test-headers)
- dependencies checks pass, (make test-math-dependencies)
- docs build, (make doxygen)
- code passes the built in C++ standards checks (make cpplint)
the code is written in idiomatic C++ and changes are documented in the doxygen
the new changes are tested

andrjohns · 2022-10-26T13:36:50Z

@nhuurre Would you be free to look at this PR? Multi-dimensional (h-adaptive) integration isn't exactly a strong suit of mine!

The code is based on the Julia implementation

nhuurre

It's not like I'm an expert on numeric integration either! Or C++. But here's some comments anyway.

stan/math/prim/functor/hcubature.hpp

SteveBronder

Not sure if I have time to be the main reviewer but a few small comments

stan/math/prim/functor/hcubature.hpp

test/unit/math/prim/functor/hcubature_test.cpp

stan/math/prim/functor/hcubature.hpp

…ox, alloc, clean_GenzMalik, make_box

…FranziStan into feature/issue-2682-hcubature pull changes priority_queue.

andrjohns · 2023-01-04T11:51:26Z

@rok-cesnovar and @serban-nicusor-toptal (apologies for tagging both, not sure who is best here), I can't replicate the OpenCL failure (with test/unit/math/opencl/mrrr_test) seen in develop and in this PR.

I've tested with the stanorg/ci:gpu-cpp17 image and the Jenkinsfile make/local changes:

CXX=clang++-7 -Werror
STAN_OPENCL=true

I've tested using an Intel Integrated GPU, as well as three Nvidia GPUs (V100, A100, P100).

Any suggestions on the next steps for trying to replicate? Are either of you able to access the CI machine itself to run the test manually and see if it replicates outside of the Jenkins run?

serban-nicusor-toptal · 2023-01-04T15:51:03Z

Hey @andrjohns I do have non-root access to one of the Linux instances.
If you can guide me in what I need to execute on the server to help sort this out that would be great, I can do it on the server and post the outputs here.

andrjohns · 2023-01-05T11:02:20Z

Hey @andrjohns I do have non-root access to one of the Linux instances. If you can guide me in what I need to execute on the server to help sort this out that would be great, I can do it on the server and post the outputs here.

Brilliant, thanks! Can you run the following using the develop branch:

echo STAN_OPENCL=true >> make/local
python3 runTests.py -j4 test/unit/math/opencl/mrrr_test

And let me know if it passes or fails?

serban-nicusor-toptal · 2023-01-05T12:38:26Z

Uh, the host OS does not have clang++-7 or gcc installed as it runs docker containers on top. I will have to write a message to the administrator of the servers and ask if we can get root to install these or what's the best way to tackle this from his point of view. Is there any information I can provide in the meanwhile to help with this ? Like os, gpu drivers versions and model
or anything else that might help pinpoint the problem?

andrjohns · 2023-01-05T14:32:11Z

Oh of course, didn't think of that! Can you ty running it in the docker container (the gpu-cpp17 one)?

serban-nicusor-toptal · 2023-01-05T14:51:58Z

Full log

------------------------------------------------------------
make -j4 test/unit/math/opencl/mrrr_test
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I lib/opencl_3.0.0   -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.78.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest     -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DINTEGRATED_OPENCL=0       -DGTEST_HAS_PTHREAD=0  -c -MT test/unit/math/opencl/mrrr_test.o -MM -E -MG -MP -MF test/unit/math/opencl/mrrr_test.d test/unit/math/opencl/mrrr_test.cpp
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/nvector/serial/nvector_serial.c -o lib/sundials_6.1.1/src/nvector/serial/nvector_serial.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_math.c -o lib/sundials_6.1.1/src/sundials/sundials_math.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_diag.c -o lib/sundials_6.1.1/src/cvodes/cvodes_diag.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_spils.c -o lib/sundials_6.1.1/src/cvodes/cvodes_spils.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodea.c -o lib/sundials_6.1.1/src/cvodes/cvodea.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg.c -o lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_direct.c -o lib/sundials_6.1.1/src/cvodes/cvodes_direct.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodea_io.c -o lib/sundials_6.1.1/src/cvodes/cvodea_io.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_ls.c -o lib/sundials_6.1.1/src/cvodes/cvodes_ls.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_bbdpre.c -o lib/sundials_6.1.1/src/cvodes/cvodes_bbdpre.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_bandpre.c -o lib/sundials_6.1.1/src/cvodes/cvodes_bandpre.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_io.c -o lib/sundials_6.1.1/src/cvodes/cvodes_io.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes.c -o lib/sundials_6.1.1/src/cvodes/cvodes.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_nls.c -o lib/sundials_6.1.1/src/cvodes/cvodes_nls.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_nls_sim.c -o lib/sundials_6.1.1/src/cvodes/cvodes_nls_sim.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg1.c -o lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg1.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_matrix.c -o lib/sundials_6.1.1/src/sundials/sundials_matrix.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_futils.c -o lib/sundials_6.1.1/src/sundials/sundials_futils.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_direct.c -o lib/sundials_6.1.1/src/sundials/sundials_direct.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_nonlinearsolver.c -o lib/sundials_6.1.1/src/sundials/sundials_nonlinearsolver.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_version.c -o lib/sundials_6.1.1/src/sundials/sundials_version.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_band.c -o lib/sundials_6.1.1/src/sundials/sundials_band.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_iterative.c -o lib/sundials_6.1.1/src/sundials/sundials_iterative.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_linearsolver.c -o lib/sundials_6.1.1/src/sundials/sundials_linearsolver.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_memory.c -o lib/sundials_6.1.1/src/sundials/sundials_memory.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_context.c -o lib/sundials_6.1.1/src/sundials/sundials_context.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_nvector.c -o lib/sundials_6.1.1/src/sundials/sundials_nvector.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_nvector_senswrapper.c -o lib/sundials_6.1.1/src/sundials/sundials_nvector_senswrapper.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sundials/sundials_dense.c -o lib/sundials_6.1.1/src/sundials/sundials_dense.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunmatrix/band/sunmatrix_band.c -o lib/sundials_6.1.1/src/sunmatrix/band/sunmatrix_band.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunmatrix/dense/sunmatrix_dense.c -o lib/sundials_6.1.1/src/sunmatrix/dense/sunmatrix_dense.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunlinsol/band/sunlinsol_band.c -o lib/sundials_6.1.1/src/sunlinsol/band/sunlinsol_band.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunlinsol/dense/sunlinsol_dense.c -o lib/sundials_6.1.1/src/sunlinsol/dense/sunlinsol_dense.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunnonlinsol/newton/sunnonlinsol_newton.c -o lib/sundials_6.1.1/src/sunnonlinsol/newton/sunnonlinsol_newton.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/sunnonlinsol/fixedpoint/sunnonlinsol_fixedpoint.c -o lib/sundials_6.1.1/src/sunnonlinsol/fixedpoint/sunnonlinsol_fixedpoint.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_spils.c -o lib/sundials_6.1.1/src/idas/idas_spils.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_ic.c -o lib/sundials_6.1.1/src/idas/idas_ic.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idaa_io.c -o lib/sundials_6.1.1/src/idas/idaa_io.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_io.c -o lib/sundials_6.1.1/src/idas/idas_io.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_direct.c -o lib/sundials_6.1.1/src/idas/idas_direct.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_nls_stg.c -o lib/sundials_6.1.1/src/idas/idas_nls_stg.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_bbdpre.c -o lib/sundials_6.1.1/src/idas/idas_bbdpre.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas.c -o lib/sundials_6.1.1/src/idas/idas.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idaa.c -o lib/sundials_6.1.1/src/idas/idaa.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_ls.c -o lib/sundials_6.1.1/src/idas/idas_ls.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_nls_sim.c -o lib/sundials_6.1.1/src/idas/idas_nls_sim.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/idas/idas_nls.c -o lib/sundials_6.1.1/src/idas/idas_nls.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol_direct.c -o lib/sundials_6.1.1/src/kinsol/kinsol_direct.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol_bbdpre.c -o lib/sundials_6.1.1/src/kinsol/kinsol_bbdpre.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol_io.c -o lib/sundials_6.1.1/src/kinsol/kinsol_io.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol_ls.c -o lib/sundials_6.1.1/src/kinsol/kinsol_ls.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol_spils.c -o lib/sundials_6.1.1/src/kinsol/kinsol_spils.o
g++ -pipe   -pthread -D_REENTRANT  -O3 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -DNO_FPRINTF_OUTPUT     -O3  -c -x c -include lib/sundials_6.1.1/include/stan_sundials_printf_override.hpp lib/sundials_6.1.1/src/kinsol/kinsol.c -o lib/sundials_6.1.1/src/kinsol/kinsol.o
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I lib/opencl_3.0.0   -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.78.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest      -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DINTEGRATED_OPENCL=0       -DGTEST_HAS_PTHREAD=0 -DGTEST_HAS_PTHREAD=0  -c -o lib/benchmark_1.5.1/googletest/googletest/src/gtest-all.o lib/benchmark_1.5.1/googletest/googletest/src/gtest-all.cc
touch lib/tbb/tbb-make-check
ar -rs lib/sundials_6.1.1/lib/libsundials_nvecserial.a lib/sundials_6.1.1/src/nvector/serial/nvector_serial.o lib/sundials_6.1.1/src/sundials/sundials_math.o
ar -rs lib/sundials_6.1.1/lib/libsundials_cvodes.a lib/sundials_6.1.1/src/cvodes/cvodes_spils.o lib/sundials_6.1.1/src/cvodes/cvodes_diag.o lib/sundials_6.1.1/src/cvodes/cvodea.o lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg.o lib/sundials_6.1.1/src/cvodes/cvodes_direct.o lib/sundials_6.1.1/src/cvodes/cvodea_io.o lib/sundials_6.1.1/src/cvodes/cvodes_ls.o lib/sundials_6.1.1/src/cvodes/cvodes_bbdpre.o lib/sundials_6.1.1/src/cvodes/cvodes_bandpre.o lib/sundials_6.1.1/src/cvodes/cvodes_io.o lib/sundials_6.1.1/src/cvodes/cvodes.o lib/sundials_6.1.1/src/cvodes/cvodes_nls.o lib/sundials_6.1.1/src/cvodes/cvodes_nls_sim.o lib/sundials_6.1.1/src/cvodes/cvodes_nls_stg1.o lib/sundials_6.1.1/src/sundials/sundials_matrix.o lib/sundials_6.1.1/src/sundials/sundials_futils.o lib/sundials_6.1.1/src/sundials/sundials_direct.o lib/sundials_6.1.1/src/sundials/sundials_nonlinearsolver.o lib/sundials_6.1.1/src/sundials/sundials_version.o lib/sundials_6.1.1/src/sundials/sundials_band.o lib/sundials_6.1.1/src/sundials/sundials_iterative.o lib/sundials_6.1.1/src/sundials/sundials_linearsolver.o lib/sundials_6.1.1/src/sundials/sundials_math.o lib/sundials_6.1.1/src/sundials/sundials_memory.o lib/sundials_6.1.1/src/sundials/sundials_context.o lib/sundials_6.1.1/src/sundials/sundials_nvector.o lib/sundials_6.1.1/src/sundials/sundials_nvector_senswrapper.o lib/sundials_6.1.1/src/sundials/sundials_dense.o lib/sundials_6.1.1/src/sunmatrix/band/sunmatrix_band.o lib/sundials_6.1.1/src/sunmatrix/dense/sunmatrix_dense.o lib/sundials_6.1.1/src/sunlinsol/band/sunlinsol_band.o lib/sundials_6.1.1/src/sunlinsol/dense/sunlinsol_dense.o lib/sundials_6.1.1/src/sunnonlinsol/newton/sunnonlinsol_newton.o lib/sundials_6.1.1/src/sunnonlinsol/fixedpoint/sunnonlinsol_fixedpoint.o
tbb_root="../tbb_2020.3" CXX="g++" CC="gcc" LDFLAGS='-Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"  ' 'make' -C "lib/tbb" -r -f "/tmp/math/lib/tbb_2020.3/build/Makefile.tbbmalloc" compiler=gcc cfg=release stdver=c++1y malloc CXXFLAGS="-Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation  "
make[1]: Entering directory '/tmp/math/lib/tbb'
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/backend.cpp
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/large_objects.cpp
ar -rs lib/sundials_6.1.1/lib/libsundials_idas.a lib/sundials_6.1.1/src/idas/idas_spils.o lib/sundials_6.1.1/src/idas/idas_ic.o lib/sundials_6.1.1/src/idas/idaa_io.o lib/sundials_6.1.1/src/idas/idas_io.o lib/sundials_6.1.1/src/idas/idas_direct.o lib/sundials_6.1.1/src/idas/idas_nls_stg.o lib/sundials_6.1.1/src/idas/idas_bbdpre.o lib/sundials_6.1.1/src/idas/idas.o lib/sundials_6.1.1/src/idas/idaa.o lib/sundials_6.1.1/src/idas/idas_ls.o lib/sundials_6.1.1/src/idas/idas_nls_sim.o lib/sundials_6.1.1/src/idas/idas_nls.o lib/sundials_6.1.1/src/sundials/sundials_matrix.o lib/sundials_6.1.1/src/sundials/sundials_futils.o lib/sundials_6.1.1/src/sundials/sundials_direct.o lib/sundials_6.1.1/src/sundials/sundials_nonlinearsolver.o lib/sundials_6.1.1/src/sundials/sundials_version.o lib/sundials_6.1.1/src/sundials/sundials_band.o lib/sundials_6.1.1/src/sundials/sundials_iterative.o lib/sundials_6.1.1/src/sundials/sundials_linearsolver.o lib/sundials_6.1.1/src/sundials/sundials_math.o lib/sundials_6.1.1/src/sundials/sundials_memory.o lib/sundials_6.1.1/src/sundials/sundials_context.o lib/sundials_6.1.1/src/sundials/sundials_nvector.o lib/sundials_6.1.1/src/sundials/sundials_nvector_senswrapper.o lib/sundials_6.1.1/src/sundials/sundials_dense.o lib/sundials_6.1.1/src/sunmatrix/band/sunmatrix_band.o lib/sundials_6.1.1/src/sunmatrix/dense/sunmatrix_dense.o lib/sundials_6.1.1/src/sunlinsol/band/sunlinsol_band.o lib/sundials_6.1.1/src/sunlinsol/dense/sunlinsol_dense.o lib/sundials_6.1.1/src/sunnonlinsol/newton/sunnonlinsol_newton.o lib/sundials_6.1.1/src/sunnonlinsol/fixedpoint/sunnonlinsol_fixedpoint.o
ar -rs lib/sundials_6.1.1/lib/libsundials_kinsol.a lib/sundials_6.1.1/src/kinsol/kinsol_direct.o lib/sundials_6.1.1/src/kinsol/kinsol_bbdpre.o lib/sundials_6.1.1/src/kinsol/kinsol_io.o lib/sundials_6.1.1/src/kinsol/kinsol_ls.o lib/sundials_6.1.1/src/kinsol/kinsol_spils.o lib/sundials_6.1.1/src/kinsol/kinsol.o lib/sundials_6.1.1/src/sundials/sundials_matrix.o lib/sundials_6.1.1/src/sundials/sundials_futils.o lib/sundials_6.1.1/src/sundials/sundials_direct.o lib/sundials_6.1.1/src/sundials/sundials_nonlinearsolver.o lib/sundials_6.1.1/src/sundials/sundials_version.o lib/sundials_6.1.1/src/sundials/sundials_band.o lib/sundials_6.1.1/src/sundials/sundials_iterative.o lib/sundials_6.1.1/src/sundials/sundials_linearsolver.o lib/sundials_6.1.1/src/sundials/sundials_math.o lib/sundials_6.1.1/src/sundials/sundials_memory.o lib/sundials_6.1.1/src/sundials/sundials_context.o lib/sundials_6.1.1/src/sundials/sundials_nvector.o lib/sundials_6.1.1/src/sundials/sundials_nvector_senswrapper.o lib/sundials_6.1.1/src/sundials/sundials_dense.o lib/sundials_6.1.1/src/sunmatrix/band/sunmatrix_band.o lib/sundials_6.1.1/src/sunmatrix/dense/sunmatrix_dense.o lib/sundials_6.1.1/src/sunlinsol/band/sunlinsol_band.o lib/sundials_6.1.1/src/sunlinsol/dense/sunlinsol_dense.o lib/sundials_6.1.1/src/sunnonlinsol/newton/sunnonlinsol_newton.o lib/sundials_6.1.1/src/sunnonlinsol/fixedpoint/sunnonlinsol_fixedpoint.o
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/backref.cpp
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/tbbmalloc.cpp
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -o itt_notify_malloc.o -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/itt_notify.cpp
sh ../tbb_2020.3/build/version_info_linux.sh g++ -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -Wall -Wextra -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1  -std=c++1y -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include >version_string.ver
echo "INPUT (libtbbmalloc.so.2)" > libtbbmalloc.so
g++ -E -x c++ ../tbb_2020.3/src/tbbmalloc/lin64-tbbmalloc-export.def -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -Wall -Wextra -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1  -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1  -Wno-parentheses -Wno-sized-deallocation -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include > tbbmalloc.def
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -Wall -Wextra -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1  -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -D__TBBMALLOC_BUILD=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/proxy.cpp
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -Wall -Wextra -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1  -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -D__TBBMALLOC_BUILD=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc ../tbb_2020.3/src/tbbmalloc/tbb_function_replacement.cpp
echo "INPUT (libtbbmalloc_proxy.so.2)" > libtbbmalloc_proxy.so
g++ -E -x c++ ../tbb_2020.3/src/tbbmalloc/lin64-proxy-export.def -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -Wall -Wextra -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1  -Wno-parentheses -Wno-sized-deallocation -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include > tbbmallocproxy.def
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I lib/opencl_3.0.0   -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.78.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest      -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DINTEGRATED_OPENCL=0       -DGTEST_HAS_PTHREAD=0 -DGTEST_HAS_PTHREAD=0  -c -o test/unit/math/opencl/mrrr_test.o test/unit/math/opencl/mrrr_test.cpp
g++ -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm   -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -fno-rtti -fno-exceptions -D__TBBMALLOC_BUILD=1 -Wno-parentheses -Wno-sized-deallocation -fPIC -flifetime-dse=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I../tbb_2020.3/src/tbbmalloc -I../tbb_2020.3/src/tbbmalloc -I. ../tbb_2020.3/src/tbbmalloc/frontend.cpp
gcc -fPIC -o libtbbmalloc.so.2 backend.o large_objects.o backref.o  tbbmalloc.o  itt_notify_malloc.o frontend.o  -ldl -lrt -shared -Wl,-soname=libtbbmalloc.so.2 -pthread -m64 -Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"   -Wl,--version-script,tbbmalloc.def
g++ -fPIC -o libtbbmalloc_proxy.so.2 proxy.o tbb_function_replacement.o  -ldl -lrt libtbbmalloc.so -shared -Wl,-soname=libtbbmalloc_proxy.so.2 -pthread -m64 -Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"   -Wl,--version-script,tbbmallocproxy.def
make[1]: Leaving directory '/tmp/math/lib/tbb'
touch lib/tbb/version_tbb_2020.3
tbb_root="../tbb_2020.3" CXX="g++" CC="gcc" LDFLAGS='-Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"  ' 'make' -C "lib/tbb" -r -f "/tmp/math/lib/tbb_2020.3/build/Makefile.tbb" compiler=gcc cfg=release stdver=c++1y  CXXFLAGS="-Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation  "
make[1]: Entering directory '/tmp/math/lib/tbb'
g++ -o concurrent_hash_map.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/concurrent_hash_map.cpp
g++ -o concurrent_queue.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/concurrent_queue.cpp
g++ -o concurrent_vector.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/concurrent_vector.cpp
g++ -o dynamic_link.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/dynamic_link.cpp
g++ -o itt_notify.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/itt_notify.cpp
g++ -o cache_aligned_allocator.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/cache_aligned_allocator.cpp
g++ -o pipeline.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/pipeline.cpp
g++ -o queuing_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/queuing_mutex.cpp
g++ -o queuing_rw_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/queuing_rw_mutex.cpp
g++ -o reader_writer_lock.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/reader_writer_lock.cpp
g++ -o spin_rw_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/spin_rw_mutex.cpp
g++ -o x86_rtm_rw_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/x86_rtm_rw_mutex.cpp
g++ -o spin_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/spin_mutex.cpp
g++ -o critical_section.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/critical_section.cpp
g++ -o mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/mutex.cpp
g++ -o recursive_mutex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/recursive_mutex.cpp
g++ -o condition_variable.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/condition_variable.cpp
g++ -o tbb_thread.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/tbb_thread.cpp
g++ -o concurrent_monitor.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/concurrent_monitor.cpp
g++ -o semaphore.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/semaphore.cpp
g++ -o private_server.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/private_server.cpp
g++ -o rml_tbb.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/rml/client/rml_tbb.cpp
g++ -o tbb_misc.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include -I. ../tbb_2020.3/src/tbb/tbb_misc.cpp
g++ -o tbb_misc_ex.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/tbb_misc_ex.cpp
g++ -o task.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/task.cpp
g++ -o task_group_context.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/task_group_context.cpp
g++ -o governor.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/governor.cpp
g++ -o market.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/market.cpp
g++ -o arena.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/arena.cpp
g++ -o scheduler.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/scheduler.cpp
g++ -o observer_proxy.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/observer_proxy.cpp
g++ -o tbb_statistics.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/tbb_statistics.cpp
g++ -o tbb_main.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/tbb/tbb_main.cpp
g++ -o concurrent_vector_v2.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/old/concurrent_vector_v2.cpp
g++ -o concurrent_queue_v2.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/old/concurrent_queue_v2.cpp
g++ -o spin_rw_mutex_v2.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/old/spin_rw_mutex_v2.cpp
g++ -o task_v2.o -c -MMD -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -std=c++1y  -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include ../tbb_2020.3/src/old/task_v2.cpp
sh ../tbb_2020.3/build/generate_tbbvars.sh
echo "INPUT (libtbb.so.2)" > libtbb.so
g++ -E -x c++ ../tbb_2020.3/src/tbb/lin64-tbb-export.def -O2 -g -DDO_ITT_NOTIFY -DUSE_PTHREAD -pthread -m64 -mrtm  -fPIC -flifetime-dse=1 -D__TBB_BUILD=1 -Wall -Wextra -Wno-parentheses -Wno-sized-deallocation -Wno-unknown-warning-option -Wno-deprecated-copy -Wno-missing-attributes -Wno-class-memaccess -Wno-sized-deallocation   -DTBB_SUPPRESS_DEPRECATED_MESSAGES=1 -I../tbb_2020.3/src -I../tbb_2020.3/src/rml/include -I../tbb_2020.3/include > tbb.def
g++ -fPIC -o libtbb.so.2 concurrent_hash_map.o concurrent_queue.o concurrent_vector.o dynamic_link.o itt_notify.o cache_aligned_allocator.o pipeline.o queuing_mutex.o queuing_rw_mutex.o reader_writer_lock.o spin_rw_mutex.o x86_rtm_rw_mutex.o spin_mutex.o critical_section.o mutex.o recursive_mutex.o condition_variable.o tbb_thread.o concurrent_monitor.o semaphore.o private_server.o rml_tbb.o tbb_misc.o tbb_misc_ex.o task.o task_group_context.o governor.o market.o arena.o scheduler.o observer_proxy.o tbb_statistics.o tbb_main.o concurrent_vector_v2.o concurrent_queue_v2.o spin_rw_mutex_v2.o task_v2.o   -ldl -lrt -shared -Wl,-soname=libtbb.so.2 -pthread -m64 -Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"   -Wl,--version-script,tbb.def
make[1]: Leaving directory '/tmp/math/lib/tbb'
g++ -std=c++1y -pthread -D_REENTRANT -Wno-sign-compare -Wno-ignored-attributes   -I lib/opencl_3.0.0   -I lib/tbb_2020.3/include    -O3  -I . -I lib/eigen_3.3.9 -I lib/boost_1.78.0 -I lib/sundials_6.1.1/include -I lib/sundials_6.1.1/src/sundials -I lib/benchmark_1.5.1/googletest/googletest/include -I lib/benchmark_1.5.1/googletest/googletest     -DBOOST_DISABLE_ASSERTS  -DSTAN_OPENCL -DOPENCL_DEVICE_ID=0 -DOPENCL_PLATFORM_ID=0 -DCL_HPP_TARGET_OPENCL_VERSION=120 -DCL_HPP_MINIMUM_OPENCL_VERSION=120 -DCL_HPP_ENABLE_EXCEPTIONS -Wno-ignored-attributes -DINTEGRATED_OPENCL=0       -DGTEST_HAS_PTHREAD=0        -Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"      test/unit/math/opencl/mrrr_test.o lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc lib/benchmark_1.5.1/googletest/googletest/src/gtest-all.o lib/tbb/libtbb.so.2      -lOpenCL  -Wl,-L,"/tmp/math/lib/tbb" -Wl,-rpath,"/tmp/math/lib/tbb"   -o test/unit/math/opencl/mrrr_test
------------------------------------------------------------
test/unit/math/opencl/mrrr_test --gtest_output="xml:test/unit/math/opencl/mrrr_test.xml"
Running main() from lib/benchmark_1.5.1/googletest/googletest/src/gtest_main.cc
[==========] Running 5 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 5 tests from MathMatrix
[ RUN      ] MathMatrix.tridiag_eigensolver_trivial
[       OK ] MathMatrix.tridiag_eigensolver_trivial (2855 ms)
[ RUN      ] MathMatrix.tridiag_eigensolver_small
[       OK ] MathMatrix.tridiag_eigensolver_small (492 ms)
[ RUN      ] MathMatrix.tridiag_eigensolver_large
[       OK ] MathMatrix.tridiag_eigensolver_large (14174 ms)
[ RUN      ] MathMatrix.tridiag_eigensolver_large_fixed
[       OK ] MathMatrix.tridiag_eigensolver_large_fixed (7136 ms)
[ RUN      ] MathMatrix.tridiag_eigensolver_large_wilkinson
[       OK ] MathMatrix.tridiag_eigensolver_large_wilkinson (1696 ms)
[----------] 5 tests from MathMatrix (26353 ms total)

[----------] Global test environment tear-down
[==========] 5 tests from 1 test suite ran. (26353 ms total)
[  PASSED  ] 5 tests.

Steps I went through:

docker run -it --entrypoint /bin/bash stanorg/ci:gpu-cpp17
git clone https://github.com/stan-dev/math.git
cd math
echo STAN_OPENCL=true >> make/local
python3 runTests.py -j4 test/unit/math/opencl/mrrr_test > log.txt

Output not captured during piping to file:

ar: creating lib/sundials_6.1.1/lib/libsundials_nvecserial.a
ar: creating lib/sundials_6.1.1/lib/libsundials_cvodes.a
ar: creating lib/sundials_6.1.1/lib/libsundials_idas.a
ar: creating lib/sundials_6.1.1/lib/libsundials_kinsol.a
/tmp/math/lib/tbb_2020.3/build/Makefile.tbb:28: CONFIG: cfg=release arch=intel64 compiler=gcc target=linux runtime=cc9.4.0_libc2.31_kernel3.10.0

Note: This ran on jenkins node, I do not have a login into jenkins2, keep this in mind when running the tests maybe there's something on just one of the hosts. You can play around with their tags in the Jenkinsfile when testing this.
To use jenkins2 host you would need agent { label 'linux && v100' } and for jenkins host agent { label 'linux && k40' }

PS: The OpenCL GPU Tests are running on the jenkins2 based on

docker {
    image 'stanorg/ci:gpu-cpp17'
    label 'v100'
    args '--gpus 1'
}

PS: For the sake of testing I created a job that runs in parallel on both hosts.

Jenkinsfile used

pipeline {
    agent none
    options {
        /* Preserve stashes from completed builds, for use with stage restarting. */
        preserveStashes(buildCount: 3)
        /* Persist artifacts and console output for the specific number of recent Pipeline runs */
        buildDiscarder(logRotator(numToKeepStr: '10', artifactNumToKeepStr: '10'))
        /* Will skip default checkout since we're already doing it */
        skipDefaultCheckout()
    }
    environment {
        STAN_NUM_THREADS = 4
        CLANG_CXX = 'clang++-7'
        GCC = 'g++'
        MPICXX = 'mpicxx.openmpi'
        N_TESTS = 100
        OPENCL_DEVICE_ID = 0
        OPENCL_DEVICE_ID_CPU = 0
        OPENCL_DEVICE_ID_GPU = 0
        OPENCL_PLATFORM_ID = 1
        OPENCL_PLATFORM_ID_CPU = 0
        OPENCL_PLATFORM_ID_GPU = 0
        PARALLEL = 4
    }
    stages {
        
        stage('Parallel Stage') {
            parallel {
            stage('OpenCL GPU tests on linux2') {
            agent {
                docker {
                    image 'stanorg/ci:gpu-cpp17'
                    label 'v100'
                    args '--gpus 1'
                }
            }
            steps {
                script {
                    sh "git clone https://github.com/stan-dev/math.git || true"
                    sh """
                        cd math
                        echo CXX=${CLANG_CXX} -Werror > make/local
                        echo STAN_OPENCL=true >> make/local
                        echo OPENCL_PLATFORM_ID=${OPENCL_PLATFORM_ID_GPU} >> make/local
                        echo OPENCL_DEVICE_ID=${OPENCL_DEVICE_ID_GPU} >> make/local
                        python3 runTests.py -j4 test/unit/math/opencl/mrrr_test
                    """
                }
            }
        }
        
        stage('OpenCL GPU tests on linux') {
            agent {
                docker {
                    image 'stanorg/ci:gpu-cpp17'
                    label 'k40'
                    args '--gpus 1'
                }
            }
            steps {
                script {
                    sh "git clone https://github.com/stan-dev/math.git || true"
                    sh """
                        cd math
                        echo CXX=${CLANG_CXX} -Werror > make/local
                        echo STAN_OPENCL=true >> make/local
                        echo OPENCL_PLATFORM_ID=${OPENCL_PLATFORM_ID_GPU} >> make/local
                        echo OPENCL_DEVICE_ID=${OPENCL_DEVICE_ID_GPU} >> make/local
                        python3 runTests.py -j4 test/unit/math/opencl/mrrr_test
                    """
                }
            }
        }
            }
        }


    }
}

If we take a look at the results of the job, we can see that the same exact steps are failing on jenkins2 ( the one configured to run our OpenCL GPU Tests ) but they are running fine on jenkins. A quick and simple solution to fix this and get everything back up would be to simply use the other host if that could work for us.
We tested this before and it ran fine, not sure what could have changed in the meanwhile on jenkins2.

Please see below the GPU information on jenkins host

Thu Jan  5 10:20:55 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K40m          Off  | XXX Off |                    0 |
| N/A   26C    P0    60W / 235W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K40c          Off  | XXX Off |                    0 |
| 23%   33C    P0    70W / 235W |      0MiB / 11441MiB |     87%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And for the jenkins2 host

Thu Jan  5 10:24:16 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.11    Driver Version: 525.60.11    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | XXX Off |                    0 |
| N/A   33C    P0    38W / 250W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-PCIE...  Off  | XXX Off |                    0 |
| N/A   33C    P0    38W / 250W |      0MiB / 16384MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+                                                                            
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

andrjohns · 2023-01-06T14:11:22Z

Thanks for the thorough work @serban-nicusor-toptal, this is great! I'll update the Jenkinsfile in this PR to run on the jenkins host, so that it's not a blocker for the PR and then open a separate issue for investigating this

andrjohns · 2023-01-06T15:22:57Z

@serban-nicusor-toptal I tried setting the constraints that you mentioned (changes here), but it still failed in the same way. Would you mind double-checking whether I've missed a constraint/label?

serban-nicusor-toptal · 2023-01-06T15:41:06Z

Hey @andrjohns I think this is because ( can confirm it is, the code that ran was still with v100 )

Pull requests from forks will all be treated as untrusted. 
This means that where Jenkins requires a trusted file (e.g. Jenkinsfile) the contents of that file will be retrieved from the target branch on the origin repository and not from the pull request branch on the fork repository.

This is a security feature in Jenkins to ensure no malicious Jenkinsfile code can be injected.
I re-ran this pipeline on the new instance so it can pass the review and get unstuck, the best way forward would be to patch develop to use the other host and then merge that into active PRs to fix this error. ( where needed, forks will pull develop by default )
I don't think it would make a big difference for us if we just change the host, @rok-cesnovar what do you think ?

serban-nicusor-toptal · 2023-01-09T17:08:23Z

@andrjohns it seems to have failed in Threading Tests now, does that have anything to do with the machine error we encountered in the OpenCL tests ?

Franzi2114 · 2023-01-16T08:29:39Z

Hey @andrjohns, it seems you two solved the continuous integration test problem! Is this PR now ready for merging or is there something else to do for me?

andrjohns · 2023-01-17T06:14:18Z

Hey @andrjohns, it seems you two solved the continuous integration test problem! Is this PR now ready for merging or is there something else to do for me?

Perfect! I'll just revert the Jenkinsfile change that I made, since that shouldn't be needed. I'll approve the PR and then you can merge when the tests all pass

andrjohns

Ready to merge when tests pass. Thanks for sticking through the long journey!

Franzi2114 · 2023-01-17T07:39:10Z

when tests pass

Hm .... the mrrr_test seems again to have failed ...

andrjohns · 2023-01-17T07:42:48Z

when tests pass

Hm .... the mrrr_test seems again to have failed ...

@serban-nicusor-toptal did we have a fix for changing the Jenkins host for the OpenCL tests?

serban-nicusor-toptal · 2023-01-19T13:55:59Z

Hey @andrjohns I just merged the fix into develop, feel free to merge develop into this branch and it should be fine.

Franzi2114 · 2023-01-19T16:40:10Z

merged the fix into develop

Hey @andrjohns, @serban-nicusor-toptal, now, the mrrr Test seems to be successful, great!
But another error occurs: the symmetric_eigensolver_test in opencl fails. Has this something to do with my program? I can't find any hint to hcubature.

andrjohns · 2023-01-27T04:59:25Z

@Franzi2114 Can you merge in the latest develop again? It should pass CI now, and then I'll merge this (finally!)

andrjohns · 2023-01-27T13:42:48Z

Made it! Sorry for the delay here @Franzi2114!

Franzi2114 · 2023-01-27T15:16:26Z

Made it! Sorry for the delay here @Franzi2114!

Great!! Thanks a lot :) This was a long journey ^^

Then I will merge the develop branch into the 7-parameter-ddm-pdf branch so that the third part of that pr can go on.

Franzi2114 added 5 commits October 24, 2022 13:41

hcubature added

10c4144

hcubature tests added

b9e2179

added 1d example

342d36b

clang

2302cd8

Tests and doku for hcubature finished

d03f7bf

Franzi2114 mentioned this pull request Oct 25, 2022

Feature/issue 2682 add 7 parameter ddm pdf #2822

Merged

5 tasks

corrected EXPECT_NEAR error

fc7ed1f

nhuurre reviewed Oct 26, 2022

View reviewed changes

Franzi2114 added 5 commits October 27, 2022 11:40

n changed to dim

e96f8c9

hcubature: removed swaps

a1ccf55

hcubature: not used templates removed

b5ef176

all other n changed to dim

48e5882

some allocs changed into vectors

eeb9b13

SteveBronder reviewed Nov 1, 2022

View reviewed changes