Test one compiler per OS

[seantalts]

Note

Pull request #944 only addresses part of this issue / proposal:

• mac / clang 6 / libc++ Threading unit tests
• linux / gcc 5 / stdlibc++ OpenMPI unit tests
• mac clang 6 libc++ OR linux clang 6 libc++[2] distribution tests (without row vectors)

Additional tests on merge to develop
linux / gcc 5 / stdlibc++ Threading unit tests
mac / clang 6 libc++ OR linux / gcc 5 stdlibc++[1] GPU unit tests
mac clang 6 libc++ OR linux clang 6 stdlibc++[2] distribution tests (with row vectors)

• It does not address testing the header tests on mac + linux + windows.
• It does not address testing on Windows.

Summary:

As we discussed in the meeting on July 12th and in this discourse thread, we'd like to change our CI configuration to test just one per OS.

Description:

Here are the proposed supported compilers for each OS:
Mac: clang 5
Linux: clang 5 (maybe a gcc - TBD)
Windows: gcc 4.9.3

For tests, these are the tests that should be tested on each of those configurations:

1. header tests. make test-headers
2. unit tests. ./runTests.py test/unit
3. distribution tests. ./runTests.py test/prob

These tests only need to be run on one platform since they're not compiler dependent:

1. CppLint. make cpplint
2. Math dependencies (checks the source to make sure it's not including headers in a bad way). make test-math-dependencies
3. Building doxygen. make doxygen

From what I understand, inter-clang compatibility is pretty high, while GCC is needed for Windows RTools (until they switch to a modern GCC in April 2019). GCC had major changes across the 5.0 boundaries, but I think testing both modern ish non-Apple clang and old-ish GCC will still keep our code supported by the compilers basically all of our existing users have access to.

This ticket will involve changing the travis and Jenkins config (and for Jenkins agents, the installed software) to test the above compilers.

Once this is in, we should update the wiki page here: https://github.com/stan-dev/stan/wiki/Supported-C---Compilers-and-Language-Features

cc @bgoodri @syclik @bob-carpenter

Current Version:

v2.18.0

[bgoodri]

g++ is much more commonly used on Linux than clang++

…

On Mon, Jul 16, 2018 at 9:37 AM seantalts ***@***.***> wrote: Summary: As we discussed in the meeting on July 12th, we'd like to change our supported compilers to support (and test!) just one per OS. Description: Here are the proposed supported compilers for each OS: Mac: clang 6.0 <https://cran.r-project.org/bin/macosx/tools/> Linux: clang 6.0 Windows: gcc 4.9.3 <https://cran.r-project.org/bin/windows/Rtools/> From what I understand, inter-clang compatibility is pretty high, while GCC is needed for Windows RTools (until they switch to a modern GCC in April 2019). GCC had major changes across the 5.0 boundaries, but I think testing both modern ish non-Apple clang and old-ish GCC will still keep our code supported by the compilers basically all of our existing users have access to. This ticket will involve changing the travis and Jenkins config (and for Jenkins agents, the installed software) to test the above compilers. Once this is in, we should update the wiki page here: https://github.com/stan-dev/stan/wiki/Supported-C---Compilers-and-Language-Features cc @bgoodri <https://github.com/bgoodri> @syclik <https://github.com/syclik> @bob-carpenter <https://github.com/bob-carpenter> Current Version: v2.18.0 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#943>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqhBYyIeTSoIgCvPCA0SxoS5kjXyJks5uHJcZgaJpZM4VRJGI> .

[seantalts]

I like clang for Linux because:

1. It's much faster and more memory efficient (meaning we can run a higher -jX and tests will run faster). This saves the project actual dollars.
2. clang seems like the future - has many more companies backing it and generally seems like it has a steeper trajectory
3. clang has better static analysis tooling
4. I don't think stats exist describing our Linux users compiler availability. I'm saying availability here because we don't necessarily care if our code somehow doesn't compile on Linux gcc despite it compiling for Windows gcc and the user is able to easily install clang and have things work. To make the tradeoff explicit, we need to resize our support commitments to fit the person-hours of those working on things like these at the expense of a low chance of making some Linux users install an extra package.

[bgoodri]

In the early days of Stan, clang++ was much faster to compile and used less RAM. That is much less true today as clang++ has taken more time and RAM in order to improve execution speed. The static analysis thing is nice but you can get that on a Mac or just by running it occasionally on Linux. That doesn't have much to do with whether unit tests pass or models run much more slowly. But my main point is that if you are going to only test one compiler per OS, it should be the one that you anticipate being used the most, which is gcc on Linux by orders of magnitude

…

On Mon, Jul 16, 2018 at 10:57 AM seantalts ***@***.***> wrote: I like clang for Linux because: 1. It's much faster and more memory efficient (meaning we can run a higher -jX and tests will run faster). This saves the project actual dollars. 2. clang seems like the future - has many more companies backing it and generally seems like it has a steeper trajectory 3. clang has better static analysis tooling 4. I don't think stats exist describing our Linux users compiler availability. I'm saying availability here because we don't necessarily care if our code somehow doesn't compile on Linux gcc despite it compiling for Windows gcc and the user is able to easily install clang and have things work. To make the tradeoff explicit, we need to resize our support commitments to fit the person-hours of those working on things like these. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqhKKmpghHdFgP1hWR2L1KuCuesNMks5uHKnmgaJpZM4VRJGI> .

[bbbales2]

Yeah clang is nice and all, but it performs about the same as gcc for me (edit: agreeing with other Ben here). I think gcc does a better job of flagging code with warnings (int/size_t comparisons etc.) in terms of minor arguments for/against. I definitely think both should be tested. If it's only one, then I agree gcc is the one.

[seantalts]

@bbbales2 can you provide some reasoning? Why should both compilers be tested on Linux given what I wrote above?

I can try out a modern GCC (which version?) and see how long the distribution tests take. I know gcc-4.9.3 is at least twice as slow as clang-3.8 but I suppose you might be referring to those versions as "the early days"

[bgoodri]

The early days were actually g++-4.6, which didn't support C++11.

…

On Mon, Jul 16, 2018 at 11:37 AM seantalts ***@***.***> wrote: @bbbales2 <https://github.com/bbbales2> can you provide some reasoning? Why should both compilers be tested on Linux given what I wrote above? I can try out a modern GCC (which version) and see how long the distribution tests take. I know gcc-4.9.3 is at least twice as slow as clang-3.8 but I suppose you might be referring to those versions as "the early days" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqu8NOWk3ousnrJWriupB3H5L_GISks5uHLNTgaJpZM4VRJGI> .

[bob-carpenter]

I've found warnings keep leapfrogging one another in g++ and clang++. I use both if I can't sort something out. clang++ used to be a lot faster, but now I'm finding my models take forever to compile on the Mac. We really really need to address the compilation time problem if the compilers are going to keep going crazy with compilation time.

…

On Jul 16, 2018, at 5:22 PM, Ben Bales ***@***.***> wrote: Yeah clang is nice and all, but it performs about the same as gcc for me. I think gcc does a better job of flagging code with warnings (int/size_t comparisons etc.) in terms of minor arguments for/against. I definitely think both should be tested. If it's only one, then I agree gcc is the one. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[seantalts]

One reason I'm particularly fond of matching the compiler on Mac and Linux is because those users are usually pretty flexible and we generally want to avoid giving too many compilation targets for developers or it will start to approach front-end developer's dreaded cross-browser compatibility matrices. This gives me a strong bias towards selecting the same compiler for both, and given most of our users are on RStudio on Mac (which needs clang to link) and the Chrome team's recent writeup of clang vs. gcc I'm still very biased towards only testing Clang for Linux in CI, even if it means we need to ask some Linux users to install an extra package (under what would be pretty weird circumstances).

Running the performance numbers for distribution tests on gcc vs clang for Linux would help convince me otherwise, though. Anyone know what version I should target? I can't (quickly) find the docs that describe what RStudio is usually built with for Linux.

[bgoodri]

We don't have the conference bodyguard budget to tell Linux people "just install clang-6.0". For unit testing, it seems obvious to me that you want diversity of compilers rather than homogeneity. Also, it is fine to say that we only have the resources to test a finite set of things, but it is wrong to suggest that we are not going to fix compiler issues for compilers that we do not test. We are already responsible for 12 OS / compiler combinations for each R package and there are a few more with PyStan where we have to fix things when they go wrong. It is just a matter of how many things we catch before they get to users.

…

On Mon, Jul 16, 2018 at 11:54 AM seantalts ***@***.***> wrote: One reason I'm particularly fond of matching the compiler on Mac and Linux is because those users are usually pretty flexible and we generally want to avoid giving too many compilation targets for developers or it will start to approach front-end developer's dreaded cross-browser compatibility matrices. This gives me a strong bias towards selecting the same compiler for both, and given most of our users are on RStudio on Mac (which needs clang to link) and the Chrome team's recent writeup of clang vs. gcc <http://blog.llvm.org/2018/03/clang-is-now-used-to-build-chrome-for.html> I'm still very biased towards only testing Clang for Linux in CI, even if it means we need to ask some Linux users to install an extra package. Running the performance numbers for distribution tests on gcc vs clang for Linux would help convince me otherwise, though. Anyone know what version I should target? I can't (quickly) find the docs that describe what RStudio is usually built with for Linux. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#943 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqsbKVdLH-ixDnhHTPHhnDcGolGv-ks5uHLdEgaJpZM4VRJGI> .