Reuse of `Makevars.in` and `configure` script in glmmTMB

[gavinsimpson]

Dear data.table developers,

Having recently moved to using MacOS X, I am trying to enable OpenMP support out-of-the-box on that OS for CRAN's binaries for various packages that I make frequent use of.

You have already solved this problem through a lot of hard work, trial and error, and testing. When I started looking into enabling OpenMP support in CRAN's MacOS X binaries for other packages, I came across data.table's Makevars.in and configure script solution. Testing these locally with (small) modifications where needed, I was able to get this working for the glmmTMB and mgcv packages.

I recently proposed to add this code to glmmTMB, to get some testing to see if this worked more broadly: glmmTMB/glmmTMB#1214 I have not yet approached Simon Wood, mgcv's maintainer, about adding similar files to mgcv.

As it seems to be working in glmmTMB, I wanted to reach out to you to i) inform you that this process is ongoing and that there is an intention to reuse your code in a package with a very different licence to that of R data.table, ii) ask if you had any specific concerns that you would like addressed, iii) enquire as to how you would like to see your efforts in developing these scripts acknowledged.

As I understand the MPL-2, it is allowed for glmmTMB (AGPL-3) to include the modified Makevars.in, configure, and cleanup (unmodified) files/scripts so long as (from answer to Q9 in the FAQ):

To see the complete set of requirements, read the license. However, generally:

• You must inform the recipients that the source code is made available to them
  under the terms of the MPL (Section 3.1), including any Modifications (as
  defined in Section 1.10) that you have created.

• You must make the grants described in Section 2 of the license.

• You must respect the restrictions on removing or altering notices in the source
  code (Section 3.4).

I am conscious of the various licensing discussions that you have had and wish to respect the decisions you have reached as well as the spirit of those discussions. If you have any comments or questions on the above, please let me know. If you'd rather respond off list, I am at ucfagls AT gmail DOT com.

Thanks in advance

Gavin

[aitap]

Dear Gavin, Thank you for your interest in enhancing OpenMP support in binary CRAN packages for macOS and in data.table! It may sound surprising, but our ./configure is actually not the best way to check for OpenMP support in an R package. For one, it ignores the Make variables described in WRE 1.1.2.1 [1] because the initial check for OpenMP support was incorrectly written and then wrongly fixed in 2020 [2]. Additionally, it fails to detect some broken configurations on macOS which successfully compile a shared library that will later fail to load [3]. We keep it this way because it works well enough. When something breaks, adding one more special case to the pile [4] is low effort (it's just one special case), low risk (other special cases remain in the pile, so probably won't break), and high reward (it fixes a specific issue). Rewriting the configure script to work right is high effort (all the special cases, including weird configurations, need to be re-tested), high risk (there will be one configuration we'll forget to test that'll turn out to be very important for users), and low reward (at best things will work as well as they do now). For mgcv and glmmTMB, I think it's worth at least trying to write correct new code. On all operating systems except macOS, using OpenMP in C code is as easy as setting PKG_CFLAGS = $(SHLIB_OPENMP_CFLAGS) PKG_LIBS = $(SHLIB_OPENMP_CFLAGS) in src/Makevars, no configure script needed. This works well because there's usually only one source of the OpenMP runtime, and it's either the operating system (GNU/Linux, *BSD) or Rtools (Windows). The best thing about OpenMP runtimes on macOS is that there's so many to choose from. Linking an R package with a library from Homebrew? It might link its own libomp. Running reticulate to interface with deep learning packages installed from binary wheels? Those may include one too. Starting with version 4.3, R brings its own OpenMP runtime too, but doesn't use it by default. When multiple OpenMP runtimes meet in one process, bad things happen. For example, Intel OpenMP crashes the process as soon it detects it's being double-initialised. Since OpenMP is generally not supported on macOS, R leaves SHLIB_OPENMP_CFLAGS empty there. If you care about CRAN binaries of your package having OpenMP support on R ≥ 4.3, you additionally need to test for (1) running on macOS and (2) PKG_CPPFLAGS='-Xclang -fopenmp' PKG_LIBS=-lomp producing a shared library that you can load and execute code from The above should be enough. data.table has additional special cases for Homebrew and plain -fopenmp on Darwin. I think the latter should be handled by R detecting OpenMP support at configuration time and setting SHLIB_OPENMP_CFLAGS, but I cannot easily test that. I think the right way to test for OpenMP support looks more like [5] (don't try anything else if $(SHLIB_OPENMP_CFLAGS) works! when testing compiler flags, make sure to load-test the resulting shared libraries!), except it needs to be even simpler: use the compiler flags beyond $(SHLIB_OPENMP_CFLAGS) only on macOS and minimise the number of special cases. [1] https://cran.r-project.org/doc/manuals/R-exts.html#OpenMP-support [2] 3d24927 [3] #6622 [4] #6633 [5] #6642

…

-- Best regards, Ivan

[gavinsimpson]

Thanks for the comments and suggestions @aitap --- I wasn't sure about keeping Homebrew support for these packages (glmmTMB, mgcv). R Core do not / Simon Urbanek does not support Homebrew so if the aim is to get openmp support in CRAN's binaries for mgcv and glmmTMB, this suggests fixing that one issue and leaving everything else well-alone. Right now openmp doesn't work at all for those packages on macosx so any improvement is better, so long as it doesn't make things worse elsewhere.