Bundle NetCDF C++ bindings and build privately as needed

[PhilMiller]

In recent efforts to run ngen on NOAA's Hera supercomputer, I encountered multiple difficulties related to our use of the NetCDF C++ bindings. These related to installation location, version differences with no way to tell, and C++ ABI evolution. These issues occur on other machines, and with Hera's move to Rocky 8 as well.

Discussion below has concluded that we wish to continue using the C++ bindings, and that they're feasible to manage for ourselves when not suitably system-provided. This will ensure that if they're not built with/as-part-of the NetCDF package, they're built with exactly the C++ compiler version and options that we're using for the rest of the code.

[PhilMiller]

@program-- thoughts?

[program--]

I think that's a reasonable suggestion. The only difference between the two is convenience. It may be useful to have some abstractions similar to mdarray/mdframe on top of the C API (at least for things like exceptions), but in my experience with it and the RNetCDF package, that's not wholly necessary.

[PhilMiller]

For others: the problem is that netcdf-cxx4 is a separate package from core netcdf. Sometimes it's not installed at all, sometimes it's installed but in a different location, sometimes it's installed with unexpected C++ compiler/version/ABI that doesn't match anything sensible.

[PhilMiller]

Oh, yeah, and netcdf-cxx4 has shown poor versioning hygiene - 4.3.1 added a function not present in 4.3.0.

[donaldwj]

The main problem with the C API is it very verbose and can hard to read and slow to develop code with, it also suffers from the problem of correct code using the C API spends more than 50% of time checking return conditions.

[donaldwj]

So, the reasons for the C++ API where

• readability
• error handling
• memory safety (destructors release netcdf api allocations preventing possible memory leaks)

If we think that we would be better with C API for less setup issues and can easily switch the writing code.

[PhilMiller]

Those are all pretty reasonable concerns.

Using unique_ptr with a custom deleter would mitigate some of the memory safety, but it would still require discipline on our part to wrap up every object that gets created.

As an alternative, we could consider only finding netcdf-cxx4 in exactly the same path as the base netcdf, and building our own copy with a submodule under extern/ if it's not there.

[donaldwj]

I was going to suggest bundling netcdf-cxx4 into extern. It is not very big but we will need to check the license to see if it permits that.

[PhilMiller]

Yep, license is approximately equivalent to MIT or 2-clause BSD. Do whatever we want, maintain the copyright markings on it.

[jduckerOWP]

Just as a comment here, but I've seen this same issue as well with the NOAA Cheyenne cluster for not having the latest updated netcdf-cxx4 libraries as well. This is likely a widespread issue with supercomputers not updating their netcdf library dependencies frequently and therefore is advised if we could constrain ngen library netcdf dependencies to just the baseline package if possible.

[PhilMiller]

I also just checked the upgraded nodes of Hera with Rocky 8 installed in place of CentOS 7. They also have netcdf modules that just include the base C/Fortran package, without netcdf-cxx4