Separate the major components of HDF5

[Dave-Allured]

A while back, I asked for a way to independently build the major components of HDF5. Now I think it is the right time to completely break up the major components into their own separate packages. This most obviously applies to the primary language wrappers, but also would work well for supplementals including the manuals and tools.

This move would reduce user install problems for the core C libraries; untangle compiler dependencies; and help segregate debugging by components. I think it would also help the HDF Group to more easily manage individual components, into the future.

If you would like to prioritize, try the fortran wrapper first, 'cuz that is the one that is bugging me the most.

[jhendersonHDF]

Hi @Dave-Allured,

not wanting to let this one slip by, I wanted to ask what problem you're having and wanting to solve with this issue? Or I guess it makes more sense to ask what you're looking for as an end result of splitting up the components of the library? Doing so would certainly complicate our binary building process a bit, as well as maybe complicate things for users who want a "full build" as before. I think it would help to understand what your pain points with the Fortran build are before we start considering splitting out the wrappers from the main library.

[lackyO3]

One use case for this is specifically for building the Fortran interface. Fortran compilers construct so-called .mod files which are the equivalent of C header files, but are generated automatically when the Fortran module is compiled. There is, however, no standard for the structure of these files and they are incompatible between compiler vendors and sometimes between the versions of compilers from the same vendor. To support HDF5 for two incompatible instances of Fortran compilers a complete install of the HDF5 library is required even though the underlying C libraries that are 'wrapped' by the Fortran interface aren't materially different.

It would seem cleaner to be able to build multiple Fortran interfaces all wrapping identical HDF5 library builds. Of course the Fortran .mod files for the different instances of Fortran would need to be segregated from each other. I presume the best approach would be to use the file systems directory structure, but I'm unaware of a common approach to this.

(There should be a request for this from years ago, perhaps in the HDF Group's Jira bug tracker.)

[brtnfld]

The .mod file incompatibility problem that @lackyO3 describes is real and worth solving, but I don't think splitting the Fortran interface into a separate repository is the right approach. As a side note, I've ranted for years about the problematic nature of .mod files for library packages.

But anyway, here's why:

1. Tight coupling between C and Fortran layers. The Fortran wrappers must track the C API exactly — when a C function signature changes, the corresponding Fortran INTERFACE block and module code must update in lockstep. Keeping them in the same repo makes that a single commit; splitting them apart turns it into a cross-repo coordination problem with version pinning, compatibility matrices, and release synchronization overhead.
2. The Fortran build has bidirectional dependencies with the C layer. The generated type-mapping files (H5fortran_types.F90, H5f90i_gen.h) are produced by H5match_types.c, which needs both C-side type sizes and Fortran-compiler-specific KIND values. A separate Fortran repo can't configure or build independently — it inherently needs the C build artifacts and detection results, which undermines the case for full separation.
3. The .mod problem is a build/packaging concern, not a source organization one. The goal, that is to build the C library once, then build the Fortran layer N times for N compilers, doesn't require a repo split. It requires a build system mode that builds only the Fortran interface against an already-installed C library. That mode would re-run the Fortran compiler probes (KIND detection, intrinsic availability, etc.), regenerate the type mappings via H5match_types, and build/install only the Fortran-specific artifacts. Something like:

# Build and install C library once
cmake -DHDF5_BUILD_FORTRAN=OFF ... && make install

# Build Fortran interface per compiler, against installed C lib
FC=gfortran cmake -DHDF5_FORTRAN_ONLY=ON -DHDF5_C_INSTALL=/path/to/hdf5 ...
FC=ifx      cmake -DHDF5_FORTRAN_ONLY=ON -DHDF5_C_INSTALL=/path/to/hdf5 ...

This solves the multi-compiler problem without the downsides of repo fragmentation, and it keeps the Fortran source in sync with the C API it wraps. I'm not saying this is as straightforward as the code above, but it seems doable in theory.

[lackyO3]

I interpret the problem description of building the Fortran interface "against an already-installed C library" to imply that in this scenario the C library has already been installed. In other words the goal is to be able to return to a system in which the C library was installed (e.g. make install) some time in past and be able to build and install the Fortran interface for a new compiler--as opposed to building the Fortran interface against a built (e.g. make build) but not yet installed HDF5 library.

Point 1, that the C and Fortran need to go in lock step should not be a concern. The HDF Group's adoption of semantic version for HDF5 version 2+ takes care of that. Indeed the lock step requirement isn't unique to Fortran, any foreign language interface linking to the C library will need to rely on consistency of the the interface as expressed by the semantic version. One might even argue that the HDF Group has an obligation to split Fortran out and rely on the semantic version. To not do so suggests a lack of confidence in the implementation of the semantic versioning process used.

For point 2, if the build artifacts that the Fortran interface build depends on are not available from the installed C artifacts than it isn't possible to build the Fortran interface from an installed library.

As noted in point 3, a repo split would not be required, but meeting the goal to build additional Fortran interfaces against an already-installed C library should not incur any additional burdens that any other foreign language interface has. (Except that you need to sort our where to put the .mod files produced by the respective Fortran compilers.)

[Dave-Allured]

I enthusiastically agree with analysis of multiple fortran wrappers by @lackyO3, who articulated this far better than me.

[Dave-Allured]

@brtnfld said:

... understand what your pain points with the Fortran build are ...

Pain point: When trying to update only the core C installation, fortran problems delay and sidetrack the process. A recent example, there are others:
macports/macports-ports#32874 (comment)

(This one was also reported to HDF5 issue tracker: #6419)