Output to GDS

[jedwards4b]

I have a version of nccopy in which I have inserted code to use
The HDF5 GPUDirect Storage (GDS) VFD as an output device. I have
confirmed that data is copied to a buffer on the gpu and the write
is using GDS - I would like to work with you to add this capability to
netcdf-c in a way suitable for distribution. I think maybe we can add a
#define NC_HDF5GDS 0x800000 in netcdf.h

[jedwards4b]

My question is - does the data compression happen before or after the call to H5Dwrite in hdf5var.c?
If it happens after then I am also doing the data compression on the gpu.

[DennisHeimbigner]

All compression is carried out at the point that a chunk is written. So it happens between H5Dwrite and the VFD.

[jedwards4b]

Awesome - the next thought is that maybe I can accomplish this without any changes in netcdf - can I use the filters mechanism to invoke the gds_vfd plugin - then just have the cuda calls at the application level?

[DennisHeimbigner]

I do not think that will work because of the architecture of HDF5 and filters.
If you look at libhdf5 and see how H5FDhttp is used, you may get some idea
of how to add your VFD.
Adding this to the netcdf-c code base is a bit more of a problem.
Mode flags are a bit of a scarce resource.
You might post a github discussion raising the issue of user-defined VFDs.

[jedwards4b]

The parallelio library is already using UDF0 but I think that I can use UDF1 to overload the create and open functions then just set everything else in the dispatch table to be the same as UDF0. Does that sound like a reasonable approach?

[jedwards4b]

I should be able to use a call like nc4_file_list_add to call H5Pset_driver_by_name(fapl_id, "gds", NULL) from the netcdf library
prior to calling nc4_createfile with the same ncid - is there an existing mechanism to do this or do I need to create something new?

[jedwards4b]

@DennisHeimbigner any thoughts on how to call H5Pset_driver_by_name from netcdf-c?

[DennisHeimbigner]

Unfortunately, I have not had time to think about it. There is a pressing bug I need to fix.

[jedwards4b]

I understand, please consider when you have time.

Here is another issue for your consideration. Variables which do not have an unlimited dimension are written via the vfd-gds driver however variables which do have an unlimited dimension are being copied back to the cpu and bypassing the gds driver. I got some suggestions in the discussion here but they did not seem to help. For now I've just removed the unlimited dimension from the dataset so that we can use GDS for all variables. Do you have any thoughts on why variables with the unlimited dimension are not using GDS?

[DennisHeimbigner]

One thought that occurs to me is to use the netcdf-c library .rc file mechanism.
The library reads a file called .ncrc which contains lines like this:

HTTP.VERBOSE=1

and these (key,value) pairs are accessible to the netcdf-c library code.
So one possibility is to add a new pair something like this

HDF5.VFD=