NetCDF: Malformed URL with zarr URL (netcdf 4.9.2)

[Alexander-Barth]

I am trying to access with ncdump the following URL.
ncdump (version 4.9.2) tells me that the URL is bad, but I do not see how to correct it.

~/opt/netcdf-4.9.2/bin/ncdump -h 'https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr#mode=zarr'
/home/abarth/opt/netcdf-4.9.2/bin/ncdump: https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr#mode=zarr: NetCDF: Malformed URL

I compiled netcdf from source using cmake:

CMAKE_PREFIX_PATH=/home/abarth/opt/aws-sdk-cpp/  cmake .. -DENABLE_S3=on -DENABLE_NCZARR_S3=on -DENABLE_NCZARR_ZIP=on -DCMAKE_INSTALL_PREFIX:PATH=/home/abarth/opt/netcdf-4.9.2/

# Features
--------
Benchmarks:		no
NetCDF-2 API:		yes
HDF4 Support:		no
HDF5 Support:		yes
NetCDF-4 API:		yes
CDF5 Support:		yes
NC-4 Parallel Support:	no
PnetCDF Support:	no

DAP2 Support:		yes
DAP4 Support:		yes
Byte-Range Support:	yes

S3 Support:	        yes

NCZarr Support:		yes
NCZarr Zip Support:     yes

Diskless Support:	yes
MMap Support:		yes
JNA Support:		no
ERANGE Fill Support:	yes
Relaxed Boundary Check:	yes

Multi-Filter Support:	yes
Quantization:		yes
Logging:     		no
SZIP Write Support:     yes
Standard Filters:       deflate szip bz2
ZSTD Support:           no
Parallel Filters:       yes

I can read the data with xarray in python (without providing credentials):

In [59]: ds = xr.open_dataset("https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr",engine = "zarr");

In [60]: ds
Out[60]: 
<xarray.Dataset>
Dimensions:    (elevation: 141, latitude: 380, longitude: 1016, time: 12600)
Coordinates:
  * elevation  (elevation) float32 -5.754e+03 -5.646e+03 ... -3.166 -1.018
  * latitude   (latitude) float32 30.19 30.23 30.27 30.31 ... 45.9 45.94 45.98
...

How can I access this data using the NetCDF API? Any help would be greatly appreciated. Also other example how to use netcdf + zarr would be useful.

[DennisHeimbigner]

I assume you are using aws-sdk-cpp library. As far as I know, that library will not work with non-amazon servers.
I am working to improve support for that case, but I need some test sites. Is that dataset (https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr) public?

[Alexander-Barth]

Yes, I am using the aws-sdk-cpp library. The server is indeed public (I can access the data without providing credentials in xarray), but I have no special insights how this server is configured.

[DennisHeimbigner]

When I stick that URL in a browser, I get the response below. So it may actually require credentials.
Can you see if you have profiles in ~/.aws/config or ~/.aws/credentials? Or better yet, temporarily
hide those two files (config and credentials) and see if you can still access that URL.

<Error>
<Code>AccessDenied</Code>
<BucketName>mdl-arco-time</BucketName>
<RequestId>
tx0000000000000d2af63a7-00656ba924-58d87b7f-default
</RequestId>
<HostId>58d87b7f-default-waw3-1</HostId>
</Error>

[Alexander-Barth]

I do not have thes configurations files:

$ ls -l ~/.aws/config  ~/.aws/credentials
ls: cannot access '/home/abarth/.aws/config': No such file or directory
ls: cannot access '/home/abarth/.aws/credentials': No such file or directory

I can manually get the metadata using curl, for example:

$ curl -s https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/.zgroup | head
{
    "zarr_format": 2
}

$ curl -s https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/.zmetadata | head
{
  "metadata": {
    ".zattrs": {
      "Conventions": "CF-1.0",
      "bulletin_date": "20200901",
      "bulletin_type": "reanalysis",
      "comment": "Please check in CMEMS catalogue the INFO section for product MEDSEA_MULTIYEAR_PHY_006_004 - http://marine.copernicus.eu ",
      "contact": "servicedesk.cmems@mercator-ocean.eu",
      "field_type": "daily_mean_centered_at_time_field",
      "institution": "Centro Euro-Mediterraneo sui Cambiamenti Climatici - CMCC, Italy",
abarth@work:~/.julia/dev/CommonDataModel (main) 

$ curl -s  https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/time/.zattrs | head
{
    "_ARRAY_DIMENSIONS": [
        "time"
    ],
    "calendar": "standard",
    "units": "minutes since 1900-01-01"

Surprisingly the server returns a 403 Forbidden error for any file that do not exists (rather than a 404)

$ curl -s 'https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/does_not_exists'
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><BucketName>mdl-arco-time</BucketName><RequestId>tx0000000000000005db7a8-00656d85a6-9ef59483-default</RequestId><HostId>9ef59483-default-waw3-1</HostId></Error>

I can manually download a chunk of the variable time but not the first chunk of the variable uo (size 12600 x 141 x 380 x 1016, dimensions time, depth, lat, lon) :

$ curl  https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/uo/0.0.0.0 > out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   230  100   230    0     0   1229      0 --:--:-- --:--:-- --:--:--  1229

$ cat out
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><BucketName>mdl-arco-time</BucketName><RequestId>tx00000000000000033ad1f-00656d86b7-9f064368-default</RequestId><HostId>9f064368-default-waw3-1</HostId></Error>

But it seems that the first 16 chunks (along the depth dimension), are actually below the ocean floor and are filled with fill values (as checked in python). The first actual chunk is the chunk 17, which I can also download via curl:

$ curl  https://s3.waw3-1.cloudferro.com/mdl-arco-time/arco/MEDSEA_MULTIYEAR_PHY_006_004/med-cmcc-cur-rean-d_202012/timeChunked.zarr/uo/0.17.0.0 > out
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22548  100 22548    0     0   111k      0 --:--:-- --:--:-- --:--:--  112k
abarth@work:~/.julia/dev/CommonDataModel (main) 
$ ls -l out
-rw-rw-r-- 1 abarth abarth 22548 Dec  4 09:37 out

This seems to be conformant with the OGC spec:

There is no need for all chunks to be present within an array store. If a chunk is not present then
it is considered to be in an uninitialized state. An unitialized chunk MUST be treated as if it was
uniformly filled with the value of the “fill_value” field in the array metadata. If the “fill_value” field
is null then the contents of the chunk are undefined.

Ref: https://portal.ogc.org/files/100727

Zarr is all new to me, and still full or surprises :-).