opendap access crash only with pip installation in ubuntu-latest SO

[aragong]

Hi, I am suffering crashes in my github action tests only when I use netCDF4 opendap access (I tried several datasets in different urls/servers) installed with pip install netCDF4 and ubuntu-latest machines.

I supposse that could be an xarray related error but seems to be a problem in the installation related with netCDF4. view issue in xarray repository: pydata/xarray#7773

I also obtain this error if I use google colab platform that I suppose that use the same environment in ubunutu-latest SO.

I think that is related to this other opened issue... not sure! #1179

Thank you in advance!

To report a non-security related issue, please provide:

• the version of the software with which you are encountering an issue
• environmental information (i.e. Operating System, compiler info, java version, python version, etc.)
• a description of the issue with the steps needed to reproduce it

If you have a general question about the software, please view our Suggested Support Process.

[jswhit]

Does ncdump work on that URL? If so, then it's a python interface issue, if not it's a netcdf-c lib issue.

[aragong]

I tried this in a google colab script

1 - I intalled netcdf4 using pip --> OK
2 - ncdump is not installed in the ubuntu by default --> apt install netcdf-bin
3 - ncdump to the url works fine in terminal
4 - Dataset class in python raise an error
5 - Xarray also raises an error

I am totally lost! thank you in advance! Is it something wrong with the installation of the ubuntu-latests? or when pip installs netcdf4 some requirement is lost?

[ocefpaf]

I think that is related to this other opened issue... not sure! #1179

I think it is the same issue. The problem seems to be with the openssl used in the Linux wheel. (It does not happen in both macOS and Windows wheels.)

[jswhit]

I rebuilt the linux wheels with a newer version of openssl (1.1.1l instead of 1.0.2). The wheels are at https://github.com/MacPython/netcdf4-python-wheels/releases/tag/v1.6.3rel - @aragong could you give one of these a try?

[ocefpaf]

I tested them locally and no luck. I guess that Unidata/netcdf-c#2459 was closed prematurely. Something in the SSL CA cert is not OK.

[aragong]

I rebuilt the linux wheels with a newer version of openssl (1.1.1l instead of 1.0.2). The wheels are at https://github.com/MacPython/netcdf4-python-wheels/releases/tag/v1.6.3rel - @aragong could you give one of these a try?

I gave it a try in google colab and it is still crashing, same error (-68), I let you the link to the script here

Also I have one extra doubt:
Why now I have to install netcdf-bin with apt? I do not remember have to do this in google colab or github deploys...
Is there any change in linux SO deployments??? or is because the python interface was in charge of automatically installing this library in past? sorry about my ignorance.... 😅

Edit: Now installing your wheel I do NOT have to install netcdf-bin with apt... This behavior change again! I am nor even more confused! 😅

[jswhit]

I added a test using the URL https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide//m2.hamtide11a.nc to the wheel building workflow, and I get failures on Ubuntu. However, adding the same test to the github actions workflow in the netcdf4-python repo, I don't get failures. Must be either be something in the Ubuntu environment, or the versions of the lib dependencies (most likely ssl or curl).

[jswhit]

updating openssl to 3.0.1 and curl to 8.0.1 doesn't help. As @ocefpaf suggested it's most likely something to do with the CA cert process happening in libcurl (discussion at Unidata/netcdf-c#2459).

[jswhit]

would really like to solve this, but I'm out of ideas right now. I'm tempted to blame a netcdf-c bug, save for the fact that it does work with curl installed via conda or apt-get on Ubuntu. This makes me think it still might be an issue with the way we are building ssl and/or curl.

[ocefpaf]

Maybe we could try to reduce the scope of ocefpaf/netcdf4-win-wheels#6 (no macOS, ppc, and aarch) and build the Linux wheels that way too.

[DennisHeimbigner]

Try the following to get more info:

1. export CURLOPT_VERBOSE=1
2. ncdump -h https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide//m2.hamtide11a.nc

[jswhit]

Thanks @DennisHeimbigner that suggestion helped. I was able to confirm that running ncdump with that URL right after the library is built works. Somehow after the libs are copied into the wheel and installed on another system the CA cert stuff no longer works. Maybe an environment variable needs to be set to point to the location of the certificates?

this is the error:

*   Trying 134.100.239.238:443...
* Connected to icdc.cen.uni-hamburg.de (134.100.239.238) port 443 (#0)
* ALPN: offers http/1.1
* error setting certificate verify locations:  CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
* error setting certificate verify locations:  CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
* Closing connection 0
* error setting certificate verify locations:  CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details: 

/etc/pki/tls/certs is the location of the certificates on the manylinux distro, I think on Ubuntu they live in /etc/ssl/certs