v1.7.0 and v1.7.1 -- cannot open remote files

[barronh]

The new v1.7.0 and v1.7.1 fail when reading remote files with a curl error. The v1.5.4 to v1.6.5 work for the exact same code. I'm not sure how to resolve this. It is likely related to a previous issue with earlier versions.

Environment

• Python versions 3.9, 3.10.12, 3.11
• netCDF4 versions 1.7.0 and 1.7.1
• OS Linux (RedHat Enterprise Linux 8; Ubuntu 22.04)

Steps to reproduce error:

$ python -m pip install netcdf4==1.7.0
$ python -c "import netCDF4; f = netCDF4.Dataset('https://www.ncei.noaa.gov/thredds/dodsC/model-ndgd-file/202406/20240616/LZQZ99_KWBP_202406161634')"
curl error details: 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "src/netCDF4/_netCDF4.pyx", line 2470, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2107, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: 'https://www.ncei.noaa.gov/thredds/dodsC/model-ndgd-file/202406/20240616/LZQZ99_KWBP_202406161634'

The same error is seen with 1.7.1 and the line numbers even stay the same.

Steps to resolve error:

$ python -m pip install netcdf4==1.6.5
$ python -c "import netCDF4; f = netCDF4.Dataset('https://www.ncei.noaa.gov/thredds/dodsC/model-ndgd-file/202406/20240616/LZQZ99_KWBP_202406161634')"
# no error

[barronh]

Probably a duplicate of #1335

[ocefpaf]

I can reproduce that but I have to confess I'm at a loss here. The test for this passes during the wheel build. See

netcdf4-python/.github/workflows/cibuildwheel.yml

Line 110 in 2b59eba

&& URL="https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/m2.hamtide11a.nc"

Something is different from the build env and the final wheel :-/

I guess we should be building our own curl. I'll try to tackle that next week.

[jswhit]

@ocefpaf looks like we are not building our own netcdf-c from github develop - that's what we will need to do to fix this (or wait until 4.9.3 is released). I don't think building our own curl will help, although it is a mystery why the test passes during the wheel build.

[ocefpaf]

It should be latest netcdf-c: https://github.com/ocefpaf/netcdf-manylinux/blob/5d88292a012de0e7648edb6bb02f84ad8f630b9f/install_libs.sh#L43

And at some point I do recall downloading the wheel and testing it. Not sure what went wrong here.

[jswhit]

I was looking at this and this. I wonder if some of the builds are getting those libs instead of the ones bundled in your image?

[ocefpaf]

I was looking at this and this. I wonder if some of the builds are getting those libs instead of the ones bundled in your image?

Probably not b/c the sdist happens in another image and the problem doesn't seems to exits in the macOS build, only on Linux. We can disable the ncgen tests (to remove the bew install), and isolate the sdist build more if you want to.

BTW, indeed the test inside the image is useless b/c if you check the image out, install (or build) netCDF4, and test it there the tests pass:

docker run --net=host -it --rm -v $(pwd):/home/ ghcr.io/ocefpaf/manylinux2014_x86_64-netcdf /bin/sh
cd /home
pipx install netCDF4  # get the one from PyPI
ncinfo "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/m2.hamtide11a.nc"

or build it,

docker run --net=host -it --rm -v $(pwd):/home/ ghcr.io/ocefpaf/manylinux2014_x86_64-netcdf /bin/sh
cd /home
pipx run build --wheel . --outdir dist
auditwheel repair dist/*.whl
pipx install wheelhouse/*.whl
ncinfo "https://icdc.cen.uni-hamburg.de/thredds/dodsC/ftpthredds/hamtide/m2.hamtide11a.nc"

If you copy that wheel outside of the image and install it, the test will fail.

I'm either missing something obvious, and/or building it against the images' curl and openssl is not OK.

For now I'm trying to build openssl+curl to test if that would fix it, but the image is failing when running cmake for netcdf-c. Still debugging it in ocefpaf/netcdf-manylinux#1

[ocefpaf]

OK, I think I got it. This is a wheel that is built with the latest image and I was able to install it and test against those URLs.

netCDF4-1.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.zip

I'm finalizing the image and we can send a PR to build them in the cibuildwheel GitHub Action for better testing. We have to download them and test locally though b/c the test in the image will pass regardless.

[ocefpaf]

@jswhit I'm clocking out for today but I believe that the images created in

https://github.com/ocefpaf/netcdf-manylinux/actions/runs/9651107110/job/26618225405

will solve the problem. They build netcdf-c with latest openssl and curl, from source, and ship all the necessary parts for the netcdf4 wheel.

The wheels built in https://github.com/Unidata/netcdf4-python/actions/runs/9652295373 used the Linux image already and I was able to use them locally without the any problem. The aarch64 is unsted b/c that image is quite slow and not built yet. I can't test it b/c I don't own any aarch64 hardware. The Windows and Mac Wheels are unchanged.

If you believe this is good we can tag a 1.7.1.post1 release and yank 1.7.1 from PyPI. The source code is OK and only wheels require rebuilding, so no need for a new 1.7.2 release (unless you have a few more PRs to merge and add to the mix).

[lbesnard]

as another test, I have a similar issue that affected my code. The import of the NetCDF4 library at 1.7.1 breaks things, even if not used by the rest of the code (which made it a bit hard to debug):

import netCDF4
import xarray as xr
import s3fs

s3 = s3fs.S3FileSystem(anon=True)
file_path = 's3://imos-data/IMOS/SRS/SST/ghrsst/L3S-1d/dn/2024/20240101092000-ABOM-L3S_GHRSST-SSTfnd-AVHRR_D-1d_dn.nc'

# Open the file using s3fs
with s3.open(file_path, 'rb') as f:
    ds = xr.open_dataset(f, engine='h5netcdf', concat_characters=True)

print(ds)

->

   return h5ds.get_num_scales(self._id, self._dimension)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5ds.pyx", line 72, in h5py.h5ds.get_num_scales
  File "h5py/defs.pyx", line 4461, in h5py.defs.H5DSget_num_scales
RuntimeError: Unspecified error in H5DSget_num_scales (return value <0)

But with NetCDF4 1.6.5 , the same code works.

[ocefpaf]

ds = xr.open_dataset(f, engine='h5netcdf', concat_characters=True)

@lbesnard you are using h5netcdf, that uses h5py, under the hood. netCDF4 is not used at all there.

BTW, if you downgrade to h5py 3.9.0, it should work.

[ocefpaf]

@barronh 1.7.1.post1 was just released and it should fix this problem. Can you check with your remote files?

[lbesnard]

@lbesnard you are using h5netcdf, that uses h5py, under the hood. netCDF4 is not used at all there.

I agree that in this example netCDF4 is not used. But this was actually my point. This is part of a bigger code which has other functions using the netCDF4 library. And just having this imported in a module breaks xarray as highlighted in my example.

I'll try downgrading h5py , thanks!

[ocefpaf]

@lbesnard this issue is about reading remote files files. Yours seems like a duplication of #1343, which is a recurring problem with wheels.

[barronh]

Thanks @ocefpaf and @jswhit! Your support of netcdf4 for python means so much to the community! And, your rapid response to issues is a model for us all!

I confirm that 1.7.1.post1 solves the problem. I tested on both Google Colab and our computational cluster in a clean virtual environment.

python -m pip install netcdf4==1.7.1.post1
python -c "import netCDF4; f = netCDF4.Dataset('https://www.ncei.noaa.gov/thredds/dodsC/model-ndgd-file/202406/20240616/LZQZ99_KWBP_202406161634')"

Just in case, I want to note a few things for future issue readers who are installing with pip. This error can be solved with 'netcdf4==1.7.1.post1' or 'netcdf4>=1.7.1', because in both cases pip automatically choose 1.7.1.post1. If you specify 'netcdf4==1.7.1', pip will install the wheel with the problem with a warning that it is a yanked version. In that case, the error described here will persist. This will only come up for people specifying 1.7.1 explicitly. If your system already has netcdf4==1.7.1, either uninstall first or specify 'netcdf4>=1.7.1.post1'. To be clear, what I am describing not an issue with netcdf4 and is just describing how to ensure pip picks up the right version.

I leave the issue for you to close. Nothing so great as checking something off a list!

[ocefpaf]

Closing this one to avoid adding more h5py incompatibility discussion in the wrong issue. Thanks!