geopandas 1.0.0 or pyogrio 0.9.0 doesn't support OGR_GMT output?

[seisman]

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?):

Extension error:
Here is a summary of the problems encountered when running the examples:

Unexpected failing examples:

    ../examples/get_started/04_table_inputs.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/get_started/04_table_inputs.py", line 111, in <module>
        fig.plot(data=gdf, style="c0.2c", fill="purple")
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 609, in new_module
        return module_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 773, in new_module
        return module_func(*bound.args, **bound.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/src/plot.py", line 258, in plot
        with lib.virtualfile_in(
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/contextlib.py", line 137, in __enter__
        return next(self.gen)
               ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/tempfile.py", line 164, in tempfile_from_geojson
        geojson.to_file(**ogrgmt_kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/geodataframe.py", line 1536, in to_file
        _to_file(self, filename, driver, schema, index, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 686, in _to_file
        _to_file_pyogrio(df, filename, driver, schema, crs, mode, metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 748, in _to_file_pyogrio
        pyogrio.write_dataframe(df, filename, driver=driver, metadata=metadata, **kwargs)
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 456, in write_dataframe
        path, driver = _get_write_path_driver(path, driver, append=append)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 564, in _get_write_path_driver
        if not ogr_driver_supports_write(driver):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "pyogrio/_ogr.pyx", line 105, in pyogrio._ogr.ogr_driver_supports_write
      File "pyogrio/_ogr.pyx", line 287, in pyogrio._ogr._get_driver_metadata_item
    pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL)

    ../examples/gallery/lines/roads.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.

[seisman]

That's weird. I tried to create the same environment locally.

mamba create --name pygmt-docs python=3.12 gmt=6.5.0 ghostscript=10.03.1 numpy pandas xarray netCDF4 packaging contextily geopandas ipython rioxarray build make pip myst-parser panel sphinx sphinx-autodoc-typehints sphinx-copybutton sphinx-design sphinx-gallery sphinx_rtd_theme

Now I have geopandas 1.0.0, pyogrio 0.9.0 and gdal 3.9.0 installed, but the script works well for me.

In [1]: import pygmt

In [2]: import geopandas as gpd

In [3]: # Example GeoDataFrame
   ...: gdf = gpd.GeoDataFrame(
   ...:     {
   ...:         "geometry": gpd.points_from_xy([2, 5, 9], [2, 3, 4]),
   ...:         "value": [10, 20, 30],
   ...:     }
   ...: )
   ...:
   ...: # Use the GeoDataFrame to specify the 'data' parameter
   ...: fig = pygmt.Figure()
   ...: fig.basemap(region=[0, 10, 0, 5], projection="X10c/5c", frame=True)
   ...: fig.plot(data=gdf, style="c0.2c", fill="purple")
   ...: fig.show()

[seisman]

There is one more failure:

../examples/gallery/lines/roads.py unexpectedly failed to execute correctly:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/lines/roads.py", line 19, in <module>
        gdf = gpd.read_file(
              ^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 294, in _read_file
        return _read_file_pyogrio(
               ^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/geopandas/io/file.py", line 547, in _read_file_pyogrio
        return pyogrio.read_dataframe(path_or_bytes, bbox=bbox, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/geopandas.py", line 261, in read_dataframe
        result = read_func(
                 ^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pyogrio/raw.py", line 196, in read
        return ogr_read(
               ^^^^^^^^^
      File "pyogrio/_io.pyx", line 1239, in pyogrio._io.ogr_read
      File "pyogrio/_io.pyx", line 219, in pyogrio._io.ogr_open
    pyogrio.errors.DataSourceError: No driver registered.

[seisman]

pyogrio.errors.DataSourceError: No driver registered.

This error comes from GDAL https://github.com/OSGeo/gdal/blob/a66c71d0e1e19a5ef9d94b33bef90074173be9c2/gcore/gdaldataset.cpp#L3965C7-L3965C7

[weiji14]

geopandas v1.0.0 was released a few hours ago and now our Docs workflow fails to build (https://github.com/GenericMappingTools/pygmt/actions/runs/9559533732/job/26634813440) which is not caught in our "GMT Dev Tests" workflow (why?)

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used? I'm reading https://github.com/geopandas/pyogrio/blob/v0.9.0/docs/source/install.md#troubleshooting-installation-errors, and it says errors like pyogrio.errors.DataSourceError: Could not obtain driver ... may be due to incompatible versions of GDAL in fiona and pyogrio, but we don't have fiona installed anymore 😕

Your example at #3301 (comment) works for me too if I run it individually. But when I do cd docs; make clean; make all to run the sphinx build, it fails the first time, but works if I run it twice. Looks like there's some flaky behaviour going on.

[seisman]

The "GMT Dev Tests" installs geopandas from PyPI, whereas the Docs workflow installs geopandas from conda-forge. Are there differences in GDAL version used?

The "Tests" workflow also has geopandas from conda-forge, but the geopandas tests all pass.

[weiji14]

Found a related issue upstream at geopandas/pyogrio#144. There is supposed to be some logic to register GDAL drivers when pyogrio is imported (geopandas/pyogrio#145), but it is not working properly in Sphinx build for some reason, and causing that pyogrio.errors.DataSourceError: Could not obtain driver: OGR_GMT (check that it was installed correctly into GDAL) error. Note that the OGR_GMT driver does exist in our GDAL installation, just that the driver isn't loaded properly.

Workaround is to manually register the drivers like so:

import pyogrio

pyogrio.core._register_drivers()

See also https://github.com/GenericMappingTools/pygmt/pull/3305/files#r1669745089 where this patch is applied.

[weiji14]

Have reported this upstream, and got a super helpful response at geopandas/pyogrio#448 (comment). It seems like there may be some conflict with the order of how rioxarray/rasterio and pyogrio is loading the GDAL drivers.

[weiji14]

Hmm, hitting into this issue again, but this time on rasterio. Traceback from https://github.com/GenericMappingTools/pygmt/actions/runs/12128509215/job/33815018072#step:7:956 at #3667:

Unexpected failing examples (2):

    ../examples/gallery/maps/tilemaps.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/maps/tilemaps.py", line 37, in <module>
        fig.tilemap(
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 610, in new_module
        return module_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/helpers/decorators.py", line 773, in new_module
        return module_func(*bound.args, **bound.kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/pygmt/src/tilemap.py", line 145, in tilemap
        raster = raster.rio.reproject(dst_crs="OGC:CRS84")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/raster_array.py", line 444, in reproject
        dst_affine, dst_width, dst_height = _make_dst_affine(
                                            ^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/raster_array.py", line 146, in _make_dst_affine
        dst_affine, dst_width, dst_height = rasterio.warp.calculate_default_transform(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/env.py", line 413, in wrapper
        return f(*args, **kwds)
               ^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/warp.py", line 556, in calculate_default_transform
        dst_affine, dst_width, dst_height = _calculate_default_transform(
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "rasterio/_warp.pyx", line 744, in rasterio._warp._calculate_default_transform
      File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset
      File "rasterio/_err.pyx", line 360, in rasterio._err.exc_wrap_pointer
    SystemError: Unknown GDAL Error. To debug: https://rasterio.readthedocs.io/en/latest/topics/errors.html#debugging-internal-gdal-functions

    ../examples/gallery/images/rgb_image.py failed leaving traceback:

    Traceback (most recent call last):
      File "/home/runner/work/pygmt/pygmt/examples/gallery/images/rgb_image.py", line 24, in <module>
        with rioxarray.open_rasterio(
             ^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rioxarray/_io.py", line 1128, in open_rasterio
        riods = manager.acquire()
                ^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 193, in acquire
        file, _ = self._acquire_with_cache_info(needs_lock)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
        file = self._opener(*self._args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/env.py", line 463, in wrapper
        return f(*args, **kwds)
               ^^^^^^^^^^^^^^^^
      File "/home/runner/micromamba/envs/pygmt/lib/python3.12/site-packages/rasterio/__init__.py", line 356, in open
        dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.__init__
    rasterio.errors.RasterioIOError: No driver registered.

Thinking if this has something to do with https://github.com/rasterio/rasterio/releases/tag/1.4.3, possibly due to changes in rasterio/rasterio#3260?