geometry / geography: Read improvements #4079

Activity

1. rambleraptor

   added

   V3

   on Oct 5, 2026
2. quinncheong commented on Oct 6, 2026

   

   quinncheong

   on Oct 6, 2026

   More actions

   Two details from reproducing the first two items on 0.12.0 and on main (068aae5), in case they help scope the work. Line numbers are from main.

   Item 1 (unquoted form) also needs the emit side. Reading is only half of it: PyIceberg writes the quoted form, which is not the spec form ("geometry(srid:4326)", "geography(srid:4326, spherical)" in the JSON serialization table of format/spec.md):

   from pyiceberg.schema import Schema
   from pyiceberg.types import GeographyType, GeometryType, NestedField
   
   for t in ["geometry('srid:4326')", "geometry(srid:4326)", "geometry(EPSG:7415)", "geography(srid:4326, spherical)"]:
       try:
           print(f"{t!r:36} -> {NestedField(1, 'g', t, required=False).field_type!r}")
       except Exception as e:
           print(f"{t!r:36} -> {type(e).__name__}: {str(e).splitlines()[0]}")
   print("emitted:", GeometryType("srid:4326").model_dump_json(), GeographyType("srid:4326", "planar").model_dump_json())
   Schema.model_validate_json('{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"geom","required":false,"type":"geometry(srid:4326)"}]}')

   "geometry('srid:4326')"              -> GeometryType(crs='srid:4326')
   'geometry(srid:4326)'                -> ValidationError: Could not parse geometry(srid:4326) into a GeometryType
   'geometry(EPSG:7415)'                -> ValidationError: Could not parse geometry(EPSG:7415) into a GeometryType
   'geography(srid:4326, spherical)'    -> ValidationError: Could not parse geography(srid:4326, spherical) into a GeographyType
   emitted: "geometry('srid:4326')" "geography('srid:4326', 'planar')"
   pyiceberg.exceptions.ValidationError: Could not parse geometry(srid:4326) into a GeometryType
   

   Java's GEOMETRY_PARAMETERS pattern (Types.java lines 67-71 on apache/iceberg main) does not strip quotes, so it reads PyIceberg's output as CRS 'srid:4326' with the quotes included. Java writes geometry(%s) / geography(%s, %s) (lines 632 and 709). This matches the split @moomindani proposed on #3530: read side there, emit side as a follow-up.

   Item 2 is a hard error, not just plain binary. Because the file column resolves to BinaryType, _cast_if_needed calls promote(BinaryType, GeometryType), and that raises (pyiceberg/io/pyarrow.py line 2036, pyiceberg/schema.py line 1691). It happens on write as well as read. Without geoarrow-pyarrow, writing a binary or large_binary WKB column through PyIceberg's own writer fails like this:

   import tempfile, uuid
   import pyarrow as pa
   from pyiceberg.io.pyarrow import PyArrowFileIO, _dataframe_to_data_files
   from pyiceberg.partitioning import UNPARTITIONED_PARTITION_SPEC
   from pyiceberg.schema import Schema
   from pyiceberg.table.metadata import new_table_metadata
   from pyiceberg.table.sorting import UNSORTED_SORT_ORDER
   from pyiceberg.types import GeometryType, LongType, NestedField
   
   schema = Schema(NestedField(1, "id", LongType(), required=False), NestedField(2, "geom", GeometryType(), required=False))
   meta = new_table_metadata(schema, UNPARTITIONED_PARTITION_SPEC, UNSORTED_SORT_ORDER, "file://" + tempfile.mkdtemp(),
                             properties={"format-version": "3"})
   df = pa.table({"id": pa.array([1], pa.int64()),
                  "geom": pa.array([bytes.fromhex("0101000000000000000000f03f0000000000000040")], pa.large_binary())})
   list(_dataframe_to_data_files(table_metadata=meta, df=df, io=PyArrowFileIO(), write_uuid=uuid.uuid4()))

     File "pyiceberg/io/pyarrow.py", line 1937, in _to_requested_schema
     File "pyiceberg/io/pyarrow.py", line 2036, in _cast_if_needed
     File "pyiceberg/schema.py", line 1691, in _
   pyiceberg.exceptions.ResolveError: Cannot promote an binary to geometry
   

   Scanning a Parquet file that stores WKB as a plain BYTE_ARRAY column with the matching field id fails with the same ResolveError in ArrowScan.to_table. With geoarrow-pyarrow installed, a geoarrow.wkb input column is rejected earlier with UnsupportedPyArrowTypeException: Column 'geom' has an unsupported type: extension<geoarrow.wkb<WkbType>>. So as far as I can tell, no input form round-trips today. That differs from mkdocs/docs/geospatial.md, which says that without geoarrow-pyarrow "geometry and geography are written as binary in Parquet while the Iceberg schema still preserves the spatial type". An end-to-end write/read test for each of the three column forms would catch this. The existing geo tests in tests/io/test_pyarrow.py only cover the type conversion.

   I have a small patch for item 1, read and emit side. It makes the quotes optional in GEOMETRY_REGEX / GEOGRAPHY_REGEX, keeps accepting the quoted form so older type strings still parse, emits the unquoted form, and adds parametrized tests to tests/test_types.py (308 pass; the new tests fail on main). It does not yet do the case-insensitive matching. I'm happy to open it as the emit-side follow-up once #3530 settles the read side, or to leave it to @moomindani if that's already in progress.

   This report was prepared with an AI coding agent (Claude Code) and reproduced before posting by me.
3. cayetanobv

   added 2 commits that reference this issue on Oct 9, 2026

   Make DuckDB 1.5.6 the tested version throughout; drop 1.5.5 references

   ...

   fb407f3

   Re-verify on DuckDB 1.5.6 (V3 geometry pruning, L3); pyiceberg 0.12.0…

   ...

   91e93b8