Currently we implement GDAL reading via OGR using pyogrio's Python interface. This leads to some Python calls that would be nice to avoid...among other things, when we have Python in multiple places in the plan (e.g., udfs, writes that use Python) we can get a hang. Even though we can probably fix those hangs otherwise, moving GDAL reads and/or writes into Rust means we can get them for free in SQL and R.
Roughly, we need to implement the Arrow interface here:
|
/// An OGR layer (borrowed from a Dataset). |
|
pub struct Layer<'a> { |
|
api: &'static GdalApi, |
|
c_layer: OGRLayerH, |
|
_dataset: PhantomData<&'a Dataset>, |
|
} |
...and wire up constructing the dataset and layer like pyogrio does. This is pretty involved, and handles the predicate pushdown (bbox, "ignored fields", limit).
https://github.com/geopandas/pyogrio/blob/main/pyogrio/_io.pyx#L1908-L1929
We currently implement the ExternalFormatSpec via Python...we just need to implement the ExternalFormatSpec directly in Rust.
Currently we implement GDAL reading via OGR using pyogrio's Python interface. This leads to some Python calls that would be nice to avoid...among other things, when we have Python in multiple places in the plan (e.g., udfs, writes that use Python) we can get a hang. Even though we can probably fix those hangs otherwise, moving GDAL reads and/or writes into Rust means we can get them for free in SQL and R.
Roughly, we need to implement the Arrow interface here:
sedona-db/c/sedona-gdal/src/vector/layer.rs
Lines 30 to 35 in e3f57ff
...and wire up constructing the dataset and layer like pyogrio does. This is pretty involved, and handles the predicate pushdown (bbox, "ignored fields", limit).
https://github.com/geopandas/pyogrio/blob/main/pyogrio/_io.pyx#L1908-L1929
We currently implement the ExternalFormatSpec via Python...we just need to implement the ExternalFormatSpec directly in Rust.