Skip to content

Commit

Permalink
Add read-only OGR ADBC (Arrow Database Connectivity) driver
Browse files Browse the repository at this point in the history
Cf https://arrow.apache.org/adbc/current/index.html for what ADBC is.

Depends on the adbc-driver-manager library.

The driver is read-only, and there is no support for spatial data currently.

Beyond official ADBC drivers (adbc-driver-sqlite,
adbc-driver-postgresql, adbc-driver-snowflake, adbc-driver-bigquery,
etc.), it can also be used to read Parquet or DuckDB datasets using libduckdb, if
libduckdb is installed and can be loaded through dynamic shared library opening.
  • Loading branch information
rouault committed Oct 15, 2024
1 parent d86dcdb commit eb8b8ad
Show file tree
Hide file tree
Showing 25 changed files with 1,807 additions and 4 deletions.
20 changes: 20 additions & 0 deletions .github/workflows/ubuntu_24.04/Dockerfile.ci
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,23 @@ RUN python3 -m pip install -U --break-system-packages -r /tmp/requirements.txt
# cfchecker requires udunits2
RUN apt-get install -y --allow-unauthenticated libudunits2-0 libudunits2-data
RUN python3 -m pip install --break-system-packages cfchecker

# Manually install ADBC packages from Ubuntu 22.04 as there are no 24.04 packages at time of writing.
RUN curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-manager102_14-1_amd64.deb \
&& curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-manager-dev_14-1_amd64.deb \
&& curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-sqlite102_14-1_amd64.deb \
&& curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-sqlite-dev_14-1_amd64.deb \
&& curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-snowflake102_14-1_amd64.deb \
&& curl -LO -fsS https://apache.jfrog.io/artifactory/arrow/ubuntu/pool/jammy/main/a/apache-arrow-adbc/libadbc-driver-snowflake-dev_14-1_amd64.deb \
&& dpkg -i libadbc-driver-manager102_14-1_amd64.deb \
&& dpkg -i libadbc-driver-manager-dev_14-1_amd64.deb \
&& dpkg -i libadbc-driver-sqlite102_14-1_amd64.deb \
&& dpkg -i libadbc-driver-sqlite-dev_14-1_amd64.deb \
&& dpkg -i libadbc-driver-snowflake102_14-1_amd64.deb \
&& dpkg -i libadbc-driver-snowflake-dev_14-1_amd64.deb

# Install libduckdb
RUN curl -LO -fsS https://github.com/duckdb/duckdb/releases/download/v1.1.2/libduckdb-linux-amd64.zip \
&& unzip libduckdb-linux-amd64.zip libduckdb.so \
&& mv libduckdb.so /usr/lib/x86_64-linux-gnu \
&& rm -f libduckdb-linux-amd64.zip
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ Supported Formats: (ro:read-only, rw:read-write, +:update, v:virtual-I/O s:subda
JSONFG -vector- (rw+v): OGC Features and Geometries JSON (*.json)
MiraMonVector -vector- (rw+v): MiraMon Vectors (.pol, .arc, .pnt) (*.pol, *.arc, *.pnt)
XODR -vector- (ro): OpenDRIVE - Open Dynamic Road Information for Vehicle Environment (*.xodr)
ADBC -vector- (ro): Arrow Database Connectivity
TIGER -vector- (rov): U.S. Census TIGER/Line
AVCBin -vector- (rov): Arc/Info Binary Coverage
AVCE00 -vector- (rov): Arc/Info E00 (ASCII) Coverage (*.e00)
Expand Down
8 changes: 4 additions & 4 deletions apps/test_ogrsf.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1655,19 +1655,20 @@ static int TestOGRLayerFeatureCount(GDALDataset *poDS, OGRLayer *poLayer,
}
delete poFeat;

const auto nFCEndOfIter = LOG_ACTION(poLayer->GetFeatureCount());
if (nFC != nClaimedFC)
{
bRet = FALSE;
printf("ERROR: Claimed feature count " CPL_FRMT_GIB
" doesn't match actual, " CPL_FRMT_GIB ".\n",
nClaimedFC, nFC);
}
else if (nFC != LOG_ACTION(poLayer->GetFeatureCount()))
else if (nFC != nFCEndOfIter)
{
bRet = FALSE;
printf("ERROR: Feature count at end of layer, " CPL_FRMT_GIB
", differs from at start, " CPL_FRMT_GIB ".\n",
poLayer->GetFeatureCount(), nFC);
nFCEndOfIter, nFC);
}
else if (bVerbose)
printf("INFO: Feature count verified.\n");
Expand Down Expand Up @@ -4187,8 +4188,7 @@ static int TestLayerGetArrowStream(OGRLayer *poLayer)
{
if (array.length != 0)
{
bRet = false;
printf("ERROR: get_next() return an array with length != 0 "
printf("WARNING: get_next() return an array with length != 0 "
"after end of iteration\n");
}
if (array.release)
Expand Down
Binary file added autotest/ogr/data/duckdb/poly.duckdb
Binary file not shown.
Loading

0 comments on commit eb8b8ad

Please sign in to comment.