feat: implement execute_partitions and read_partition

Cargo.toml

@@ -38,10 +38,12 @@ async-trait = "0.1"
 aws-config = "1.8"
 aws-credential-types = "1.2"
 datafusion = { version = "53", default-features = false, features = ["avro", "compression", "crypto_expressions", "datetime_expressions", "nested_expressions", "parquet", "recursive_protection", "regex_expressions", "sql", "unicode_expressions"] }
+datafusion-proto = { version = "53", default-features = false }
 datafusion-substrait = { version = "53", default-features = false }
 driverbase = { git = "https://github.com/adbc-drivers/driverbase-rs.git" }
 futures = "0.3.32"
 log = "0.4"
+lru = "0.18"
 object_store = { version = "0.13", features = ["aws", "gcp", "http"] }
 parking_lot = "0.12"
 prost = "0.14.3"

docs/datafusion.md

@@ -86,6 +86,26 @@ SELECT `Breed Name`, `Lifespan`
 
 {{ footnotes|safe }}
 
+### Partitioned Execution
+
+This driver supports ADBC's partitioned execution; one partition is generated per output partition. Each partition contains the physical plan, and can be distributed across CPUs or physical machines. If the plan contains a shuffle, however, distributed execution will likely be slower, as each partition must read the entire output.
+
+The statement option `datafusion.partition_mode` controls what to do if a shuffle is detected:
+
+| Value            | Behavior                                                                            |
+|------------------|-------------------------------------------------------------------------------------|
+| `auto` (default) | Collapse to a single partition when the plan shuffles, otherwise natural partitions |
+| `multi`          | One descriptor per natural output partition; logs a warning if the plan shuffles    |
+| `single`         | Always collapse to a single partition (effectively the same as regular execution)   |
+
+### Extending DataFusion
+
+It is possible to customize the embedded DataFusion by depending on this crate, then using hooks to customize the DataFusion `SessionContext` and register `PhysicalExtensionCodec`s. The customized driver can then be built into a shared library and distributed as an ADBC driver. For more, see:
+
+- `DataFusionDriver::new_with_context_init`, which accepts a callback to modify the `SessionContext`
+- `DataFusionDriver::with_codec`, which accepts a `PhysicalExtensionCodec` used to serialize/deserialize plans in partitioned execution
+- The [`adbc_ffi`](https://crates.io/crates/adbc_ffi) crate, which provides helpers to export a Rust ADBC driver as a shared library
+
 ## Compatibility
 
 {{ compatibility_info|safe }}