@@ -6,6 +6,8 @@ rust-target/
 *.iml
 .DS_Store
 tpch-data/
+# Lockfile for the example ADBC driver crate (a demo cdylib, not a published lib).
+examples/adbc-datafusion-driver/Cargo.lock
 .claude
 docs/superpowers
 docs/build/
 
@@ -0,0 +1,146 @@
+# Partitioning: DataFusion table providers → Spark tasks
+
+How DataFusion partitions become Spark tasks through the `adbc-datafusion`
+connector, how to size them, and what a `TableProvider` author should do. Pairs
+with the connector docs under `examples/adbc-datafusion-driver/`
+(`PARTITIONS.md`).
+
+## The mapping is 1:1, and the provider owns it
+
+```
+provider.scan() output partitions
+   → DataFusion physical plan partition count  N
+   → executePartitioned()  =  N descriptors
+   → N Spark InputPartitions  =  N Spark tasks (scan stage)
+```
+
+For a pushed scan (projection / filter / limit only, no DataFusion-side join or
+aggregation), the physical plan's output partitioning **is** the provider's
+`scan()` partitioning. So **the provider's `scan()` partition count is the Spark
+task count.** Spark never splits or merges a partition afterward; the cluster's
+core count only bounds how many of the `N` run concurrently (the rest run in
+waves). `df.rdd().getNumPartitions() == N`.
+
+## The Spark-task sweet spot
+
+- **Floor:** `N` ≥ total executor cores, or cores sit idle. ~2–4× cores gives
+  slack for skew and stragglers.
+- **Per-partition size:** large enough to amortize per-task overhead. Canonical
+  target ≈ **128–256 MB** of data (Spark's `spark.sql.files.maxPartitionBytes`
+  default is 128 MB), or ≥ ~100 ms–seconds of work.
+- **Ceiling — and ADBC makes it lower.** Beyond Spark's generic per-task
+  scheduling cost, this connector adds two per-partition costs: each ADBC
+  partition descriptor carries the serialized query bytes (duplicated ×N on the
+  driver), and **each task re-plans the whole query** in `read_partition`. So
+  thousands of partitions is far costlier here than for a native Spark file
+  source. Keep `N` in the hundreds, not tens of thousands; prefer bigger
+  partitions over a huge count.
+
+Rule of thumb: `N ≈ clamp(total_bytes / target_bytes, floor, ceiling)`, balanced
+by **bytes**, not by split count.
+
+## The model mismatch you must account for
+
+DataFusion and Spark size scans differently:
+
+- **Spark native file source — byte-bound.** `N = Σ ceil(file_size / 128MB)`.
+  Large data ⇒ many partitions, `N ≫ cores`, run in waves; each task ≈ 128 MB.
+- **DataFusion scan — parallelism-bound.** `ListingTable` packs files into
+  **`target_partitions` groups** (≈ machine cores), with
+  `repartition_file_min_size` (~10 MB) as a floor and optional single-file
+  splitting to *reach* `target_partitions`. So DataFusion's default
+  `N ≈ target_partitions`, **not** `bytes / 128MB`.
+
+Consequence: if a provider mimics DataFusion's default (`N = target_partitions`)
+and the connector sets `target_partitions = defaultParallelism` (≈ cores), then
+**large data → ~cores giant tasks** (`total/cores` each) → memory pressure and
+poor sizing. For feeding Spark you usually want the **byte-bound** model — more
+partitions than cores for big data.
+
+**So a custom provider feeding Spark should size by bytes, not blindly use
+`target_partitions` as the literal partition count.**
+
+## Provider recipe
+
+`scan(state, …)` receives the session. Read the connector-supplied hint, then
+decide by what the provider knows:
+
+```text
+T            = state.config().target_partitions()   // connector hint ≈ desired parallelism
+                                                     // (set from SparkContext.defaultParallelism)
+target_bytes = ~128–256 MB                           // bias larger for ADBC to keep N down
+min_bytes    = floor so partitions never get tiny    // ~ tens of MB
+
+if total_bytes known AND splittable (files / row groups / key ranges):
+    N = clamp(ceil(total_bytes / target_bytes), 1, CEILING)        // byte-bound
+    bin-pack splits into N groups BALANCED BY BYTES                // not by count
+    do not split below min_bytes; merge small splits
+
+elif split_count known but not bytes (shards / external partitions):
+    N = min(split_count, CEILING)                                  // natural partitioning
+    if split_count >> T: coalesce shards into ~T balanced groups
+
+else (size and splits unknown; opaque / streaming):
+    N = T                                                          // fall back to the hint
+    lean slightly high — stragglers hurt more than overhead
+```
+
+- **Balance by bytes, not split count.** Spark can't rebalance a fat partition
+  (each task is atomic). One 10 GB partition among 100 small ones stalls the
+  whole stage. Bin-pack splits so partition bytes are even.
+- `T` (the connector's `target_partitions`) is the **hint / cap / fallback** —
+  not the data-bound count. Use it when bytes are unknown, or as a parallelism
+  ceiling. For known data, **bytes drive `N`**.
+
+## Determinism caveat (specific to the ADBC partition path)
+
+`execute_partitions` counts `N` on the **driver**; `read_partition` re-plans on
+the **executor**. Partition `index i` must mean the same slice both times, so
+**`scan()` must produce the identical `N` in both places.**
+
+- If the provider keys `N` off `target_partitions`, the driver **pins it in the
+  descriptor** and `read_partition` re-plans with it held fixed → determinism for
+  free. (That is why the pin exists.)
+- If the provider sizes by **bytes** (ignoring `target_partitions`), the pin does
+  not help — compute `N` from **stable** inputs (a cached file listing / a fixed
+  stats snapshot). Otherwise stat drift between driver-plan and executor-read
+  changes `N` and shifts partitions.
+
+## Connector knobs
+
+- **`target_partitions`** (implemented): defaults to
+  `SparkContext.defaultParallelism` (total executor cores). The connector issues
+  `SET datafusion.execution.target_partitions = N` on the planning session; the
+  driver pins it into each descriptor, so executors re-plan identically. Reaches
+  the provider via `state.config()`. Optionally pass `k×cores`.
+- **`batch_size`** (follow-up, connector-only): would `SET
+  datafusion.execution.batch_size` on the executor connection in the reader;
+  `read_partition` inherits it via `copied_config()`. Affects Arrow batch
+  granularity, not partition count, so no pinning needed.
+- **`maxPartitionBytes`**: not a DataFusion session config and the connector
+  cannot derive it (it does not know the data size). Byte-sizing lives in the
+  provider's `scan()`; the connector can only forward it as a provider-specific
+  option.
+- **Reported partitioning / storage-partitioned joins
+  (`SupportsReportPartitioning`)**: not viable over ADBC today — the descriptor
+  carries no distribution metadata, and DataFusion's hash partitioning does not
+  match Spark's key-grouped/`bucket` model.
+
+## Recommendation (summary)
+
+- **Connector:** keep `target_partitions` defaulted from `defaultParallelism`; it
+  is the parallelism hint that reaches `scan()` and is pinned for determinism.
+- **Provider (the real lever):**
+  - bytes known + splittable → **byte-bound**: `N = ceil(total_bytes/target_bytes)`,
+    bin-packed by bytes, merged above `min_bytes`, capped well below "thousands"
+    (ADBC re-plans per partition); compute `N` from a stable stat snapshot.
+  - only splits known → natural partitioning, coalesce toward `T` if huge.
+  - nothing known → `N = T`, lean slightly high.
+  - default `target_bytes ≈ 128–256 MB`; raise it (fewer, bigger partitions)
+    before letting `N` reach the thousands.
+
+One line: **the provider's `scan()` partition count is the Spark task count; size
+it by bytes toward ~128–256 MB balanced partitions, floored at executor cores and
+capped well below thousands because ADBC re-plans per partition — and make that
+count a deterministic function of a stat snapshot so the driver plan and executor
+reads agree.**
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "adbc-datafusion-example-driver"
+version = "0.1.0"
+edition = "2021"
+publish = false
+description = "End-to-end example: an ADBC driver that exposes a custom DataFusion TableProvider"
+
+# Standalone crate that stands in for a real, separate driver repo. The empty
+# [workspace] table makes it its own workspace root so it is NOT pulled into the
+# parent datafusion-java workspace (whose members are listed explicitly).
+[workspace]
+
+[dependencies]
+# The ready-made ADBC-over-DataFusion driver. We depend on it as a LIBRARY and
+# only customize the SessionContext via its new_with_context_init hook -- no
+# fork, no datafusion-ffi module loading. Pulled from its own repo (git) since
+# that is where it lives.
+adbc-driver-datafusion = { git = "https://github.com/adbc-drivers/datafusion" }
+adbc_core = "0.23"
+adbc_ffi = "0.23"
+datafusion = "53"
+async-trait = "0.1"
+
+# Build both an rlib (so the provider/driver can be unit-tested or embedded) and
+# a cdylib (the artifact the arrow-adbc Java driver manager loads at runtime).
+[lib]
+crate-type = ["cdylib", "lib"]
+
+# Temporary: redirect adbc-driver-datafusion to the PR branch that implements
+# execute_partitions / read_partition (adbc-drivers/datafusion#32), so we can run
+# a full multi-partition system test before it merges. The crate is not on
+# crates.io, so this patches the git source (not [patch.crates-io]). Remove once
+# the PR is merged and the canonical dependency carries the change.
+[patch."https://github.com/adbc-drivers/datafusion"]
+adbc-driver-datafusion = { git = "https://github.com/timsaucer/adbc-driver-datafusion", branch = "feat/execute-partitions" }