@@ -0,0 +1,142 @@
+# Java connector work items (`datafusion-java`)
+
+Handoff for the connector-side pieces of the ADBC partition fast-path plan. The
+**driver-side** work (`adbc-driver-datafusion`, Rust) is done — landed on the driver's
+`feat/execute-partitions` branch. This file covers only the items that live in the Spark
+connector repo (`datafusion-java`) and cannot land in the driver.
+
+## What already landed upstream (driver, Rust)
+
+Context for the connector work below — already merged on the driver's
+`feat/execute-partitions` branch:
+
+- **Proto descriptors (always on).** `execute_partitions` always serializes the physical
+  plan into each descriptor; `read_partition` deserializes it and executes the given
+  partition index — no re-planning. A `PhysicalExtensionCodec` can be registered alongside
+  `ContextInit` (`DataFusionDriver::new_with_context_init_and_codec`) to cover custom
+  `ExecutionPlan` nodes; without one, the default codec covers built-in nodes. A node that
+  neither codec can encode fails the query (no Substrait/SQL fallback path).
+- **Per-connection plan cache.** `read_partition` deserializes a given plan once per
+  distinct plan-bytes and reuses it across partition indices (cap 16 plans, cleared on
+  overflow). The win is realized **only if the connector shares one connection across an
+  executor's task slots** — which is exactly Part 1 below.
+- **`partition_mode` option** (`datafusion.partition_mode` ∈ `auto`/`single`/`multi`,
+  default `auto`). Controls shuffle handling: `auto`/`single` collapse a shuffling plan to
+  one coalesced partition; `multi` keeps the natural partition count and warns. Opaque to
+  the connector — it only changes how many descriptors come back.
+- **Descriptor format.** Opaque to the connector. Internally:
+  `[u8 version=1][u32 LE index][proto plan bytes]`. No encoding discriminant — proto is the
+  only encoding. The connector must keep treating it as an opaque `byte[]`; do not parse it.
+
+---
+
+## Part 1 — per-executor connection cache
+
+Connector-only. Independently valuable; ships without any further driver dependency.
+**This is the prerequisite that makes the driver's per-connection plan cache pay off.**
+
+### Problem
+
+`AdbcColumnarPartitionReader` opens its own `AdbcDatabase` + `AdbcConnection` in every task.
+On one executor JVM, the 8 (or however many) task slots each repeat driver load, provider
+registration, and session setup against the same cdylib. The driver manager already de-dups
+the native library load; the expensive part — session construction and the `ContextInit`
+provider registration — is repeated per task.
+
+### Approach
+
+Cache the expensive ADBC objects per executor JVM, keyed by what fully determines them: the
+driver path/manifest plus the sorted database-option map (everything in
+`AdbcOptions.driverParameters()`). Two JVM-lifetime statics:
+
+```
+static final ConcurrentHashMap<Key, AdbcDatabase>   DATABASES
+static final ConcurrentHashMap<Key, AdbcConnection> CONNECTIONS   // see thread-safety
+```
+
+`Key` = (driver, immutable copy of the database-option map). `AdbcOptions` is already
+`Serializable` and rides to the executor inside `AdbcInputPartition`, so the key is
+reconstructable there with no new wire fields.
+
+### Thread-safety — decide the sharing level
+
+ADBC connections are not guaranteed thread-safe across concurrent statement execution; this
+varies by driver. Two levels:
+
+1. **Safe default — cache the `AdbcDatabase`, open a `Connection` per task.** Shares the
+   costly driver + database init; each task still gets its own short-lived connection +
+   statement. Lowest risk.
+2. **Further — cache the `AdbcConnection` too**, with one `AdbcStatement` per task. Only if
+   the DataFusion driver documents a connection as safe for concurrent statements.
+
+**Driver fact (verified):** `ContextInit` runs **per database**
+(`new_database_with_database_opts` in the driver's `src/lib.rs`), not per connection. So
+**level 1 already removes the provider-registration cost** — caching the `AdbcDatabase` is
+sufficient for the Part 1 win. Level 2 only adds value for the driver's per-connection plan
+cache, which needs a *shared connection* to dedup plan deserialization across task
+slots. Pursue level 2 only if the driver's connection is confirmed safe for concurrent
+statements; otherwise stay at level 1 and accept per-task plan deserialization.
+
+### Lifecycle
+
+Spark executors are long-lived; JVM-lifetime statics would leak connections across many jobs.
+Mitigate:
+
+- Register a shutdown/cleanup that closes cached handles. Spark `TaskCompletionListener` is
+  per-task — wrong scope; prefer a JVM shutdown hook / `ShutdownHookManager` entry, or a small
+  idle-eviction cache.
+- Bound the cache (size or idle TTL) so distinct queries/options do not accumulate handles
+  without limit.
+- Close order: statements → connection → database → release the `RootAllocator`, matching the
+  existing reader's cleanup ordering.
+
+### Touch points
+
+- `AdbcColumnarPartitionReader` (constructor): fetch-or-create the cached database/connection
+  instead of opening unconditionally.
+- New small holder class (e.g. `AdbcConnectionPool`) for the statics, the `Key`, and cleanup.
+- No descriptor/wire changes; no driver changes.
+
+### Tests
+
+- Unit: two readers with identical options share one cached database/connection; differing
+  options do not.
+- Concurrency: M threads build readers against one key concurrently → exactly one database
+  created (and at level 2, one connection), no races.
+- E2E (`AdbcSourceTest`, driver-gated): existing multi-partition assertions still pass; add a
+  counter/log assertion that provider registration ran once per executor, not once per task.
+
+---
+
+## Sequencing
+
+1. **Part 1** — per-executor connection cache (level 1: cache `AdbcDatabase`). Ships now;
+   removes per-task connection/provider-setup duplication and enables the driver's plan cache.
+
+The descriptor-shrink follow-up (ship the plan once, descriptors reference it) was
+**abandoned**: it cannot stay within ADBC's self-contained-partition contract. The full plan
+rides each of the N descriptors by design; keep descriptors opaque and ship them whole.
+
+## Risks
+
+- **R1 — connection thread-safety. RESOLVED: level 2 is unsound; ship level 1.** Verified by
+  reading the full call path. The arrow-adbc Rust FFI exporter (`adbc_ffi 0.23`
+  `driver_exporter.rs`, which our cdylib links) reaches the inner connection on *every* C call
+  via `connection_private_data`, which casts the shared `private_data` pointer to a
+  `&mut ExportedConnection` with **no lock** (`connection_read_partition` → `try_connection(&mut
+  self)` → `connection.read_partition(...)`). Even though `Connection::read_partition` is `&self`
+  at the trait level, the wrapper manufactures a `&mut` to the same connection object on each
+  call, so two task threads sharing one connection alias `&mut` → undefined behavior, regardless
+  of the driver being internally `Sync`. No layer above serializes: the Java `JniConnection`
+  holds no lock (bytecode-confirmed), the C driver manager is a thin dispatch, and ADBC's
+  contract says connections are not thread-safe. Serializing calls in Java would defeat the
+  parallelism that motivates sharing. **Decision:** default to level 1 (cache `AdbcDatabase`,
+  per-task connection); keep level 2 behind `-Dadbc.pool.shareConnection=true` only for
+  experiments / a future fixed exporter, with a loud one-time warning. The driver's
+  per-connection plan cache then dedups only within a task (one partition per task = one
+  deserialize anyway) — correct, just no cross-slot plan-cache win.
+- **R2 — cache lifecycle leaks.** JVM-lifetime statics on long-lived executors must be
+  bounded/evicted and closed on shutdown, or they leak native handles.
+- **R4 — descriptor size ×N.** Proto descriptors carry the full plan per partition (no shrink
+  is coming). The driver logs when a single proto descriptor exceeds 8 MiB; watch for it on
+  plans that inline data or very large queries.
@@ -0,0 +1,146 @@
+# Partitioning: DataFusion table providers → Spark tasks
+
+How DataFusion partitions become Spark tasks through the `adbc-datafusion`
+connector, how to size them, and what a `TableProvider` author should do. Pairs
+with the connector docs under `examples/adbc-datafusion-driver/`
+(`PARTITIONS.md`, `UPSTREAM_PARTITIONS_PLAN.md`).
+
+## The mapping is 1:1, and the provider owns it
+
+```
+provider.scan() output partitions
+   → DataFusion physical plan partition count  N
+   → executePartitioned()  =  N descriptors
+   → N Spark InputPartitions  =  N Spark tasks (scan stage)
+```
+
+For a pushed scan (projection / filter / limit only, no DataFusion-side join or
+aggregation), the physical plan's output partitioning **is** the provider's
+`scan()` partitioning. So **the provider's `scan()` partition count is the Spark
+task count.** Spark never splits or merges a partition afterward; the cluster's
+core count only bounds how many of the `N` run concurrently (the rest run in
+waves). `df.rdd().getNumPartitions() == N`.
+
+## The Spark-task sweet spot
+
+- **Floor:** `N` ≥ total executor cores, or cores sit idle. ~2–4× cores gives
+  slack for skew and stragglers.
+- **Per-partition size:** large enough to amortize per-task overhead. Canonical
+  target ≈ **128–256 MB** of data (Spark's `spark.sql.files.maxPartitionBytes`
+  default is 128 MB), or ≥ ~100 ms–seconds of work.
+- **Ceiling — and ADBC makes it lower.** Beyond Spark's generic per-task
+  scheduling cost, this connector adds two per-partition costs: each ADBC
+  partition descriptor carries the serialized query bytes (duplicated ×N on the
+  driver), and **each task re-plans the whole query** in `read_partition`. So
+  thousands of partitions is far costlier here than for a native Spark file
+  source. Keep `N` in the hundreds, not tens of thousands; prefer bigger
+  partitions over a huge count.
+
+Rule of thumb: `N ≈ clamp(total_bytes / target_bytes, floor, ceiling)`, balanced
+by **bytes**, not by split count.
+
+## The model mismatch you must account for
+
+DataFusion and Spark size scans differently:
+
+- **Spark native file source — byte-bound.** `N = Σ ceil(file_size / 128MB)`.
+  Large data ⇒ many partitions, `N ≫ cores`, run in waves; each task ≈ 128 MB.
+- **DataFusion scan — parallelism-bound.** `ListingTable` packs files into
+  **`target_partitions` groups** (≈ machine cores), with
+  `repartition_file_min_size` (~10 MB) as a floor and optional single-file
+  splitting to *reach* `target_partitions`. So DataFusion's default
+  `N ≈ target_partitions`, **not** `bytes / 128MB`.
+
+Consequence: if a provider mimics DataFusion's default (`N = target_partitions`)
+and the connector sets `target_partitions = defaultParallelism` (≈ cores), then
+**large data → ~cores giant tasks** (`total/cores` each) → memory pressure and
+poor sizing. For feeding Spark you usually want the **byte-bound** model — more
+partitions than cores for big data.
+
+**So a custom provider feeding Spark should size by bytes, not blindly use
+`target_partitions` as the literal partition count.**
+
+## Provider recipe
+
+`scan(state, …)` receives the session. Read the connector-supplied hint, then
+decide by what the provider knows:
+
+```text
+T            = state.config().target_partitions()   // connector hint ≈ desired parallelism
+                                                     // (set from SparkContext.defaultParallelism)
+target_bytes = ~128–256 MB                           // bias larger for ADBC to keep N down
+min_bytes    = floor so partitions never get tiny    // ~ tens of MB
+
+if total_bytes known AND splittable (files / row groups / key ranges):
+    N = clamp(ceil(total_bytes / target_bytes), 1, CEILING)        // byte-bound
+    bin-pack splits into N groups BALANCED BY BYTES                // not by count
+    do not split below min_bytes; merge small splits
+
+elif split_count known but not bytes (shards / external partitions):
+    N = min(split_count, CEILING)                                  // natural partitioning
+    if split_count >> T: coalesce shards into ~T balanced groups
+
+else (size and splits unknown; opaque / streaming):
+    N = T                                                          // fall back to the hint
+    lean slightly high — stragglers hurt more than overhead
+```
+
+- **Balance by bytes, not split count.** Spark can't rebalance a fat partition
+  (each task is atomic). One 10 GB partition among 100 small ones stalls the
+  whole stage. Bin-pack splits so partition bytes are even.
+- `T` (the connector's `target_partitions`) is the **hint / cap / fallback** —
+  not the data-bound count. Use it when bytes are unknown, or as a parallelism
+  ceiling. For known data, **bytes drive `N`**.
+
+## Determinism caveat (specific to the ADBC partition path)
+
+`execute_partitions` counts `N` on the **driver**; `read_partition` re-plans on
+the **executor**. Partition `index i` must mean the same slice both times, so
+**`scan()` must produce the identical `N` in both places.**
+
+- If the provider keys `N` off `target_partitions`, the driver **pins it in the
+  descriptor** and `read_partition` re-plans with it held fixed → determinism for
+  free. (That is why the pin exists.)
+- If the provider sizes by **bytes** (ignoring `target_partitions`), the pin does
+  not help — compute `N` from **stable** inputs (a cached file listing / a fixed
+  stats snapshot). Otherwise stat drift between driver-plan and executor-read
+  changes `N` and shifts partitions.
+
+## Connector knobs
+
+- **`target_partitions`** (implemented): defaults to
+  `SparkContext.defaultParallelism` (total executor cores). The connector issues
+  `SET datafusion.execution.target_partitions = N` on the planning session; the
+  driver pins it into each descriptor, so executors re-plan identically. Reaches
+  the provider via `state.config()`. Optionally pass `k×cores`.
+- **`batch_size`** (follow-up, connector-only): would `SET
+  datafusion.execution.batch_size` on the executor connection in the reader;
+  `read_partition` inherits it via `copied_config()`. Affects Arrow batch
+  granularity, not partition count, so no pinning needed.
+- **`maxPartitionBytes`**: not a DataFusion session config and the connector
+  cannot derive it (it does not know the data size). Byte-sizing lives in the
+  provider's `scan()`; the connector can only forward it as a provider-specific
+  option.
+- **Reported partitioning / storage-partitioned joins
+  (`SupportsReportPartitioning`)**: not viable over ADBC today — the descriptor
+  carries no distribution metadata, and DataFusion's hash partitioning does not
+  match Spark's key-grouped/`bucket` model. See `UPSTREAM_PARTITIONS_PLAN.md`.
+
+## Recommendation (summary)
+
+- **Connector:** keep `target_partitions` defaulted from `defaultParallelism`; it
+  is the parallelism hint that reaches `scan()` and is pinned for determinism.
+- **Provider (the real lever):**
+  - bytes known + splittable → **byte-bound**: `N = ceil(total_bytes/target_bytes)`,
+    bin-packed by bytes, merged above `min_bytes`, capped well below "thousands"
+    (ADBC re-plans per partition); compute `N` from a stable stat snapshot.
+  - only splits known → natural partitioning, coalesce toward `T` if huge.
+  - nothing known → `N = T`, lean slightly high.
+  - default `target_bytes ≈ 128–256 MB`; raise it (fewer, bigger partitions)
+    before letting `N` reach the thousands.
+
+One line: **the provider's `scan()` partition count is the Spark task count; size
+it by bytes toward ~128–256 MB balanced partitions, floored at executor cores and
+capped well below thousands because ADBC re-plans per partition — and make that
+count a deterministic function of a stat snapshot so the driver plan and executor
+reads agree.**