feat: multi-cell support — trigger once, run in every cell

[rogercampos]

What

DataDrip can now coordinate the same backfill or script across several independent deployments ("cells"): a developer triggers it once, from any cell's UI, and DataDrip creates and executes an independent copy of the run in every cell — with the UI honestly reflecting that results differ per cell.

Fully opt-in and backwards compatible: with DataDrip.current_cell_id / cell_transport unset (the defaults), behavior is exactly as today. New columns are nullable, the new table starts empty, and the Cell API rejects everything until tokens are configured — hosts can bump the gem and run the migrations with zero behavior change, then enable multi-cell purely via configuration.

How it works

developer's browser ──▶ Coordinator cell (local run + 1 CellDispatch per target cell)
                             │
                             │  CellDispatcherJob      POST create   ─▶  Remote cell's Cell API
                             │  CellStatusRefreshJob    GET status    ─▶  creates and runs its own
                             │  header actions          POST stop / DELETE   local run, reports back
                             ▼
                        page renders from the snapshots cached on the dispatch rows

• Runs are correlated by group_uuid; creation is idempotent per (group_uuid, cell_id) (unique index + find-or-return), so dispatch retries and duplicate deliveries are safe.
• Execution stays strictly cell-local — each cell's Dripper/ScriptRunner, queue, and data. Only control traffic crosses cells.
• No request ever makes a cross-cell call. Pages render from cached per-cell snapshots and a background job refreshes what has gone stale, so a page never waits on another cell and never writes while serving a GET (hosts commonly route reads to a replica where writing is forbidden).

Requirements it asks of a host

Both hold in most cell architectures, and the README now states them up front:

1. Backfiller ids are not reused between cells. A fanned-out run stores the creator's id verbatim in every cell, and that is what a cell's Cell API compares its own runs against. Per-cell AUTO_INCREMENT offsets, UUIDs or snowflake ids satisfy this; a shared sequence per cell does not. The record itself only has to exist in the coordinator's cell — others display the name snapshot taken at creation.
2. Cells can reach each other's Cell API, and run a background queue. Dispatch and status refreshes are jobs on DataDrip.queue_name.

Changes

Config & transport (lib/)

• DataDrip.current_cell_id, cell_ids, cell_transport, cell_api_tokens, cell_ui_url — plain values or callables; DataDrip.multi_cell? gates every new code path.
• Tuning knobs, all settings rather than constants because 8 cells and a 5s deadline are guesses about somebody else's deployment: cell_fanout_concurrency, cell_fanout_deadline, cell_status_refresh_interval, script_output_tail_bytes.
• DataDrip::CellTransport::Http — dependency-free HTTPS/JSON transport (configurable url/query/headers); any object with the same call contract can replace it. Its error contract: CellTransport::Error means "this cell is unusable", covering unreachable and "not in my registry" / "I refuse to send a secret over plaintext" — errors raised by a host's url callable are converted, so a misconfigured cell is a retriable dispatch failure rather than a crash.
• DataDrip::CellClient — knows the Cell API paths and always sends the intended target_cell_id.
• DataDrip::CellFanout — the one way anything talks to several cells: a bounded pool under a single shared deadline, so one hung cell costs the caller its deadline once rather than once per cell. Used by both the status refresh and the stop/delete fan-out.

Cell API (second engine: DataDrip::CellApi::Engine)

• Separate mountable so hosts can keep the human UI behind their staff gate while the machine-to-machine API sits outside it (bearer token only, constant-time compare, rotation via token array).
• Target-cell echo check → 421 Misdirected Request if the routing layer delivered to the wrong cell.
• POST /v1/backfill_runs & /v1/script_runs (idempotent create), GET /v1/groups/:group_uuid (status snapshot), stop / retry_failed_batches / delete re-applying the existing owner-only and history-preservation rules.
• Every mutation is scoped to this cell's own fanned-in leg (origin: :remote, this cell_id): a valid token cannot reach a run somebody created in this cell's own UI. Rejected auth and every mutation leave a structured log line — a stopped or deleted run should have a local explanation.

Group state

• DataDrip::MultiCellGroup owns everything that spans cells. A run row only ever describes its own cell, so asking it alone reported a group as "completed" while another cell was still working or had failed outright; the lists and page headers now show the group's worst-of status. A dispatch that never landed counts as a failure of the group; a cell never reached leaves the group unfinished rather than lowering its status; a status this version does not recognise (a cell on a newer release) is surfaced as-is and never read as finished.
• Each dispatch row caches the leg's last snapshot, which is what lets the run lists report a real group status without fanning out — one query per page, not two per row. Legs whose run reached a terminal state are never polled again, so a finished group costs nothing to view and stays readable after those cells are decommissioned; a cell that stops answering keeps rendering what it last reported, flagged stale.
• Several people watching the same run do not each enqueue the same fan-out: one request claims the refresh per interval via Rails.cache (deliberately not a DB flag — this is a read path). No shared cache simply means no deduplication.

Dispatch

• GroupCreator commits the local run and its dispatch rows in one transaction, then enqueues. A run executing here with no record that other cells were meant to run it is worse than no run at all, and the run's after_commit :enqueue now fires only once the dispatches are durable.
• Failure semantics: 4xx (validation, class not deployed yet) → marked failed, waits for the Retry dispatch button; 5xx / network → marked failed and re-raised so the host's queue retry policy applies. The recorded reason is refreshed on every attempt, since the operator deciding whether to retry needs the latest one.

UI

• "Where to run" on both new-run forms: All cells / Only this cell / Choose cells (current cell always included).
• Run lists: group status, N cells badge on coordinator runs, from <cell> badge on fanned-in runs.
• Show pages: per-cell cards with status, progress, errors, failed-batch retry, stop, a bounded tail of the script log, and optional deep links into the owning cell's UI. Cards distinguish "delivered, waiting for first report", "unreachable (showing state as of …)" and "dispatch failed". Polling self-schedules after each response and continues while any leg may still change.
• Delete is all-or-nothing. It used to report unreachable cells in a flash and destroy the run and its dispatch rows anyway, leaving a run enqueued elsewhere with nothing able to see or stop it. It now requires every cell to acknowledge; a leg that already ran (409) counts, since it will not run again either way.
• A fanned-in run is manageable by any operator in the cell executing it. Such a run has no owner there — the id it carries belongs to a record in the coordinator's cell — and requiring ownership left it unstoppable from the one cell that could stop it, with no break-glass if the coordinator was down. Ownership still decides whenever there is an owner to ask about.
• Script log output travels as a bounded tail (script_output_tail_bytes, 4KB) rather than up to ScriptRun::OUTPUT_LIMIT per cell per refresh, with a link into the owning cell for the rest.

Identity

• backfiller_name snapshot now also on script runs; remote cells store the coordinator's backfiller_id verbatim without requiring a local record — belongs_to is optional with a local-existence validation for local-origin runs only.

Schema & generators

• group_uuid / cell_id / origin / origin_cell_id on both run tables (+ unique (group_uuid, cell_id)), new data_drip_cell_dispatches table carrying each leg's cached state.
• rails generate data_drip:add_multi_cell for existing installs; install templates updated for fresh ones.
• Generated migration versions are real timestamps again: strftime(...).to_i + n rolls seconds past 59 (…120059 + 3 → …120062, not a datetime). Fixed for all four generated migrations, not only the new one.

Testing

• 433 examples, 0 failures (136 new): transport (webmock, incl. timeouts, refusals and unaddressable cells), Cell API request specs (auth, rotation, echo check, idempotency, validation, scoping to fanned-in legs, stop/retry/delete rules, output trimming), CellFanout (shared deadline, real concurrency, the configured cap, falsy-vs-failure), MultiCellGroup (worst-of ordering, unknown statuses, settled caching, unreachable retention, single-query preload), the dispatcher and refresh jobs, GroupCreator including transactional rollback, and controller specs for targeting, per-cell cards, refresh requests, retry-dispatch, and stop/delete fan-out.
• Rubocop clean; compiled Tailwind CSS regenerated and rake data_drip:css_check green.

Review notes

The commits are meant to be read in order: the feature, then a round of review fixes, then a host-agnosticism pass. The middle and last commits carry their reasoning in their messages, including two things worth a second opinion — treating an unacknowledged delete as a refusal, and letting any operator manage a fanned-in run.

🤖 Generated with Claude Code