refactor(core): Slice C — pure-Rust http::serve(Service<Request<Body>>); FfiDispatcher moves to mod ffi

# Summary

**Slice C of #182.** The big one. Refactor `serve()` to expose a pure-Rust signature: `http::serve(endpoint, opts, svc: S) -> ServeHandle` where `S: Service<Request<Body>, Response = Response<Body>, Error = Infallible>`. The current callback-shaped, handle-allocating `FfiDispatcher` becomes one specific `Service` implementation living in `ffi/dispatcher.rs`, handed to `http::serve` like any other service.

This slice is the one that flips "the FFI is the API" into "the FFI is one consumer of the API". After this, a Rust application can call `iroh_http_core::http::serve(my_tower_service)` without ever touching `u64`.

## Evidence

- `crates/iroh-http-core/src/server.rs:243-260` — `IrohHttpService` exists but is not the user-facing entry point; `serve(endpoint, opts, F)` where `F: Fn(RequestPayload)` is.
- `server.rs` is 979 LoC; the bulk is `FfiDispatcher::dispatch` (handle allocation, oneshot rendezvous, callback firing) interleaved with HTTP dispatch logic.
- `endpoint/mod.rs` is 747 LoC because so much of it is FFI plumbing that needs the endpoint state.
- Per-connection `remote_node_id` is mutated via `conn_conc.get_mut().remote_node_id = …` — see #177.
- Four ad-hoc `Drop` guards inline in `serve_with_events` — see #178.
- Dead duplex `CONNECT`/`Upgrade` branch in `FfiDispatcher::dispatch` since `raw_connect` was dropped — see #180.

## Remediation

1. Define the pure-Rust `serve` API:
   ```rust
   // http/server/mod.rs
   pub fn serve<S>(endpoint: Endpoint, opts: ServeOptions, svc: S) -> ServeHandle
   where
       S: Service<Request<Body>, Response = Response<Body>, Error = Infallible>
         + Clone + Send + Sync + 'static,
       S::Future: Send,
   { ... }
   ```
   No callbacks, no handles in the signature.
2. Move `FfiDispatcher` to `ffi/dispatcher.rs`. It becomes one specific implementation of the above `Service` trait — the one that allocates handles, fires the JS callback, and awaits a `oneshot::Receiver<ResponseHeadEntry>`.
3. Add `ffi::serve_with_callback(endpoint, opts, cb: Fn(RequestPayload))` that constructs the dispatcher and calls `http::serve(endpoint, opts, dispatcher)`. JS adapters call this; pure-Rust callers call `http::serve` directly.
4. `lib.rs` re-exports `ffi::serve_with_callback` as `serve` for FFI binary compatibility (no FFI-side breaking change).
5. Thread `remote_node_id` via a request extension inserted by a small per-connection `MapRequestLayer`, not as a mutable service field. Closes #177.
6. Extract the four inline `Drop` guards (`PeerConnectionGuard`, `ReqHeadCleanup`, `TotalGuard`, `ReqGuard`) into typed lifecycle objects (`ConnectionTracker`, `RequestTracker`) owned by the per-connection task. Closes #178.
7. Delete the dead duplex `CONNECT`/`Upgrade` branch in `FfiDispatcher::dispatch` (~80 LoC) — `raw_connect` was dropped in `8754da4`, sessions use their own ALPN. Closes #180.
8. Move `respond()` into `ffi/dispatcher.rs` — it is the partner of the dispatcher's `oneshot` rendezvous and has no place in `mod http`.
9. After this slice, `crates/iroh-http-core/src/http/server/mod.rs` should be the axum-shaped accept loop — target ≤ 200 LoC, with the rest in `pipeline.rs` + `stack.rs`.
10. **Dissolve `crates/iroh-http-core/src/endpoint/` into `crates/iroh-http-core/src/endpoint.rs`** at the crate root. Per-file disposition:
    - `endpoint/ffi_bridge.rs` → constructor / wiring folds into `ffi/handles.rs` (or a dedicated `ffi/` module if it grows).
    - `endpoint/http_runtime.rs` and `endpoint/transport.rs` (16 LoC) → fold into `endpoint.rs` or `http/transport/`, whichever owns the data.
    - `endpoint/session_runtime.rs` → fold into `http/session.rs`. If any piece is session-specific in a way that only Slice E (#187) can resolve cleanly, it may stay in `endpoint.rs` until that slice; document the carve-out inline.
    - `endpoint/mod.rs` (`IrohEndpoint` facade) → becomes `endpoint.rs` at the crate root. The folder is deleted.
11. **Dissolve the remaining root-level files** that are not in the canonical tree:
    - `events.rs` → `http/events.rs` (producers live in `http/transport/pool.rs` and the endpoint; `ffi → http` import direction is preserved).
    - `registry.rs` → `ffi/registry.rs` (FFI-only — only adapter crates touch it).
    - `stats.rs` types (`EndpointStats`, `NodeAddrInfo`, `PeerStats`, `ConnectionEvent`) → fold into `endpoint.rs`.
    - `config.rs` remainder (`NetworkingOptions`, `DiscoveryOptions`) → fold into `endpoint.rs`. (`CompressionOptions` already moved in Slice B — see #184.)
12. After this slice, `crates/iroh-http-core/src/` directly contains exactly four entries: `lib.rs`, `endpoint.rs`, `http/`, `ffi/`. Slice E (#187) extends `tests/architecture.rs` to assert this.

## Acceptance criteria

1. `http::serve(endpoint, opts, svc)` is callable from a pure-Rust integration test with a hand-rolled `Service<Request<Body>, Response<Body>, Error = Infallible>`. The test sends a request via `http::fetch` (or hyper directly) and asserts a response.
2. `ffi::serve_with_callback(endpoint, opts, cb)` reproduces today's JS-facing behaviour exactly (verified by all current adapter tests).
3. `IrohHttpService` (today's structure) is replaced by `FfiDispatcher` in `ffi/dispatcher.rs`. There is no service in `mod http` that knows about handles or callbacks.
4. `remote_node_id` is a request extension, not a service field. (Closes #177.)
5. Connection / request lifecycle is a typed object, not inline Drop guards. (Closes #178.)
6. Dead duplex CONNECT/Upgrade branch is gone. (Closes #180.)
7. `http/server/mod.rs` ≤ 200 LoC.
8. `crates/iroh-http-core/src/endpoint/` directory does not exist. `crates/iroh-http-core/src/endpoint.rs` exists and contains the `IrohEndpoint` facade plus the types listed in remediation step 10–11.
9. `crates/iroh-http-core/src/{config,events,registry,stats}.rs` no longer exist. Their contents are redistributed per remediation step 11. `lib.rs` re-exports the public symbols from their new paths so adapter callsites are unchanged.
10. `crates/iroh-http-core/src/` directly contains exactly: `lib.rs`, `endpoint.rs`, `http/`, `ffi/`. (Architecture-test enforcement lands in Slice E #187.)
11. `npm run ci` green; 92 interop pairs pass.

## Subsumes

- #177 (remote_node_id as request extension) — close as part of this slice.
- #178 (extract ConnectionTracker / RequestTracker) — close as part of this slice.
- #180 (delete dead duplex CONNECT/Upgrade branch) — close as part of this slice.

## Notes

This is the riskiest slice. Recommend landing as multiple sub-PRs:
- C.1: introduce `http::serve(svc)` signature; today's `serve(endpoint, opts, F)` becomes a thin wrapper that internally constructs the dispatcher and calls `http::serve`. No code moves yet.
- C.2: move `FfiDispatcher` into `ffi/dispatcher.rs`; rename wrapper to `ffi::serve_with_callback`.
- C.3: remote_node_id extension (closes #177).
- C.4: ConnectionTracker / RequestTracker (closes #178).
- C.5: delete dead duplex branch (closes #180).
- C.6: dissolve `endpoint/` directory into `endpoint.rs`; redistribute `events.rs`, `registry.rs`, `stats.rs`, and the `config.rs` remainder per remediation steps 10–11.

Each is independently revertable. Stop signal still applies (ADR-013): if any sub-PR takes more than ~2 compile iterations fighting tower types, stop and revisit the design in #182.

## References

- Epic: #182
- Subsumes #177, #178, #180
- Supersedes #176 (the cosmetic-only subsystem split — close as part of this slice)



---

## Carry-forward from Slice B (#184)

Slice B left the layer composition as one `ServiceBuilder` + one terminal
`boxed_clone()`. That blocks two things Slice C wants:

- a real `cfg.decompression` toggle (today: always-on, with a
  doc-comment in `StackConfig` explaining why),
- a `pub(crate) fn build_decompression_layer()` factory reusable on the
  client side (today: inlined in `build_stack`).

Root cause: `impl Layer<ServeService> + Clone` as a return type erases
the `<L::Service as Service<Request<Body>>>::Future: Send + 'static`
bound that `boxed_clone()` needs further down. `Either<Decomp, NoOp>`
then fails `Service<_>` and the entire chain refuses to box.

**Fix in this slice** — adopt axum's boxed-per-layer pattern. Each
factory takes a `ServeService` and returns a `ServeService`, doing its
own `boxed_clone()`:

```rust
let svc = apply_decompression(svc);
let svc = apply_compression(svc, &cfg.compression);
let svc = apply_timeout(svc, cfg.timeout);
let svc = apply_load_shed(svc, cfg.load_shed);
let svc = apply_body_limit(svc, cfg.max_request_body_bytes);
apply_handle_layer_error(svc)
```

Cost: one extra `BoxCloneService` allocation per layer per connection
(construction-time only, never per request). Benefit: every layer
factory is `pub(crate)` and reusable from `build_client_stack`,
`cfg.decompression` becomes a real `option_layer` toggle, and
`AddExtensionLayer` (#177), connection/request trackers (#178), trace
layer compose without type-soup gymnastics.

This is the runtime form of ADR-014 D2's "type-erased seam" — D2 only
required erasure at the outer boundary; this extends it to every layer.


Navigation Menu

refactor(core): Slice C — pure-Rust http::serve(Service<Request<Body>>); FfiDispatcher moves to mod ffi #185

Summary

Evidence

Remediation

Acceptance criteria

Subsumes

Notes

References

Carry-forward from Slice B (#184)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

refactor(core): Slice C — pure-Rust http::serve(Service<Request<Body>>); FfiDispatcher moves to mod ffi #185

Description

Summary

Evidence

Remediation

Acceptance criteria

Subsumes

Notes

References

Carry-forward from Slice B (#184)

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions