iOS foreground restart can leave HTTP server half-live and unreachable

[momics]

Problem

On iOS/Tauri mobile, an HTTP-serving node can become permanently unreachable after the app is backgrounded and later foregrounded, even after manually starting the HTTP server again.

Observed sequence:

1. Run the iroh-http example app on desktop and on an iOS device.
2. On the iOS device, start the HTTP server.
3. From the desktop app, fetch from the iOS node. This works.
4. Background the iOS app.
5. After some time, the iOS server becomes unreachable from desktop. This is expected; iOS suspending/killing background networking is not the bug.
6. Confirm from desktop that fetch no longer works.
7. Reopen the iOS app.
8. The HTTP tab is no longer active, and the iOS app shows the HTTP server as not running.
9. On iOS, go back to the HTTP tab and start the HTTP server again.
10. From desktop, fetch the iOS node again.
11. The desktop client still cannot reach the iOS server. Instead of failing quickly with a connection error, the fetch hangs until a long timeout.
12. Force quit the iOS app.
13. As soon as the iOS app is force quit, the desktop fetch fails promptly with a connection error.
14. Relaunch the iOS app, start the HTTP server again, and fetch from desktop again.
15. Fetch works again.

Expected behavior:

• It is acceptable that the iOS node becomes unreachable while the app is backgrounded.
• After reopening the app and starting the HTTP server again, the node should either become reachable again or fail deterministically with a clear transport/endpoint error.
• The desktop client should not remain stuck on stale state until a long timeout when the iOS app has returned but the server is not actually reachable.
• Force quitting the iOS app should not be the only way to make desktop observe the connection as gone and recover on the next fresh start.

Root Cause Analysis

The current mobile foreground recovery appears to distinguish only whether the native endpoint handle still exists, not whether the underlying Iroh transport, UDP socket, relay path, mDNS advertisement, or HTTP accept loop is usable after iOS suspension.

The public serve restart path starts a new HTTP accept loop on the existing node. It does not rebuild the underlying Iroh endpoint. If iOS invalidates the socket or transport while Rust and JS still retain the endpoint handle, the app can remain in a half-live state: the UI can show the server as stopped, restarting serve can appear to succeed, but remote peers still cannot establish working connections.

On the desktop/client side, outbound fetch can reuse cached QUIC/path state until the transport observes a close. The fact that force quitting the iOS app causes the desktop fetch to fail promptly suggests the desktop still has stale connection/path state to the iOS node that is not invalidated while the resumed iOS app is half-live.

Contributing factors:

• Endpoint registry liveness and transport liveness are currently too easy to conflate.
• The mobile lifecycle probe checks for endpoint-handle existence, which can pass even when the transport is unusable.
• Restarting serve() is not equivalent to recreating the Iroh endpoint/socket.
• There is no explicit iOS foreground recovery path that marks the old transport as suspect, tears it down, recreates the endpoint with the same identity/options, restarts serve/discovery, and clears stale state.
• Stale pooled client connections can present as long timeouts instead of fast connection failures.

TDD Fix Plan

1. RED: Write a mobile lifecycle test where a simulated foreground event occurs while the endpoint handle still exists but a transport health probe fails. Assert that the node is marked dead or recovery is triggered.
   GREEN: Replace the registry-only foreground probe with a health contract that can fail when the underlying endpoint/transport is unusable.

2. RED: Write a Tauri guest-JS test for the reconnect policy: after foreground health fails, the node closes deterministically and active serve state/promises do not remain half-running.
   GREEN: Ensure lifecycle recovery clears serve state and resolves/rejects serve lifecycle promises consistently.

3. RED: Write an integration-style test for close -> recreate -> serve with the same supplied key/options. Assert the recreated node keeps the same public key and accepts fetches from a peer using fresh address information.
   GREEN: Add or document a recovery path that recreates the endpoint using the original node options and restarts application serve handlers, or expose lifecycle signaling that lets apps do this reliably.

4. RED: Write a client-side stale-connection test where a cached connection/path becomes unusable without an immediate close reason. Assert fetch is bounded by the configured timeout and the next attempt uses fresh connection state.
   GREEN: Invalidate stale pooled connections after timeout/open-stream failures where appropriate, and make mobile guidance require bounded request timeouts.

5. RED: Write a mobile discovery/advertising lifecycle test that simulates native advertiser/browser cancellation and restart across background/foreground. Assert stale advertisements/address-lookup entries do not survive incorrectly.
   GREEN: On recovery, stop/restart native advertisements/browses or clear address lookup entries when the endpoint is rebuilt.

REFACTOR: Split "endpoint handle exists" from "transport is usable" in the public lifecycle vocabulary. Keep low-level primitives explicit, then layer automatic mobile recovery on top so application behavior remains predictable.

Acceptance Criteria

• iOS foreground recovery does not report a node as healthy solely because its endpoint handle exists.
• After background-induced transport loss, apps can deterministically close and recreate the node without killing the app process.
• Restarting serve on a half-dead endpoint either restores reachability or fails clearly; it must not silently leave remote fetches hanging beyond the configured timeout.
• Recreated nodes using the same supplied key retain the same public key and become reachable again from another peer.
• Desktop/client stale pooled state is invalidated or bounded after the iOS peer becomes half-live/unreachable.
• Stale discovery/address-lookup state is bounded or cleared during recovery.
• New tests pass.
• Existing lifecycle, serve, fetch, and Tauri adapter tests still pass.

[momics]

Investigation update from branch codex/ios-lifecycle-diagnostics (6dd2ff8).

What we changed for diagnostics / mitigation:

• Updated the Tauri example lockfile to Tauri 2.11.x.
• Added example diagnostics: boot counter/id, visibility/page events, endpoint stats, transport diagnostics (pool:hit, pool:miss, pool:evict), path changes, and peer stats before/after fetch.
• Added a 10s request timeout in the example fetch path so the desktop client stops hanging indefinitely during repro.
• Added core behavior: on fetch timeout or transport failure, evict and close the cached pooled QUIC connection for that peer. Regression test: pool_evicts_connection_after_fetch_timeout.
• Added Tauri adapter behavior: when create_endpoint is called with a key whose node id already has a live native endpoint in this Tauri process, replace the old native endpoint and close_force() it. Regression test: test_create_endpoint_replaces_existing_same_node_id. This mirrors the previous lifecycle pattern from momics/tauri-plugin-http-server, where a JS/WebView restart can leave native server state alive.

Observed behavior after the latest change:

• Desktop stale pool handling works: a fetch on the old pooled connection times out, emits pool:evict, and the next fetch uses pool:miss rather than staying glued to the dead connection.
• iOS recovery improved materially: after WebView refresh / boot increment, manually starting the HTTP server again makes desktop fetch work without force-quitting the iOS app.
• There is still an ordering race on foreground: one pending or immediate request may succeed while native networking resumes, then JS/WebView refresh catches up and the UI/server state is reset. After that, fetch times out until serve() is registered again.

Current interpretation:

The original unrecoverable state looks like a combination of two issues:

1. Desktop reused a pooled QUIC connection that had not observed closure after iOS background suspension. This is mitigated by evicting the pool entry on timeout/transport failure.
2. On iOS, Tauri/WebView JS can reboot while Rust/native endpoint state survives. The new JS boot reuses the persisted key, so without native replacement there can be stale same-identity endpoint ownership. This is mitigated by replacing same-node-id Tauri endpoints on create_endpoint.

Remaining work:

• Decide whether the example/app should persist "was serving" and automatically re-register serve() after a WebView boot/foreground reload.
• Research whether Tauri/iOS exposes a better lifecycle hook for WebView content process termination / resumed events so the library can document or improve this behavior.
• Add docs for Tauri mobile: after iOS WebView reload, JS serve handlers must be re-registered; native endpoint replacement now prevents the old endpoint from blocking recovery.

Validation run locally:

• cargo test -p iroh-http-core --test http_pool
• cargo check -p iroh-http-core
• cargo test in packages/iroh-http-tauri
• cargo check in examples/tauri/src-tauri
• npm run build in examples/tauri

Branch: https://github.com/momics/iroh-http/tree/codex/ios-lifecycle-diagnostics