Replace Pingora proxy with custom tonic-based service

[SantiagoPittella]

We want to remove the dependency on Pingora and replace it with our own gRPC service built with tonic.
This new service will maintain shared state across "workers" and handle request delegation.
The service should be fully compatible with the existing ProverApi interface, so clients do not need to be aware of which worker is actually handling the request.

Tasks

• Core service implementation

  • Create a tonic service that:
    • Maintains shared state of worker availability.
    • Queues requests and assigns them to free workers.
    • Exposes the same API (ProverApi) as the workers.

• Status endpoint

  • Add an endpoint to expose the overall service status.

• Rate limiting

  • Add configurable rate limiting to protect the workers from overload.

• Health checks

  • Implement worker health checks to automatically detect and exclude unhealthy workers.

• Worker management

  • Add functionality to dynamically add or remove workers.

• Metrics, logging, and tracing

  • Integrate metrics collection, structured logs, and tracing for observability.

• Self-reflection

  • Add an endpoint for self-reflection / introspection (e.g., list workers, queue length, recent activity).

[SantiagoPittella]

@Mirko-von-Leipzig @bobbinth if you guys agree with the "roadmap" I can start implementing it.

[Mirko-von-Leipzig]

I think the roadmap looks good ❤️

What I would like to see before the implementation begins is an initial design of the core of the service. As an example, do we need tokio style multi-threading or can we get away with something more basic and lightweight.

[SantiagoPittella]

Prover Proxy (tonic-based) | Orchestrator

Overview

The orchestrator maintains shared state across workers, queues requests, delegates to available workers, and exposes the same ProverApi surface so clients are unaware of which worker fulfills the request.

Goals

• Maintain compatibility with existing ProverApi without client changes.
• Centralize scheduling, backpressure, and health management.
• Provide clear observability (metrics, logs, tracing) and admin endpoints.
• Keep the implementation simple, safe, and easy to evolve.

Concurrency Model Decision

Do we need Tokio multi-threading?

Yes. Our workload is I/O-bound with some lightweight CPU usage. The default multi-threaded Tokio runtime (with work-stealing) should be enough since it provides a fair task scheduling for concurrent RPCs, background health-checks, and queue management.

We will use a single Tokio process with the default multi-threaded runtime and avoid manual OS threads. No custom thread pools should be required

High-Level Architecture

• Prover Orchestrator gRPC Service (tonic): Implements the same ProverApi service as workers. Receives requests, enqueues them if necessary, and delegates to workers.
• Worker Registry: Shared, in-memory state of workers (address, status, last-seen). Supports dynamic add/remove and health-based exclusions.
• Request Queue: Bounded FIFO queue providing backpressure. When a worker becomes available, the scheduler assigns the next request.
• Scheduler: Matches queued requests to free workers, enforces simple fairness, and propagates deadlines/cancellations.
• Health Checker: Periodic probes (lightweight RPC) to detect unhealthy workers and prevent assignment.
• Rate Limiting: Global limiter initially; extensible to per-IP limits. Could be implemented via tower layers.
• Status & Introspection: gRPC endpoints for status, list of workers, queue depth.
• Observability: tracing spans, logs, and metrics via metrics/prometheus exporters.

Component Diagram (or at least an initial version)

graph TD
  C[Clients] -->|ProverApi| O[Orchestrator]
  O -->|Delegate RPC| W1[Worker 1]
  O -->|Delegate RPC| W2[Worker 2]
  O -->|Delegate RPC| Wn[Worker n]
  O -->|Admin gRPC| A[Admin/Introspection]
  O -->|Metrics| M[Prometheus/OTel]

Loading

Scheduling & Delegation

Flow

1. Receive a ProverApi call.
2. Try queue_tx.try_send(...).
   • On full queue: return RESOURCE_EXHAUSTED.
3. A scheduler task pulls from queue_rx and selects a healthy worker with available capacity.
4. Orchestrator forwards the RPC to the worker using a pooled tonic::transport::Channel.
5. On completion/failure, update worker, record metrics, and proceed to next queued request.

Worker Selection

• Prefer worker with the lowest usage, skipping unhealthy and "in-use" workers.
• If no worker is available, briefly wait (e.g., tokio::time::sleep(5-10ms)), re-check, and assign if possible.

Backpressure

• Bounded queue size Q. When full, fail fast with RESOURCE_EXHAUSTED.

Health Checks

• Periodic task (e.g., every 1s-5s) to call a lightweight HealthCheck RPC on each worker, similar to what Pingora does.
• Mark as unhealthy if the worker is not responding to the health check.
• Mark back healthy if it is responding to the health check.

Rate Limiting

• We can use a global limiter using tower layers.
• On limit exceeded: return RESOURCE_EXHAUSTED.

Status & Introspection Endpoints

• Add gRPC GetStatus returning: uptime, queue length, healthy/unhealthy counts, and build/version info.
• Add admin gRPC service for self-reflection.
• Add admin methods to add/remove workers.

[bobbinth]

Putting this on hold for now as well.

[Mirko-von-Leipzig]

I'm thinking about this again in the context of others running proving services.

Formal proving services will have their own proxy setups, load-balancing etc all in house or via the cloud, and its unlikely and even undesirable for us to try implement something that can generically manage this. What they will initially want is our proving worker, at least initially.

I propose we tackle most of the deferred proving issues with a gentle rewrite/refactor which simplifies things.

Proposal

We keep the existing proving worker and nuke everything. The downside of this is that now we no longer provide a way to queue requests. We instead add this functionality back with a --queue N (or some better name), where --queue=0 is the same as we currently have for a raw worker.

This can somewhat trivially be added to the worker with a permit/semaphor system. A channel to the actual "worker" thread also works except that cancelled requests in-queue cannot be trivially "removed" from the queue again.

i.e. system starts with N+1 permits, each request acquires a permit which is then made available again when the proof completes.

This also drops support for multiple provers and multiple workers, but I don't think we should try maintain that functionality.

---

I may start working on a PoC because yanking pingora will be cathartic.