performance: optimize per-service memory allocations

[howardjohn]

Before we allocate

• 1312 bytes for eviction mpsc. We replace this with a lighter mpsc. see
  Excessive per-connection memory cause by channel hyperium/hyper#4057 for more context
• 512 bytes for another part of the same mpsc; remoevd
• 512 bytes for the tokio task; we now build it lazily

New: 320 bytes per service (87% reduction)

5k services before/after:

Signed-off-by: John Howard john.howard@solo.io

[Copilot]

Pull request overview

This PR reduces per-service memory overhead in the load balancer by replacing the eviction event channel implementation and making the eviction worker task start lazily on first eviction, instead of eagerly at EndpointSet construction.

Changes:

• Replaced the eviction event channel from tokio::sync::mpsc to futures::channel::mpsc with a smaller buffer.
• Introduced EvictionWorkerState and EvictionStarter to lazily spawn the eviction worker only when needed.
• Updated eviction scheduling to ensure the worker is started before sending eviction events, and simplified maybe_sleep_until by using pending() when no timer is needed.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[stevenctl]

lgtm, for my own knowledge when is it appropriate to use one mpsc impl or the other?

[howardjohn]

lgtm, for my own knowledge when is it appropriate to use one mpsc impl or the other?

Before yesterday I thought: always use tokio. But tokio has a 32xsizeof<T> cost so I suppose whenever T is large + the number of MPSC is large? The downside is that the futures one is not an intrusive linked list so it does an additional heap alloc per message; in this case we do not care about throughput though as eviction is rare.