Hermes Agent plugin for fail-closed, session-derived custom request headers across named local and OpenAI-compatible providers.
-
Updated
Jul 29, 2026 - Python
#
session-affinity
Here are 2 public repositories matching this topic...
One OpenAI-compatible endpoint in front of three vLLM replicas. Pins each conversation to the same GPU so its KV cache stays warm, sends cheap work to the small tier, and tracks token budgets per agent.
python portfolio prometheus load-balancing fastapi opentelemetry kv-cache rendezvous-hashing llmops llm-serving vllm session-affinity inference-gateway
-
Updated
Aug 10, 2026 - Python
Add this topic to your repo
To associate your repository with the session-affinity topic, visit your repo's landing page and select "manage topics."