Repository navigation
slm(monitoring): wire performance_benchmark in as a fleet benchmark, and fix its import of a module that does not exist #16305
Description
Activity
performance_benchmark.py:27importsVMSfrommonitoring/performance_monitor.py.VMSis a static host map built fromNetworkConstants(main, frontend, npu-worker, redis, ai-stack, browser). The benchmark uses it for its API endpoint URLs (:140-164) and its per-host latency loop (:463-468). A fleet benchmark should get its targets from the SLM's node registry, which replaces this import along with thesrc.constantsone.(Edited after review: #16308 now keeps
performance_monitor.py. It is removed by whichever of #16305, #16306 and #16307 lands last, together with its file-size baseline entries and the twologs/performance_monitor.logchecks inautobot-infrastructure/shared/scripts/logging/quick-deploy-verification.sh.)- added a commit that references this issue
on Sep 11, 2026 The design needs a decision before any code. I read this from base
d5a0c6dcband executed nothing.What
performance_benchmark.pydoes today- API benchmark (
:129). For the whole run (120 s in comprehensive mode) it keeps hitting the main backend: GETs to/api/health,/api/system/statusand/api/chats, and POSTs to/api/knowledge_base/search(:158) and/api/llm/request(:164). That last one runs real LLM inference for as long as the benchmark runs. - Database benchmark (
:238). It writesbenchmark_key_{0..999}into Redis DBs 0/1/4/7/8 (:382-385) and never deletes them. Those are the production databases. - Network benchmark (
:456). Pings each host in a staticVMSmap, from wherever the benchmark runs. - System benchmark (
:541). CPU, memory, disk (a 50 MB temp file),nvidia-smiand OpenVINO, all on the machine running it. From the SLM, that benchmarks the SLM host, not the nodes. - Other. It imports
matplotlibandnumpyfor charts (:25-26); neither is an SLM dependency. Its imports don't resolve:src.constants, andperformance_monitor, which needs the main backend'sConfigManager.
What the SLM has to work with
- No command channel to agents.
- Per-node work goes through the allowlisted SSH execute (
api/nodes_execution.py:620), the playbook executor, or a probe from the SLM. - Results can live in
Node.extra_data, following theextra["npu"]pattern (api/npu.py:54-62). There is no generic node-job table.
Options
- A. Probe from the SLM only. For each registered node, measure the network latency from the SLM to the node, plus HTTP latency to the node's health endpoints. Store the latest result in
extra_data["benchmark"], and expose it through an authenticated route and a panel in the Performance view.- Mutates nothing on the nodes.
- Doesn't measure node hardware.
- B. A, plus the system benchmark on each node. CPU, memory, disk and GPU run on the node itself, through the playbook executor, using a self-contained benchmark shipped with the agent. That gives real hardware numbers per node.
- It loads the node, so it runs on demand only.
- It's refused while the node is in maintenance or busy.
- C. Port the module as it is. Not viable: it runs unbounded LLM inference, and every run leaves 5 x 1000 junk keys in production Redis.
Recommendation: B.
- Drop: the LLM and knowledge-search load test, the Redis write benchmark, and the charts (the GUI renders the numbers).
- Keep: a Redis round-trip latency probe (PING, or a GET on a key with a TTL), if Redis latency is wanted.
- Imports: resolve them against the SSOT config and the node registry.
- Storage: the latest result per node, in
extra_data["benchmark"]. History can come later if wanted. - Entry point: an authenticated route behind service management, plus a per-node "Run benchmark" action in the Performance view, with i18n in every locale.
- Test: one end-to-end run against a fake node, through TestClient.
Please pick A, B, or a variant. #16305 is on hold until then. #16306 (removing the optimizer) doesn't depend on this, and goes ahead after #16308.
- API benchmark (
Owner decision, 11 Sep 2026: option B, latency probes plus an on-node run.
- The SLM probes each node's service latency from the SLM side.
- A real system benchmark (CPU, disk, memory) runs on each node through the existing playbook executor. The SLM has no agent command channel, and the playbook executor is the sanctioned path to a node.
- Results are stored per node, in
extra_data["benchmark"], and served from a route guarded like the other SLM admin routes (_SM). They are shown from the SLM Performance view, which gets an action to run a benchmark. - Dropped: the LLM-inference load test (it POSTed to
/api/llm/requestfor the whole run, i.e. real inference) and the Redis-write load test (1,000 keys written into each of DBs 0/1/4/7/8 and never deleted). - The module's unresolvable
src.constantsimport goes with the port.
- added a commit that references this issue
on Sep 13, 2026
Child of #16282. Owner decision, 11 Sep 2026: wire it in, don't retire it.
autobot-slm-backend/monitoring/performance_benchmark.pyhas no live equivalent, and it can't be imported: it importssrc.constants, which exists nowhere in the repository.Acceptance criteria