Repository navigation
refactor(backend): move the main backend's remaining nvidia-smi / rocm-smi parsers onto autobot_shared.gpu_telemetry #16289
Description
Activity
I've split AC 2. Moving the parsers keeps each caller's behaviour, so
GPUCollectorparses rows separately and still returns the first GPU. Getting every GPU to the consumers means changing five single-GPU sites, two of them in files at their size ceilings. That's a different risk class, so it now lives in #16297 (blocked by this issue). I agreed this with the reviewer during #16292.- added 6 commits that reference this issue
on Sep 11, 2026 - added 6 commits that reference this issue
on Sep 13, 2026
#16280 adds
autobot_shared/gpu_telemetry.py, which holds nvidia-smi and rocm-smi parsing in one place, and moveshardware_acceleration.pyonto it. The main backend's other call sites still spawn and parse the tools themselves, each with its own idea of a missing value.Call sites (base 65609f9, tests excluded)
autobot-backend/diagnostics.py:135autobot-backend/worker_node.py:201autobot-backend/utils/performance_monitoring/hardware.py:28autobot-backend/utils/performance_monitoring/collectors.py(GPUCollector):111autobot-backend/utils/hardware_metrics.py:226,:312autobot-backend/utils/gpu_optimization/gpu_detection.py:57,:236:77,:269,:281GPUCollector.collect(collectors.py:139) splits the tool's entire multi-line output on commas. On a multi-GPU host it builds metrics from the first GPU's row alone and never reports the others.The SLM's
monitoring/modules have their own copies too. #16282 covers those.Acceptance criteria
autobot_shared.gpu_telemetry(query_nvidia_gpus/run_vendor_tool/parse_rocm_smi_json).GPUCollectorparses each GPU's row separately and still returns the first. A two-GPU test, with the second GPU named differently, pins that the first GPU's metrics are intact and that no field of the second row leaks into them. Reporting every GPU to the consumers is feat(backend): the performance monitor reports only one GPU; its five consumers assume a single GPUMetrics #16297.Refs #15180, #16280, #16282.