nvProbe

NVIDIA GPU & CUDA Benchmark Suite
Automate CUDA workloads • HPL & HPCG • MLPerf inference • Custom kernels • Interactive reports
nvprobe.scszero.com

pip install nvprobe && nvprobe setup && nvprobe run --local

---

Features

Bandwidth	MatMul / Attention	Conv2D
H2D / D2H / D2D across buffer sizes	fp32, fp16, int8 custom CUDA kernels	2D convolution benchmarks
HPL (FP64 Linpack)	HPCG	MLPerf Inference
Datacenter GPUs: A100, H100, B200, L40S…	Conjugate Gradients	ONNX Runtime via cmx4mlperf
Memtest	Burn
VRAM integrity: solid, checkerboard, random, walking-1 patterns	Sustained clock stability, thermal/power throttling detection

• Bundled CUDA runtime — CuPy [ctk] via pip, no system toolkit required
• Auto-downloaded HPC tools — NVIDIA HPC Benchmarks cached in ~/.nvprobe/tools/
• Interactive HTML reports — Chart.js charts with GPU / transfer / precision dropdowns and oscilloscope-style glow
• GPU diagnostics — max clocks, power limits, ECC state, PCIe link speed, NVLink active links captured per run
• Memory integrity — memtest with solid, checkerboard, random, and walking-1 patterns to detect VRAM faults
• Thermal soak — sustained burn test with time-series clock/temp/power tracking and throttling detection
• A/B comparison — compare two result sets side-by-side
• Slurm integration — generate, submit, monitor, collect from HPC clusters
• SQLite storage — all results persisted; CSV / JSON export

Quick Start

Step	Command	What it does
1	pip install nvprobe	Install the package
2	nvprobe setup	Install CuPy, download HPL/HPCG, generate configs
3	nvprobe env	Verify GPU detection, driver, CUDA version
4	nvprobe run --local	Run all benchmarks locally
5	nvprobe report --open	Generate & open interactive HTML report

Or from source:

git clone https://github.com/SergioZ3R0/nvprobe.git && cd nvprobe
pip install -e . && nvprobe setup && nvprobe run --local

More commands

Command	Description
nvprobe compare --a results/run1 --b results/run2	Compare two runs
nvprobe run --config configs/cluster.yaml	Run with custom YAML config
nvprobe slurm submit --config configs/cluster.yaml	Submit Slurm job
nvprobe slurm status	Check Slurm job status
nvprobe setup --cuda 13	Setup with specific CUDA version

Charts

Chart.js canvas-based charts with interactive controls:

• Bandwidth — filter by GPU and transfer type (H2D / D2H / D2D)
• MatMul / Attention — filter by GPU and precision (fp32 / fp16)
• Burn — time-series line chart of SM/MEM clocks and temperature over sustained load
• Memtest — bar chart of detected VRAM errors per test pattern
• Range slider — zoom into any x-axis region
• Moving average — smoother trend lines for dense data

YAML Config

name: my-run
gpu:
  models: ["L40S", "B200"]
slurm:
  enabled: true
  partition: gpu
  gpus_per_node: 8
precisions: [fp32, fp16]
benchmarks:
  - name: bandwidth
    params:
      sizes_mb: [1, 4, 16, 64, 256, 1024]
  - name: custom
    params:
      kernels: [matmul, attention]

Project Structure

nvprobe/
├── nvprobe/
│   ├── cli.py                     # CLI entry point
│   ├── config.py                  # YAML config loader
│   ├── runner.py                  # Benchmark orchestration
│   ├── slurm.py                   # Slurm job management
│   ├── reporter.py                # Plotly HTML report generator
│   ├── db.py                      # SQLite storage + CSV/JSON export
│   └── benchmarks/
│       ├── base.py                # Base class, GPU detection, diagnostics
│       ├── bandwidth.py           # Memory bandwidth tests
│       ├── burn.py                # Sustained burn / throttling detection
│       ├── custom.py              # Custom CUDA kernels
│       ├── hpl.py                 # HPL wrapper
│       ├── hpcg.py                # HPCG wrapper
│       ├── memtest.py             # VRAM integrity test
│       ├── mlperf.py              # MLPerf via cmx4mlperf
│       └── _cuda/                 # Raw CUDA kernels
├── configs/
│   ├── default.yaml
│   └── local.yaml
├── nvprobe.svg
├── index.html
├── README.md
└── pyproject.toml

Notes

• HPL / HPCG — NVIDIA HPC Benchmarks binaries are validated for datacenter GPUs (A100, H100, B200, L40S…). They may crash (SIGSEGV) on RTX series. Bandwidth and custom kernels work on any CUDA GPU.
• MLPerf cuDNN — mlcr discovers cuDNN via system CUDA paths. If installed via pip install nvidia-cudnn-cuXX, pre-register with: mlcr get,cudnn,nvidia --input=$(python3 -c 'import nvidia.cudnn; print(nvidia.cudnn.__path__[0]'))

Requirements

Python 3.10+ • NVIDIA GPU with CUDA drivers • nvidia-smi in PATH • Slurm (optional)

License

Apache License 2.0