Official Docker image missing `bandwidthTest` binary for mem-bw benchmark in offline environments

[NePixe1]

What's the issue, what's expected?:
Issue: When running the mem-bw benchmark using the official Docker image superbench/superbench:v0.12.0-cuda12.9 in offline environments, it fails with error: The binary does not exist - benchmark: mem-bw, binary name: bandwidthTest. The Docker image contains CUDA 12.9 but lacks the CUDA samples package which includes the required bandwidthTest tool.

Expected: The official Docker image should either include all necessary dependencies for running benchmarks, or provide clear documentation for offline setup, or implement fallback mechanisms for missing binaries.

How to reproduce it?:
In an offline environment, pull the official Docker image: docker pull superbench/superbench:v0.12.0-cuda12.9
Run a privileged container: docker run --privileged --rm -it superbench/superbench:v0.12.0-cuda12.9 bash
Inside the container, execute: sb run --no-docker -l localhost -c superbench_test.yaml
superbench:
enable: ['mem-bw']
The mem-bw benchmark will fail with the missing bandwidthTest error.

Log message or shapshot?:

[2025-12-10 02:27:29,029 HOSTNAME:326471][executor.py:251][INFO] Executor is going to execute mem-bw.
[2025-12-10 02:27:34,189 HOSTNAME:326471][micro_base.py:138][ERROR] The binary does not exist - benchmark: mem-bw, binary name: bandwidthTest, binary directory: None.
[2025-12-10 02:27:34,189 HOSTNAME:326471][executor.py:133][INFO] benchmark: mem-bw, return code: 31, result: {'return_code': [31]}.
[2025-12-10 02:27:34,189 HOSTNAME:326471][executor.py:140][ERROR] Executor failed in mem-bw.
[2025-12-10 02:27:35,261 HOSTNAME:323963][ansible.py:80][INFO] Run succeed, return code 0.
[2025-12-10 02:27:35,262 HOSTNAME:323963][runner.py:449][INFO] Runner is going to run mem-bw in local mode, proc rank 2.
[2025-12-10 02:27:35,262 HOSTNAME:323963][ansible.py:110][INFO] Run bash -c 'set -o allexport && source /tmp/sb.env && set +o allexport && cd $SB_WORKSPACE && PROC_RANK=2 CUDA_VISIBLE_DEVICES=2 numactl -N $((2/4)) sb exec --output-dir outputs/2025-12-10_02-26-55 -c sb.config.yaml -C superbench.enable=mem-bw' on remote ...
[WARNING]: Platform linux on host localhost is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
localhost | CHANGED | rc=0 >>
[2025-12-10 02:27:38,092 HOSTNAME:326760][executor.py:251][INFO] Executor is going to execute mem-bw.
[2025-12-10 02:27:43,221 HOSTNAME:326760][micro_base.py:138][ERROR] The binary does not exist - benchmark: mem-bw, binary name: bandwidthTest, binary directory: None.
[2025-12-10 02:27:43,221 HOSTNAME:326760][executor.py:133][INFO] benchmark: mem-bw, return code: 31, result: {'return_code': [31]}.
[2025-12-10 02:27:43,221 HOSTNAME:326760][executor.py:140][ERROR] Executor failed in mem-bw.
[2025-12-10 02:27:44,294 HOSTNAME:323963][ansible.py:80][INFO] Run succeed, return code 0.
[2025-12-10 02:27:44,294 HOSTNAME:323963][runner.py:449][INFO] Runner is going to run mem-bw in local mode, proc rank 3.
[2025-12-10 02:27:44,295 HOSTNAME:323963][ansible.py:110][INFO] Run bash -c 'set -o allexport && source /tmp/sb.env && set +o allexport && cd $SB_WORKSPACE && PROC_RANK=3 CUDA_VISIBLE_DEVICES=3 numactl -N $((3/4)) sb exec --output-dir outputs/2025-12-10_02-26-55 -c sb.config.yaml -C superbench.enable=mem-bw' on remote ...
[WARNING]: Platform linux on host localhost is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
localhost | CHANGED | rc=0 >>
[2025-12-10 02:27:47,117 HOSTNAME:327049][executor.py:251][INFO] Executor is going to execute mem-bw.
[2025-12-10 02:27:52,280 HOSTNAME:327049][micro_base.py:138][ERROR] The binary does not exist - benchmark: mem-bw, binary name: bandwidthTest, binary directory: None.
[2025-12-10 02:27:52,280 HOSTNAME:327049][executor.py:133][INFO] benchmark: mem-bw, return code: 31, result: {'return_code': [31]}.
[2025-12-10 02:27:52,280 HOSTNAME:327049][executor.py:140][ERROR] Executor failed in mem-bw.
[2025-12-10 02:27:53,361 HOSTNAME:323963][ansible.py:80][INFO] Run succeed, return code 0.
[2025-12-10 02:27:53,362 HOSTNAME:323963][runner.py:449][INFO] Runner is going to run mem-bw in local mode, proc rank 4.
[2025-12-10 02:27:53,362 HOSTNAME:323963][ansible.py:110][INFO] Run bash -c 'set -o allexport && source /tmp/sb.env && set +o allexport && cd $SB_WORKSPACE && PROC_RANK=4 CUDA_VISIBLE_DEVICES=4 numactl -N $((4/4)) sb exec --output-dir outputs/2025-12-10_02-26-55 -c sb.config.yaml -C superbench.enable=mem-bw' on remote ...
[WARNING]: Platform linux on host localhost is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
localhost | CHANGED | rc=0 >>
[2025-12-10 02:27:56,253 HOSTNAME:327338][executor.py:251][INFO] Executor is going to execute mem-bw.
[2025-12-10 02:28:01,476 HOSTNAME:327338][micro_base.py:138][ERROR] The binary does not exist - benchmark: mem-bw, binary name: bandwidthTest, binary directory: None.
[2025-12-10 02:28:01,476 HOSTNAME:327338][executor.py:133][INFO] benchmark: mem-bw, return code: 31, result: {'return_code': [31]}.
[2025-12-10 02:28:01,476 HOSTNAME:327338][executor.py:140][ERROR] Executor failed in mem-bw.
[2025-12-10 02:28:02,575 HOSTNAME:323963][ansible.py:80][INFO] Run succeed, return code 0.
[2025-12-10 02:28:02,575 HOSTNAME:323963][runner.py:449][INFO] Runner is going to run mem-bw in local mode, proc rank 5.
[2025-12-10 02:28:02,575 HOSTNAME:323963][ansible.py:110][INFO] Run bash -c 'set -o allexport && source /tmp/sb.env && set +o allexport && cd $SB_WORKSPACE && PROC_RANK=5 CUDA_VISIBLE_DEVICES=5 numactl -N $((5/4)) sb exec --output-dir outputs/2025-12-10_02-26-55 -c sb.config.yaml -C superbench.enable=mem-bw' on remote ...
[WARNING]: Platform linux on host localhost is using the discovered Python interpreter at /usr/bin/python3.12, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.
localhost | CHANGED | rc=0 >>

Additional information:

• Environment: Ubuntu 22.04 x86_64, offline (no internet access)
• Following official TIP: Using privileged container with --no-docker mode as suggested in SuperBench documentation
• Workaround limitations: In offline environments, cannot download source code from GitHub, cannot use alternative package sources

[guoshzhao]

The bandwidthTest sample was out-of-date and has been removed as of the CUDA Samples 12.9 release (see the change log). For up-to-date bandwidth measurements, refer instead to the NVBandwith utility.

Please check:
https://github.com/NVIDIA/cuda-samples/tree/master/Samples/1_Utilities
https://microsoft.github.io/superbenchmark/docs/user-tutorial/benchmarks/micro-benchmarks#mem-bw

And nvbandwidth benchmark is provided from cuda 12.9 images:
https://microsoft.github.io/superbenchmark/docs/user-tutorial/benchmarks/micro-benchmarks#nvbandwidth

[NePixe1]

Thank you for your reply and clarification! I I'll follow your suggestion and switch to the nvbandwidth benchmark instead.
Additionally, I've encountered another issue and would like to ask for your advice:

When running the cudnn-function benchmark, the return code is 33 instead of the previously seen 31. How can I determine what error return code 33 corresponds to?

From the test results, all GPUs (0-7) returned the same error code 33:
"cudnn-function/return_code:0": 33, "cudnn-function/return_code:1": 33, "cudnn-function/return_code:2": 33, "cudnn-function/return_code:3": 33, "cudnn-function/return_code:4": 33, "cudnn-function/return_code:5": 33, "cudnn-function/return_code:6": 33, "cudnn-function/return_code:7": 33,

many test results returned -1 instead of normal performance data, for example:
"cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:0": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:1": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:2": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:3": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:4": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:5": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:6": -1, "cudnn-function/name_cudnnconvolutionbackwarddata_algo_4_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_32_14_14]_inputstride_[6272_196_14_1]_inputtype_0_mode_1_outputdims_[32_128_14_14]_outputstride_[25088_196_14_1]_pada_[1_1]_tensorop_false_time:7": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:0": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:1": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:2": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:3": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:4": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:5": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:6": -1, "cudnn-function/name_cudnnconvolutionforward_algo_6_arraylength_2_convtype_0_dilationa_[1_1]_filterdims_[32_128_3_3]_filterstridea_[1_1]_inputdims_[32_128_14_14]_inputstride_[25088_196_14_1]_inputtype_0_mode_1_outputdims_[32_32_14_14]_outputstride_[6272_196_14_1]_pada_[1_1]_tensorop_false_time:7": -1,

Does this mean that certain GPU models fail on specific dimension convolution operations? Or is this some specific error state?

Is there documentation available for SuperBench return codes? Specifically, what does return code 33 mean?

Thank you!

[guoshzhao]

@NePixe1 You can find the return code definition here: (it's not a document but python file)https://github.com/microsoft/superbenchmark/blob/main/superbench/benchmarks/return_code.py

    MICROBENCHMARK_BINARY_NOT_EXIST = 31
    MICROBENCHMARK_RESULT_PARSING_FAILURE = 33

For result with -1, yes, it means certain GPU models fail on specific dimension convolution operations.