On DGX 11, this is so far slower than IB on Azure added in #69

(dask-benchmark) mroeschke@dgx11:~$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	mlx5_0	mlx5_1	mlx5_2	mlx5_3	CPU Affinity	NUMA Affinity
GPU0	 X 	NV1	NV1	NV2	NV2	SYS	SYS	SYS	PIX	PHB	SYS	SYS	0-19,40-59	0
GPU1	NV1	 X 	NV2	NV1	SYS	NV2	SYS	SYS	PIX	PHB	SYS	SYS	0-19,40-59	0
GPU2	NV1	NV2	 X 	NV2	SYS	SYS	NV1	SYS	PHB	PIX	SYS	SYS	0-19,40-59	0
GPU3	NV2	NV1	NV2	 X 	SYS	SYS	SYS	NV1	PHB	PIX	SYS	SYS	0-19,40-59	0
GPU4	NV2	SYS	SYS	SYS	 X 	NV1	NV1	NV2	SYS	SYS	PIX	PHB	20-39,60-79	1
GPU5	SYS	NV2	SYS	SYS	NV1	 X 	NV2	NV1	SYS	SYS	PIX	PHB	20-39,60-79	1
GPU6	SYS	SYS	NV1	SYS	NV1	NV2	 X 	NV2	SYS	SYS	PHB	PIX	20-39,60-79	1
GPU7	SYS	SYS	SYS	NV1	NV2	NV1	NV2	 X 	SYS	SYS	PHB	PIX	20-39,60-79	1
mlx5_0	PIX	PIX	PHB	PHB	SYS	SYS	SYS	SYS	 X 	PHB	SYS	SYS
mlx5_1	PHB	PHB	PIX	PIX	SYS	SYS	SYS	SYS	PHB	 X 	SYS	SYS
mlx5_2	SYS	SYS	SYS	SYS	PIX	PIX	PHB	PHB	SYS	SYS	 X 	PHB
mlx5_3	SYS	SYS	SYS	SYS	PHB	PHB	PIX	PIX	SYS	SYS	PHB	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

UCX

(dask-benchmark) mroeschke@dgx11:~$ python -m dask_cuda.benchmarks.local_cudf_merge --devs 0,1,2,3,4,5,6,7 --chunk-size 100_000_000 --protocol ucx

Merge benchmark
--------------------------------------------------------------------------------
Backend                   | dask
Merge type                | gpu
Rows-per-chunk            | 100000000
Base-chunks               | 8
Other-chunks              | 8
Broadcast                 | default
Protocol                  | ucx
Device(s)                 | 0,1,2,3,4,5,6,7
RMM Pool                  | True
Frac-match                | 0.3
TCP                       | None
InfiniBand                | None
NVLink                    | None
Worker thread(s)          | 1
Data processed            | 23.84 GiB
Number of workers         | 8
================================================================================
Wall clock                | Throughput
--------------------------------------------------------------------------------
16.12 s                   | 1.48 GiB/s
15.64 s                   | 1.52 GiB/s
21.54 s                   | 1.11 GiB/s
================================================================================
Throughput                | 1.34 GiB/s +/- 119.48 MiB/s
Bandwidth                 | 133.58 MiB/s +/- 12.66 MiB/s
Wall clock                | 17.76 s +/- 2.67 s
================================================================================

TCP

(dask-benchmark) mroeschke@dgx11:~$ python -m dask_cuda.benchmarks.local_cudf_merge --devs 0,1,2,3,4,5,6,7 --chunk-size 100_000_000

Merge benchmark
--------------------------------------------------------------------------------
Backend                   | dask
Merge type                | gpu
Rows-per-chunk            | 100000000
Base-chunks               | 8
Other-chunks              | 8
Broadcast                 | default
Protocol                  | tcp
Device(s)                 | 0,1,2,3,4,5,6,7
RMM Pool                  | True
Frac-match                | 0.3
Worker thread(s)          | 1
Data processed            | 23.84 GiB
Number of workers         | 8
================================================================================
Wall clock                | Throughput
--------------------------------------------------------------------------------
59.60 s                   | 409.63 MiB/s
53.44 s                   | 456.87 MiB/s
52.81 s                   | 462.28 MiB/s
================================================================================
Throughput                | 441.62 MiB/s +/- 14.13 MiB/s
Bandwidth                 | 32.21 MiB/s +/- 467.30 kiB/s
Wall clock                | 55.28 s +/- 3.06 s
================================================================================

Cloud benchmark comparing performance with on-prem #17

Description

mmccartyopenedon Jul 13, 2022

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

mmccarty
openedon Jul 13, 2022