Data Transfer from CPU to GPU is not optimized

[javak87]

What happened?

The initial investigation shows that training on the A100 (JWB) is faster than on the GH200 (Santis). Here is how to reproduce the result:

git checkout d24c4b6800b45bd1f859e61d8b29eab5a540c176
../WeatherGenerator-private/hpc/launch-slurm.py --time 180 --nodes=1

here is the result:

run_id	HPC	PR	Ingested Samples per GPU
pyizojg7	Santis	develop (1 node) (180 mins)	6684
cc0xrzbm	JWB	develop (1 node) (180 mins)	7688

What are the steps to reproduce the bug?

No response

Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.

No response

[javak87]

assign me

[github-actions]

Issue assigned to @javak87.

[javak87]

The performance of the GH200 in Santis has been checked and compared with the A100 in JWB. It seems that the memory transfer between the CPU and GPU is not optimized.

Here is the performance of the A100:

Here is the performance of the GH200:

A closer look shows that the throughput on the A100 is around 6 GB/s, while on the GH200 it is 248 MB/s, which is really strange.

Given the existing NVLink between the CPU and GPU on the GH200, the throughput should be around 450 GB/s, which is far from the current throughput.

I'll look at the transfer and check the reason.

[clessig]

I am a little big confused. For our default config I get 1.5 samples/sec on Santis, and less than 1 sample/sec on an A100. So the GH200 are clearly faster.

[javak87]

I ran the experiment regarding this commit:
d24c4b6800b45bd1f859e61d8b29eab5a540c176

Could you confirm GH200 is faster for this commit (1 node)?

[javak87]

I updated the develop branch and ran this:
../WeatherGenerator-private/hpc/launch-slurm.py --time 20 --nodes=1

Here is GH200 (Santis) output:

0: 000 : 00020/01024 : 000020 : loss = 7.1191E-01 (lr=1.41E-06, s/sec=0.967)
0: 	
0: LossPhysical.ERA5.mse.avg : 1.0729E+00 	
0: LossPhysical.ERA5.mae.avg : 7.9329E-01 	
0: LossPhysical.loss_avg : 1.0170E+00 	
0: 
0: 
0: 000 : 00030/01024 : 000030 : loss = 6.8035E-01 (lr=1.90E-06, s/sec=0.965)
0: 	
0: LossPhysical.ERA5.mse.avg : 1.0218E+00 	
0: LossPhysical.ERA5.mae.avg : 7.7223E-01 	
0: LossPhysical.loss_avg : 9.7192E-01 	
0: 
0: 
0: 000 : 00040/01024 : 000040 : loss = 6.6549E-01 (lr=2.56E-06, s/sec=0.998)
0: 	
0: LossPhysical.ERA5.mse.avg : 9.9822E-01 	
0: LossPhysical.ERA5.mae.avg : 7.6058E-01 	
0: LossPhysical.loss_avg : 9.5069E-01 

and here is A100 (JWB):

0: 000 : 00020/01024 : 000020 : loss = 6.9762E-01 (lr=1.41E-06, s/sec=0.718)
0: 	
0: LossPhysical.ERA5.mse.avg : 1.0495E+00 	
0: LossPhysical.ERA5.mae.avg : 7.8508E-01 	
0: LossPhysical.loss_avg : 9.9660E-01 	
0: 
0: 
0: 000 : 00030/01024 : 000030 : loss = 6.7679E-01 (lr=1.90E-06, s/sec=0.678)
0: 	
0: LossPhysical.ERA5.mse.avg : 1.0168E+00 	
0: LossPhysical.ERA5.mae.avg : 7.6699E-01 	
0: LossPhysical.loss_avg : 9.6684E-01 	
0: 
0: 
0: 000 : 00040/01024 : 000040 : loss = 6.6692E-01 (lr=2.56E-06, s/sec=0.689)
0: 	
0: LossPhysical.ERA5.mse.avg : 1.0005E+00 	
0: LossPhysical.ERA5.mae.avg : 7.6154E-01 	
0: LossPhysical.loss_avg : 9.5274E-01 

I also profile the code for both machines and here is the result on A100 (JWB):

Here is the result for GH200 (Santis):

It seems that after commit d24c4b6, that part has improved. However, the problem still exists, and the memcpy is still not optimized on Santis. There is still room for improvement—around a 3–5% improvement is expected.

[clessig]

Is this with multiple forecast steps? For 1-step forecasting, I get consistently 1.2-1.3 samples/sec on Santis.

[javak87]

I don't know the default config is multiple or one-step. I pulled the last develop branch and simply ran:

../WeatherGenerator-private/hpc/launch-slurm.py --time 20 --nodes=1

[clessig]

Santis: samples/sec=1.0
JUWELS-Booster: samples/sec=0.58

[clessig]

Santis: samples/sec=1.0 JUWELS-Booster: samples/sec=0.58

This is interactive.

[clessig]

I don't know the default config is multiple or one-step. I pulled the last develop branch and simply ran:

../WeatherGenerator-private/hpc/launch-slurm.py --time 20 --nodes=1

This is masking, no forecast steps.