Understand performance difference in cudacpp between SA and madevent

[valassi]

This is a followup to PR #454 where I am doing some first tests (with logs) of madevent+cudacpp MEs, comparing physics and performance to madevent and to cudacpp.

I see that the cudacpp ME calculation is faster in standalone (bridge!) mode than in madevent. IN principle I thought the standalone bridge mode was exactly equivalent?

• There is most likely an issue with helicity filtering, which should be excluded from the measurements
• But there may be more than just that, to be understood
• (Note that in principle both are using the same CUDA grids, so any slowdown - In Fortran MadEvent, use larger arrays (beyone 16k) to allow efficient usage of GPU #460 - from using too small CUDA grids should not be here)

See for instance
https://github.com/madgraph5/madgraph4gpu/blob/d1fd8b201803b366579f925017f09a85001ce9f8/epochX/cudacpp/tmad/logs_ggttggg_mad/log_ggttggg_mad_d_inl0_hrd0.txt

DATE: 2022-05-20_15:00:32

Working directory: /data/avalassi/GPU2020/madgraph4gpuX/epochX/cudacpp/gg_ttggg.mad/SubProcesses/P1_gg_ttxggg

*** EXECUTE MADEVENT (create results.dat) ***
 [XSECTION] Cross section = 80.03
 [COUNTERS] PROGRAM TOTAL          :    4.2070s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.2606s
 [COUNTERS] Fortran MEs      ( 1 ) :    3.9463s for       96 events => throughput is 2.43E+01 events/s

*** EXECUTE CMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 80.03
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00014921 = 1 + 0.00015
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     96
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.3e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    4.2596s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.2727s
 [COUNTERS] Fortran MEs      ( 1 ) :    3.5988s for       96 events => throughput is 2.67E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.3881s for       96 events => throughput is 2.47E+02 events/s

*** EXECUTE CHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+BRDHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.888841e+02                 )  sec^-1

*** EXECUTE CHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+MESHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.892185e+02                 )  sec^-1

*** EXECUTE GMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 404.9
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00007834 = 1 + 7.8e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     64
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.1e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    3.6001s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.5031s
 [COUNTERS] Fortran MEs      ( 1 ) :    2.4043s for       64 events => throughput is 2.66E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.6928s for       64 events => throughput is 9.24E+01 events/s

*** EXECUTE GCHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURHST+RMBHST+BRDDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.032552e+02                 )  sec^-1

*** EXECUTE GCHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURDEV+RMBDEV+MESDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.367983e+02                 )  sec^-1

TEST COMPLETED

This is ggttggg. In Madevent+cuda I get 9.2E01 events per second. In standalone cuda (bridge mode) I get 2.0E2 events per second. The factor two is probably just helicity filtering?

[valassi]

I am about to merge PR #463, which fixes the main part of this issue, namely the fact that helicity filtering was giving a factor ~2 overhead to cudacpp timing measurements.

The situation is now better for complex processes, but there are still some differences for less complex processes.

(1) Example, ggttggg

https://github.com/madgraph5/madgraph4gpu/blob/1f40443dfa2166e083e74fd52beaeae055327e84/epochX/cudacpp/tmad/logs_ggttggg_mad/log_ggttggg_mad_d_inl0_hrd0.txt

In cpp

*** EXECUTE CMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 80.03
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00014921 = 1 + 0.00015
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     96
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.3e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    4.5175s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.4516s
 [COUNTERS] Fortran MEs      ( 1 ) :    3.7333s for       96 events => throughput is 2.57E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.3326s for       96 events => throughput is 2.89E+02 events/s

*** EXECUTE CHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+BRDHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.891759e+02                 )  sec^-1

*** EXECUTE CHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+MESHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.895414e+02                 )  sec^-1

Here the throughput is solidly 2.89E2 for cpp, whether for madevent, for SA bridge, or for SA no bridge.

In cuda

*** EXECUTE GMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 404.9
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00007834 = 1 + 7.8e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     64
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.1e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    3.9412s
 [COUNTERS] Fortran Overhead ( 0 ) :    1.1635s
 [COUNTERS] Fortran MEs      ( 1 ) :    2.4934s for       64 events => throughput is 2.57E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.2842s for       64 events => throughput is 2.25E+02 events/s

*** EXECUTE GCHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURHST+RMBHST+BRDDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.030810e+02                 )  sec^-1

*** EXECUTE GCHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURDEV+RMBDEV+MESDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.366721e+02                 )  sec^-1

Here the throughput in cuda is 2.02 in SA bridge and 2.4E2 in SA no bridge. Surprisingly, in madevent it seems even a bit faster than SA bridge, 2.2E2? But anyway, not slower... an dthe madevent number is what counts in the end.

(2) Take an example on the opposite spectrum, simple processes, ggtt.
https://github.com/madgraph5/madgraph4gpu/blob/1f40443dfa2166e083e74fd52beaeae055327e84/epochX/cudacpp/tmad/logs_ggtt_mad/log_ggtt_mad_d_inl0_hrd0.txt

In cpp

*** EXECUTE CMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 435.3
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99992635 = 1 - 7.4e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00000527 = 1 + 5.3e-06
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =  16416
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = -2.5e-05 +- 2.6e-07
 [COUNTERS] PROGRAM TOTAL          :    1.1307s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.8089s
 [COUNTERS] Fortran MEs      ( 1 ) :    0.2806s for    16416 events => throughput is 5.85E+04 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.0412s for    16416 events => throughput is 3.99E+05 events/s

*** EXECUTE CHECK -p 512 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTX_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+BRDHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 6.210578e+05                 )  sec^-1

*** EXECUTE CHECK -p 512 32 1 ***
Process                     = SIGMA_SM_GG_TTX_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+MESHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 6.286714e+05                 )  sec^-1

Here the cpp throughput is 4.0E5 in madevent, against 6.2E5 in SA bridge, and 6.3E5 in SA no bridge. There is clearly a large overhead, which is not seen in the SA bridge vs no bridge mode.

Similarly in cuda

*** EXECUTE GMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 448.3
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99992657 = 1 - 7.3e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00000531 = 1 + 5.3e-06
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =  16384
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = -2.5e-05 +- 2.5e-07
 [COUNTERS] PROGRAM TOTAL          :    1.1979s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.9113s
 [COUNTERS] Fortran MEs      ( 1 ) :    0.2858s for    16384 events => throughput is 5.73E+04 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.0008s for    16384 events => throughput is 2.16E+07 events/s

*** EXECUTE GCHECK -p 512 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTX_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURHST+RMBHST+BRDDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.997137e+07                 )  sec^-1

*** EXECUTE GCHECK -p 512 32 1 ***
Process                     = SIGMA_SM_GG_TTX_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURDEV+RMBDEV+MESDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 9.277936e+07                 )  sec^-1

Here the cuda throughput is 2.2E7 in madevent, against 3.0E7 in SA bridge, and 9.3E7(!) in SA bridge. There is again a large overhead from the madevent mode, but also a large overhead from the SA bridge mode.

What do do now? I opened issue #464 to modify the timers in the SA bridge mode, but this is really unnecessary, especially because for complex processes the difference is low.

From now on, I suggest we keep track of two measurements:

• the cudacpp (and fortran!) throughputs in madevent, because this is what the experiments will see
• the cudacpp throughput in no-bridge standalone mode, because this is our maximum achieved

I am merging PR #463 and I will close this #461.