First patch for automatic Fortran vs cuda/cpp comparisons

[valassi]

This is a WIP PR for the issues described in #417, namely some automatic comparison of Fortran vs Cudacpp, both for physics and computing performance

[valassi]

I mark this as ready and will soon merge it.

There are very many things that still need to be done, but many things are done, including (not exhaustive list)

• I have madX.sh and teeMadX.sh scripts which generate some logs with interesting numbers, both for physics values and for computing performance (see an example below). They include some process-dependent tuning of how many events should be executed through madevent (with fortran, cuda, cpp MEs) and also of how many events per loop should be used.
• There is some initial progress at running many more GPU events via Madevent (see In Fortran MadEvent, use larger arrays (beyond 16 events, up to 16k) to allow efficient usage of GPUs #455). However there are too many arrays in Fortran that need to be dimensioned this way, and overall I do not manage to go beyond 16k events per grid, else it segfaults (I guess it is out of memory)
• NB the code is still doing the main ME calculation in Fortran, with cudacpp as only a cross check. We need multichannel and the full API with color/helicity choice to go beyond this step.

Amongst the things that still need work (non exhaustive list)

• Go beyond 16k events per grid, see In Fortran MadEvent, use larger arrays (beyone 16k) to allow efficient usage of GPU #460
• Understand the performance differences between cudacpp in standalone mode and in madevent mode, Understand performance difference in cudacpp between SA and madevent #461
• Fix the eemumu process, which is returning a zero cross section Simple ee->mumu returns 0 cross section in 311 branch oliviermattelaer/mg5amc_test#13
• Go beyond the cross section calculation in madevent and into unweighted event generation, which is failing with "Error: failed to reduce to color indices" in gg to ttg (only for event generation, not for cross sections) oliviermattelaer/mg5amc_test#14
• Once multichannel is done (and later random choices of color/helicity) - MadEvent final bridge interface: add multichannel, alphas, color index #404 - do the full calculation of xsec and unweighted event generation in cudacpp instead of fortran, and compare physics results,
• In parallel (this can be done also now at the ME level), understand the E-3 level differences in MEs between fortran and cudacpp, Physics results are different in fortran and cudacpp MEs (by factors or orders of magnitude) #417

[valassi]

All tests passed, I am self merging

[valassi]

PS Example of a tmad log

https://github.com/madgraph5/madgraph4gpu/blob/d1fd8b201803b366579f925017f09a85001ce9f8/epochX/cudacpp/tmad/logs_ggttggg_mad/log_ggttggg_mad_d_inl0_hrd0.txt

DATE: 2022-05-20_15:00:32

Working directory: /data/avalassi/GPU2020/madgraph4gpuX/epochX/cudacpp/gg_ttggg.mad/SubProcesses/P1_gg_ttxggg

*** EXECUTE MADEVENT (create results.dat) ***
 [XSECTION] Cross section = 80.03
 [COUNTERS] PROGRAM TOTAL          :    4.2070s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.2606s
 [COUNTERS] Fortran MEs      ( 1 ) :    3.9463s for       96 events => throughput is 2.43E+01 events/s

*** EXECUTE CMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 80.03
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00014921 = 1 + 0.00015
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     96
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.3e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    4.2596s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.2727s
 [COUNTERS] Fortran MEs      ( 1 ) :    3.5988s for       96 events => throughput is 2.67E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.3881s for       96 events => throughput is 2.47E+02 events/s

*** EXECUTE CHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+BRDHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.888841e+02                 )  sec^-1

*** EXECUTE CHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CPP [gcc 10.2.0] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CPP:DBL+CXS:CURHST+RMBHST+MESHST/512y+CXVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.892185e+02                 )  sec^-1

*** EXECUTE GMADEVENT_CUDACPP (create results.dat) ***
 [XSECTION] Cross section = 404.9
 [MERATIOS] ME ratio CudaCpp/Fortran: MIN = 0.99997121 = 1 - 2.9e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: MAX = 1.00007834 = 1 + 7.8e-05
 [MERATIOS] ME ratio CudaCpp/Fortran: NENTRIES =     64
 [MERATIOS] ME ratio CudaCpp/Fortran - 1: AVG = 1.1e-05 +- 2.9e-06
 [COUNTERS] PROGRAM TOTAL          :    3.6001s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.5031s
 [COUNTERS] Fortran MEs      ( 1 ) :    2.4043s for       64 events => throughput is 2.66E+01 events/s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    0.6928s for       64 events => throughput is 9.24E+01 events/s

*** EXECUTE GCHECK -p 2 32 1 --bridge ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURHST+RMBHST+BRDDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.032552e+02                 )  sec^-1

*** EXECUTE GCHECK -p 2 32 1 ***
Process                     = SIGMA_SM_GG_TTXGGG_CUDA [nvcc 11.6.124 (gcc 10.2.0)] [inlineHel=0] [hardcodePARAM=0]
Workflow summary            = CUD:DBL+THX:CURDEV+RMBDEV+MESDEV/none+NAVBRK
EvtsPerSec[MECalcOnly] (3a) = ( 2.367983e+02                 )  sec^-1

TEST COMPLETED