Add comments on MPI ports vs sockets

[MakisH]

It looks like users do not really understand the issue with MPI ports, and they do not understand how much of a compromise sockets are.
This adds a reference to Benjamin's dissertation (scaling results), but we may need to add even more information.

@uekerman what else could we improve here?

[uekerman]

Looks good. I added a bit more.

[MakisH]

@uekerman I added a few more clarifications, based on our discussion.

[Scriptkiddi]

would it be possible to add a quantification for very large coupling meshes?
like very large coupling meshes ( > 1000000 vertices) to help people decide quickly in which category they fall? Because I would guess different fields have different understandings of what a very large mesh is.

[fsimonis]

@Scriptkiddi The basic guideline is to use sockets.

The size of the communicated mesh depend on many factors. Vertex count is only a part of the story. Mesh connectivity, defined mappings, defined and exchanged data, watch integrals and filtering strategies all impact this communicated size.

You will need to use the inbuilt profiling, or you do your own.

The general guideline for the inbuilt profiling (which is missing in the docs) is:

1. Start using sockets
2. Enable sync mode in the configuration.
3. Run your full testcase for a few time steps.
4. Establish your baselines:
   • For the mesh transfer, checkout the events partition.sendGlobalMesh in initialize
   • For the data transfer, m2n.sendData and m2n.receiveData events in advance.
5. Figure out if these impact the overall simulation time. If not, then you are done.
6. Change to <m2n:mpi > and rerun the case.
7. Compare the events in 4 to your established baseline and select the preferred method.

[uekerman]

would it be possible to add a quantification for very large coupling meshes?

it mainly depends on how this relates to the computational cost of your solver. good point, but hard to quantify 😕