Distinguish between MPI traceback and fdistdump stderr

[jwrona]

MPI process manager mixes its own and the managed processes' stdout and stderr streams together. This is fine until some error occurs, then the streams contain for example (using MPICH):

stdout:

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 7
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

stderr:

application called MPI_Abort(MPI_COMM_WORLD, 7) - process 2
fdistdump_mpich: unrecognized option '--invalid-switch'
application called MPI_Abort(MPI_COMM_WORLD, 7) - process 1
application called MPI_Abort(MPI_COMM_WORLD, 7) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 7) - process 3

Users usually don't want to see messages from the MPI process manager and also want to see fdistdump error only once.

[jwrona]

Possible solutions:

• Use file descriptors other than 1 (stdout) and 2 (stdedd). Their redirection from the managed processes to the mpiexec is not a standard feature. MPICH supports it, Open MPI does not.
• Use an output labels:

MPICH:

$ mpiexec -n 2 -l ./a.out
[0] output to stdout
[0] output to stderr
[1] output to stderr
[1] output to stdout

Open MPI:

$ mpiexec -n 2 --tag-output ./a.out
[1,0]<stdout>:output to stdout
[1,0]<stderr>:output to stderr
[1,1]<stderr>:output to stderr
[1,1]<stdout>:output to stdout