Improve LOOP datatype exchanges performances for sizes above the eage…

[derbeyn]

…r limit.

When a datatype is internally described by a (LOOP-{DATA}*-ENDLOOP) structure, and this datatype size is greater than the btl eager limit, the following process is applied:

1. a loop is packed up to the eager size.
2. the remaing of the loop is processed single DATA by single DATA.

This process has been enhanced by artificially inserting a LOOP element during the packing of the remaining elements. This has the effect of considering the second part of a LOOP as a LOOP too, instead of a series of DATA.

As shown in the attached graph, performances are improved above the eager limit, when exchanging this kind of datatypes.
For example, for a 258240 bytes buffer, we move from 1502.635195 down to 965.680833 microseconds.

This graph has been generated with the master, and the following MCA parameters set:
OMPI_MCA_pml=ob1
OMPI_MCA_btl=openib
OMPI_MCA_btl_openib_allow_ib=1

Signed-off-by: Nadia Derbey Nadia.Derbey@atos.net

[hjelmn]

@bosilca Looks good to me. Can you verify this?

[ggouaillardet]

@derbeyn could you please provide an example of derived datatype that benefits from this optimization ?

[bosilca]

Any vector type would work. You should use it together with a handcrafted eager, one that would leave the bConverted in the middle of one of the vector stride.

[derbeyn]

@ggouaillardet sorry for the delay in answering, I was on vacation.
As @bosilca said, a vector exchange is enough to see the optimization.
Attaching a source file to give you an idea:

simple_test_vector.c.txt

[jsquyres]

@derbeyn @bosilca Ping. Is this PR going anywhere?

[derbeyn]

Not sure it has ever been reviewed?

[bosilca]

@derbeyn can you check the status of the new datatype engine (#6695) with regard to this topic ?

[bosilca]

This issue has been addressed in #6695