Measure performance of LILAC's redistribution at various processor counts

[billsacks]

We should measure the performance of LILAC's data redistribution (between the atmosphere and land decompositions) at various processor counts. We should determine the time it takes both to do this redistribution, and (if possible) determine the slowdown caused by introducing this global synchronization point.

(@gutmann raised this point yesterday; it is something we've talked about doing ever since the initial LILAC proposal, but I wanted to open an issue to make sure it gets done.)

It will probably be easier to do this after #894 is resolved: Blocked by #894 .

[billsacks]

I did a very unscientific look at this issue in the course of investigating a different performance issue. With a 2-day CONUS run on 72 processors (2 nodes) on cheyenne, I found:

• If I bypassed LILAC and CTSM entirely (so didn't run any land code in WRF), runtime was about 3 min 7 sec per simulated day
• If I did all of the LILAC stuff (data redistribution between decompositions, etc.) but returned from CTSM immediately, runtime was about 3 min 10 sec per simulated day

This difference (1.6%) is probably within machine variability (I only did a single run of each case), but at least indicates that, at this relatively low processor count, the time taken by LILAC's redistribution is relatively small, and is roughly in line with my gut-level expectations based on coupler timings in CESM.