Fix bug in determining when decompositions can be reused by the SMIOL library

[mgduda]

This PR fixes a bug in logic for determining when decompositions can be reused by the SMIOL library.

The MPAS_io_set_var_indices() routine in mpas_io.F contains logic to determine whether a decomposition can be reused from the set of existing decompositions; this can save time, as the creation of a new decomposition structure can incur a non-trivial cost.

The checks to determine whether a SMIOL decomposition can be reused previously employed the following criteria:

• The number of compute offsets (indices) on an MPI rank matches
• The offsets themselves for the MPI rank all agreed

These two criteria alone left open the possibility that all MPI ranks may individually find apparently compatible decompositions, but that the decomposition selected by each MPI rank may be different, leading to I/O errors.

Consider the following two decompositions with sets of compute offsets on two MPI ranks:

           MPI rank 0    MPI rank 1
Decomp 1:  {0,2}         {1}
Decomp 2:  {0}           {1}

If a new field in which rank 0 has offset {0} and rank 1 has offset {1} was considered, MPI rank 0 would select decomp 2 as a matching decomposition, while MPI rank 1 would select decomp 1 (because decompositions are tested in order).

Now, a third criterion has been added to the checks, namely, that the decomposition must have the same global number of offsets. It is worth noting that this new criterion is a necessary condition for correctly reusing decompositions, but combined with the existing criteria, may not be sufficient to guarantee correct reuse of decompositions under all possible conditions. Nonetheless, the addition of the third criterion enables the correct selection of decompositions under conditions
that previously resulted in failure.

[mgduda]

@abishekg7 @gdicker1 One way to induce a failure in the current release code is to run on a mesh with 12 cells using 12 MPI tasks. In /glade/u/home/duda/ you can find x1.12.grid.nc and x1.12.graph.info files for producing this test case.

[abishekg7]

Looks good to me.

Tested the failing case with the current master (using the 12-cell grid), and seems to fail with this message Message from rank 0 and tag 5 truncated; 4 bytes received but buffer size is 8. Whereas the current PR run finishes successfully. The output netcdf is bit-identical between the serial and 12-rank runs.