The parallel I/O implementation doesn't seem to play nicely with the test suite

[mnlevy1981]

I ran an SMS.TL319_t232.G1850MARBL_JRA.derecho_intel test and generated baselines for it (/glade/derecho/scratch/mlevy/baselines/one-offs/alpha06e_control/SMS.TL319_t232.G1850MARBL_JRA.derecho_intel on Derecho). Some of the MOM6 netcdf files that got copied over are the patch files:

$ ls -1 *mom*.nc*
mom6.h.bgc.z.0001-01.nc
mom6.h.native.0001-01-05.nc.0009
mom6.h.ocean_geometry.nc
mom6.h.rho2.0001-01-05.nc.0009
mom6.h.static.nc.0009
mom6.h.z.0001-01-05.nc.0009
mom6.ic.0001-01-01.nc

[mnlevy1981]

I won't say #245 fixes this issue, but it gives us some time to explore actual solutions -- we definitely want to set AUTO_MASKTABLE=TRUE for some configurations once it plays nicely with the test suite and any other issues are addressed.

[alperaltuntas]

@mnlevy1981, I believe you need the following FMS patch: ESCOMP/FMS#6. Could you confirm that PR fixes the issue you've encountered? I'll then merge and tag it.

Also, setting AUTO_MASKTABLE to False turns off both land block elimination and auto parallel IO. We'd only want to disable the latter. To do so, we need to set TARGET_IO_PES = 1 and auto_merge_nc = .false. (in input.nml).

Just a heads-up: Although I’m not currently aware of any remaining issues, I’m still considering disabling parallel I/O for beta06 out of an abundance of caution. But I'd disable it for only the t232 and non-test configurations. I may submit a PR to MOM_interface before beta06.

[mnlevy1981]

@alperaltuntas I thought I had updated FMS, but apparently not. I'm rerunning my tests with your FMS branch and AUTO_MASKTABLE=TRUE. I'll close #245 because it sounds like it's not the right way to turn this off anyway.

If the FMS PR fixes the issues I saw with parallel I/O and the test suite, I'd suggest turning it on for testing and also for any case with MARBL enabled (we can talk about that implementation later)

[mnlevy1981]

@alperaltuntas I thought I had updated FMS, but apparently not. I'm rerunning my tests with your FMS branch and AUTO_MASKTABLE=TRUE.

My tests with the FMS updates only copied complete netCDF files to the baselines directory, thanks for pointing that out!