wrong ocean_vgrid.nc in all 0.1 deg runs

[aekiss]

There's an assumption that the values in topog.nc are all from the even-index (counting from 0) entries in ocean_vgrid.nc, but topog.nc is stored at single precision whereas ocean_vgrid.nc is double-precision. This difference can mess up this comparison, sometimes resulting in partial cells that are thicker than full cells because kmt is not incremented when it should be.

To avoid this problem, this specially-adjusted file must be used with KDS75: /g/data3/hh5/tmp/cosima/bathymetry/ocean_vgrid.nc
which is compatible with both single- and double-precision arithmetic.

All the 0.1deg input versions /short/public/access-om2/*/mom_01deg/ocean_vgrid.nc use the same, incorrect ocean_vgrid.nc, which differs from the correct /g/data3/hh5/tmp/cosima/bathymetry/ocean_vgrid.nc.

For 01deg_jra55v13_iaf this only makes a small difference (although for a lot of cells):

number of cells with partial > full thickness at bottom = 576032
max partial-full thickness at bottom = 0.00020424750073289033
min partial/full thickness at bottom = 0.19999809021620032
max partial/full thickness at bottom = 1.000001057895908

but for 01deg_jra55v13_ryf9091 the partial thickness is almost 1m greater than the full thickness at over 40,000 points, which can be about 1.5x thicker than the full thickness:

number of cells with partial > full thickness at bottom = 406464
max partial-full thickness at bottom = 0.9999998038009146
min partial/full thickness at bottom = 0.19999809021620032
max partial/full thickness at bottom = 1.5071401140150567

because we changed the minimum thickness min_thick from 10m to 1m with a new topography (/short/public/access-om2/input_08022019/mom_01deg/topog.nc) to fix #99. This is a map of the thickness difference (partial-full) where it is not negative. It's either 0 or 1m. This can be a large relative error in these shallow regions.

This script was used to detect this problem.

Credit to @russfiedler for diagnosing the underlying cause. Russ says the main consequence is that there could be some non advective cells introduced since kmt doesn't extend deep enough.

[aekiss]

note to self: private slack discussion with Russ is here: https://arccss.slack.com/archives/DKVPAUV3K/p1565938564001300

[aekiss]

Just linking to another problem with input_08022019: #159

[aekiss]

@russfiedler I've run some tests in /g/data3/hh5/tmp/cosima/access-om2-01/01deg_jra55v13_ryf9091_topo_bug_test.

output002 behaves correctly using correct ocean_vgrid.nc but default min_thickness = 1.0

output001 uses bad ocean_vgrid.nc and min_thickness = 0.5 and has many cells with excess thickness, but the excess is small:

number of cells with partial > full thickness at bottom = 390987
max partial-full thickness at bottom = 0.00020424750073289033

so setting min_thickness = 0.5 greatly improves the problem but doesn't eliminate it

[aekiss]

I've copied /g/data3/hh5/tmp/cosima/bathymetry/ocean_vgrid.nc into /short/public/access-om2/input_rc/mom_01deg/

[russfiedler]

The second case behaves at I think it is intended. It detects rounding errors and makes a larger than usual cell. I think you'll find that the max value is max(zeta(new)-zeta(old)) i=1:151:2

[aekiss]

The default min_thickness=1 in the code is evidently dangerous and should be set to something smaller, say 1e-4.

Or will there be roundoff problems if it it too small relative to ocean max depth?

[aekiss]

Is the problem with min_thickness=1m exacerbated by the fact that the underlying GEBCO bathymetry has a 1m resolution? (I know it is then spatially averaged, but it's very flat over those northeastern Siberian shelves)

[aekiss]

scratch that last comment

[russfiedler]

I'd make the min_thickness>spacing(1.e4) which is just under 1mm. This should eliminate all problems for depths < 1e4 m maybe make it a bit larger to be safe.

[aekiss]

OK, 1e-3 it is then.

The most recent ACCESS-CM2 diag table I've seen had min_thickness=25.0 - is that cause for concern?

[russfiedler]

This is only a problem for grids using the Mosaic style grid_spec.nc. CM2 uses Version 2 I think but it is something to keep in mind.

[aekiss]

@AndyHoggANU this seems a relatively small problem, but should it be fixed in 01deg_jra55v13_ryf9091 before you start the reference runs with extra diags?

@russfiedler will a change to the kmt mask midway through a simulation be problematic for restarts?

[AndyHoggANU]

Actually, I would prefer to run it with the new mask for a year or so before I start putting out diagnostics ... but not sure I will have time to do this before I go away next week. Maybe we should first try to figure out if we can easily restart with the new mask and take it from there?

[russfiedler]

Yes, it can be a bit tricky at times when you dig out tiles for the new mask. Care needs to taken with not only the tracers but also the thickness restart I think.

[aekiss]

Any objections to me merging the updated PR mom-ocean/MOM5#309 into master?
See mom-ocean/MOM5#303 (comment)

[aekiss]

Closing this issue - I've merged the updated PR mom-ocean/MOM5#309 into master.

[russfiedler]

The incorrect vgrid.nc is still being distributed for the 0.1 degree model. /g/data4/ik11/inputs/access-om2/input_08022019/mom_01deg/ocean_vgrid_dz_1_to_200m_6000m_75lev.nc. This is not the same as /g/data/hh5/tmp/cosima/bathymetry/ocean_vgrid.nc which is needed if above problems are to be avoided.

[aekiss]

This has been fixed in the ak-dev branches of the configs, but these are yet to be merged into master, so we should leave this issue open until they are.

[aekiss]

To be specific, it is fixed in /g/data4/ik11/inputs/access-om2/input_rc/mom_01deg, which is what the ak-dev branches of the configs use.

This is almost ready for testing and use. Details are in /g/data4/ik11/inputs/access-om2/input_rc/README.txt. 1st order conservative weights is the only outstanding issue: #71 (comment)

[aekiss]

closing.
/g/data/ik11/inputs/access-om2/input_20200530/mom_01deg/ocean_vgrid.nc is correct, and is linked to by the latest
/g/data/ik11/inputs/access-om2/input_20201022/mom_01deg/ocean_vgrid.nc which the latest ak-dev configurations use.

[aekiss]

For future reference, this is the tool to fix the doubles in ocean_vgrid.nc so they are exactly representable in single precision:
https://github.com/COSIMA/topogtools/blob/master/float_vgrid.f90