Update mam4xx to include fix for wet scavenging

[mjschmidt271]

This brings in changes to mam4xx's wet_dep.hpp that correct the size of a work array to fix out-of-bounds errors that have led to recent fails for the mam4_wetscav_standalone test.

Note: This PR points to a branch in mam4xx that has yet to be merged into main--I will force push when that associated PR is approved/merged.

[mjschmidt271]

@bartgol @mahf708 @AaronDonahue @jgfouca

Can't add reviewers, so please take a look when you have a chance!

[singhbalwinder]

Thanks, @mjschmidt271. Can you please also uncomment the wet scavenging test in the PR? it was commented out in #6882 PR

[mjschmidt271]

Thanks, @mjschmidt271. Can you please also uncomment the wet scavenging test in the PR? it was commented out in #6882 PR

Ah, good call--will do!

[bartgol]

Did you verify that with this mods we are now deterministic on GPU? If so, you could re-enable all mam4 tests that were previously disabled. The checks will fail since we don't have the up-to-date baselines, but at least we can then gen baselines right after merging this. Also, you should add some cmake code for the mam4 tests that run the np1_vs_npN tests, so we can already verify non-determinism there.

Edit: turns out we only run np1 on GPU, but I am planning to change that soon. Back when we tested on weaver, we had all builds in || on the same compute node, so we did not know if we had 1 or 2 gpus (3 builds for 4 GPUs), so we just hard-coded to 1 max rank. Now we run ONE build on a blake node, which has 2 GPUs, so we should run also np2 tests.

[bartgol]

There are also currently disabled mam tests in the multiple-process subfolders.

[mjschmidt271]

@bartgol - I don't believe this will affect the non-determinism in the mam4_aero_microphys_standalone test, but is restricted to the wet scavenging issue you'd previously mentioned with out-of-bounds access. As such, mam4_wetscav_standalone is re-enabled but not mam4_aero_microphys_standalone.

I know James is working on the microphysics non-det, though, so I'd guess he'll knock it out before I even have a chance to take a look again 🙂

And I will put those extra test cases next on my list--thanks for that info!

[bartgol]

@bartgol - I don't believe this will affect the non-determinism in the mam4_aero_microphys_standalone test, but is restricted to the wet scavenging issue you'd previously mentioned with out-of-bounds access. As such, mam4_wetscav_standalone is re-enabled but not mam4_aero_microphys_standalone.

I know James is working on the microphysics non-det, though, so I'd guess he'll knock it out before I even have a chance to take a look again 🙂

And I will put those extra test cases next on my list--thanks for that info!

Ah I see. Thanks!