Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Removed OMP in pressure calculation to avoid race condition #243

Merged
merged 1 commit into from
Feb 17, 2023

Conversation

jmaerz
Copy link
Collaborator

@jmaerz jmaerz commented Feb 17, 2023

Hi @JorgSchwinger, Tomas and I, we yesterday identified a potential race condition that (among others) causing issues when running BLOM/iHAMOCC in single column mode. Further places in iHAMOCC seem to be in ocprod (settling routine - potential race condition due to k versus kdonor?) and in carchm (to be checked). In this PR, I only addressed the pressure calculation. Currently, single column mode with master is only advisable with flag -D openmp=disabled to get reproducible results.

@jmaerz jmaerz added the iHAMOCC Issue mainly concerns the iHAMOCC code base label Feb 17, 2023
@jmaerz jmaerz self-assigned this Feb 17, 2023
@TomasTorsvik
Copy link
Contributor

I suspect the single column case is more sensitive to these OpenMP thread race issues, since all the threads will try to access the same section of memory at the same time, which is less likely to happen for larger cases. However, the test shows that there is potential for random errors, so we should definitely try to eliminate them.

@JorgSchwinger
Copy link
Contributor

What is a race condition? Anyway I'll approve this, if it helps to get the single column to run.

@jmaerz
Copy link
Collaborator Author

jmaerz commented Feb 17, 2023

@JorgSchwinger with my own limited understanding: depending on the time the different OpenMP threads finish, the results vary (due to the interdependence between the k and k+1) - here it means that a split in the k loop results in rp(i,j,k+1)=dp(i,jkn) since rp(i,j,k) can still be 0. Tomas figured out that on Betzy, someone was setting the number of OMP threads to 1 which prevents from such race conditions (and results should then be identical). I tapped into this issue since I used the single_column setup with OMP.

@jmaerz jmaerz merged commit 5507719 into NorESMhub:feature-hamocc_beyond-CMIP6 Feb 17, 2023
@jmaerz jmaerz deleted the fix-rp-OpenMP branch November 23, 2023 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
iHAMOCC Issue mainly concerns the iHAMOCC code base
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants