Bugfix/smith84/1754 openmp

[smithsg84]

Fix for compilation failure when downstream library/application is built without OpenMP (SYCL) when RAJA was installed with OpenMP (SYCL).

• This PR is a bugfix
• It does the following:
  • Refactors mutex to address ODR issues related to #ifdef OpenMP code.
  • Refactors OpenMP compilation guards into RAJA_ENABLE_OPENMP (flag for OpenMP use when building RAJA) and RAJA_OPENMP_ACTIVE (flag for OpenMP in application/library using RAJA). This matches similar flags for CUDA/HIP.
  • Fix compilation guards for SYCL, some ENABLE_SYCL guards changed to SYCL_ACTIVE.
  • Fixes Don't require downstream projects to enable OpenMP #1754 Investigate use of C++ std library use to replace OpenMP locking #1948

This was tested by compiling example RAJA applications from the RAJA/examples directory against RAJA installed with different options. Also tested compiling RAJAPerSuite without OMP (SYCL) against RAJA built with OMP (SYCL).

For discussion:

Steve is very concerned this simple fix does not fully address the all the issues. There is a violation of C++ ODR principle by having compilation of the same functions / structures be controlled by each RAJA library. If two libraries are built, say the "with-OMP" library and the "without-OMP" library and then linked together into a single RAJA app the functions / data structures with the #if defined(RAJA_OPENMP_ACTIVE) preprocessor logic will be different.

Example for problematic conditional logic:
https://github.com/LLNL/RAJA/blob/85e009758125c97717f1d637a90520839280b8a5/include/RAJA/policy/cuda/MemUtils_CUDA.hpp#L209

Possible fix would be to template on the presences/absence of OpenMP to disambiguate rather then preprocessor. Steve not sure ramifications of doing a template fix.

[rhornung67]

@smithsg84 please note that LC GitLab CI checks will not run on a PR that is in draft stage; therefore, the PR cannot be merged. If you want this to go through all CI checks, please click the "Ready for review" button.

[adayton1]

Do we know why omp::mutex was used instead of std::mutex to begin with? Was the omp version more performant?

[rhornung67]

I do not know. That code was developed at a time when the dpc++ compiler was being shaken out in terms of SYCL support. Maybe that was related to the choice.

[rhornung67]

@smithsg84 I fixed the code formatting issues and restarted the CI checks on this. Everything is green. Is it good to merge?

[rhornung67]

Folks, while this PR doesn't address all the issues, it is an improvement. We will make an issue for future work to be done. It is ready to merge. Please review.