Enable array reshaping for better inlining on NVHPC

[prathi-wind]

Description

Many functions were unable to be inlined by the NVHPC compiler because there was a reshape array option that was not enabled for inlining.
Fixes #(issue) [optional]

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• [ X] Something else

Scope

• [X ] This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

• Test A
• Test B

Test Configuration:

• What computers and compilers did you use to test this:

Checklist

• I have added comments for the new code
• I added Doxygen docstrings to the new code
• I have made corresponding changes to the documentation (docs/)
• I have added regression tests to the test suite so that people can verify in the future that the feature is behaving as expected
• I have added example cases in examples/ that demonstrate my new feature performing as expected.
  They run to completion and demonstrate "interesting physics"
• [ X] I ran ./mfc.sh format before committing my code
• New and existing tests pass locally with my changes, including with GPU capability enabled (both NVIDIA hardware with NVHPC compilers and AMD hardware with CRAY compilers) and disabled
• This PR does not introduce any repeated code (it follows the DRY principle)
• I cannot think of a way to condense this code and reduce any introduced additional line count

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran a Rocprof Systems profile using ./mfc.sh run XXXX --gpu -t simulation --rsys --hip-trace, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature

[prathi-wind]

Currently, this compiles for when in debug mode, but fails to compile when in no-debug

[sbryngelson]

I would imagine that debug mode disables any inlining at all, certainly reshaping. That said, the error is strange. It fails compiling pre_process? pre_process doesn't use openacc statements...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 43.47%. Comparing base (2f8eef1) to head (0e90d6e).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #858   +/-   ##
=======================================
  Coverage   43.47%   43.47%           
=======================================
  Files          68       68           
  Lines       19766    19766           
  Branches     2375     2375           
=======================================
  Hits         8593     8593           
  Misses       9726     9726           
  Partials     1447     1447           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sbryngelson]

here's the error:

NVFORTRAN-S-0275-Unable to open file pre_process_lib/s_compute_pressure-5-oToBA while extracting or inlining (/storage/coda1/d-coc/0/sbryngelson3/runners/actions-runner-02/_work/MFC/MFC/build/staging/4aa13650e3/fypp/pre_process/m_variables_conversion.fpp.f90: 1058)

[sbryngelson]

Found this

Your findings are correct:
The -Minline=reshape flag in the NVIDIA HPC compilers (nvc, nvc++, nvfortran) allows the compiler to inline Fortran code even when the array shapes do not match between the caller and the callee. This is a more aggressive inlining than the default behavior, which typically requires array shapes to match for correctness and safety.

Consequences of Using -Minline=reshape

Potential Risks:

• Correctness: Allowing inlining when array shapes do not match can break code if the underlying assumptions about array shapes are violated. Fortran's array semantics rely on shape consistency, and bypassing this can lead to memory errors, incorrect results, or runtime crashes if the code depends on shape matching for correctness.
• Undefined Behavior: If your code or libraries rely on the Fortran standard's requirement for matching array shapes, using this flag could introduce subtle bugs that are hard to detect and debug.
• Portability: Code compiled with this flag may behave differently than on other compilers or without the flag, reducing portability and increasing maintenance complexity.

When Is It Safe?

• If you are certain that the shape mismatch will not cause issues in your specific code paths (for example, if only the data content matters and not the shape metadata), it may be safe. However, this is rarely the case in general Fortran code.
• For most users, especially with large or legacy codebases, enabling this flag is not recommended unless you have a strong reason and have thoroughly tested your application.

Is It Likely to Break Code?

• Yes, it is possible. Since the flag relaxes a key safety check in Fortran's array handling, it can break code that assumes shape matching. Bugs may not always be immediately obvious and could manifest as incorrect results or segmentation faults.

Recommendation

• Only use -Minline=reshape if you have a specific performance need, a deep understanding of your code's array usage, and have done extensive testing.
• For most users, avoid this flag to ensure correctness and maintainability.

"Allows inlining in Fortran even when array shapes do not match."

Summary:
Using -Minline=reshape can break code that depends on matching array shapes and should be used with caution. It is not generally safe for all codebases.

[1] https://docs.nvidia.com/hpc-sdk/compilers/hpc-compilers-user-guide/index.html
[2] https://docs.nvidia.com/hpc-sdk/
[3] https://scienceit-docs.lbl.gov/hpc/software/compilers/nvhpc/
[4] https://docs.discoverer.bg/nvidia_hpc_sdk.html
[5] https://rci.stonybrook.edu/HPC/software/nvhpc
[6] https://docs.alcf.anl.gov/polaris/compiling-and-linking/nvidia-compiler-polaris/
[7] https://hpc.cea.fr/tgcc-public/en/html/toc/fulldoc/Parallel_programming.html
[8] https://www.osc.edu/resources/available_software/software_list/nvhpc
[9] https://docs.icer.msu.edu/Compiling_for_GPUs/
[10] https://ncar-hpc-docs.readthedocs.io/en/latest/compute-systems/derecho/compiling-code-on-derecho/