MPI Implementation for hyperbolic-parabolic system and the P4est Mesh

[TJP-Karpowski]

MPI Implementation for parabolic system and the P4est Mesh

Hello Trixi-Team,
Currently, the hyperbolic-parabolic system seem not to be (MPI-ready)[https://discourse.julialang.org/t/nan-value-with-parallelization-on-trixi-jl-taylor-green-vortex-test-case/116765].
This PR implements this capability.
Peyvan et al. (2025) described the necessary adaptations , although I could not find any corresponding commits or repository. Thus, I implemented the described extension to the cache_parabolic part, enabling MPI parallelization for hyperbolic-parabolic systems by extending the cache_parabolic with element, interface, and boundary caches and implementing the required MPI logic to use the parabolic system with MPI parallelization.

Testcases

To test the implementation, the Taylor Green Vortex (TGV) testcases are reused in the MPI test sets. Further, I created another testcase with static refinement in two diagonal quadrants of the TGV domain to force MPIMortars to be tested.
I have verified that the local i.e., only thread-based parallelized, and the MPI code result in the same profiles of K and epsilon (-dK/dt) for both the baseline Mesh with homogeneous refinement:

and also for the unevenly refined mesh in examples/p4est_3d_dgsem/elixir_navierstokes_taylor_green_vortex_amr_mortar.jl

In these figures, the DNS data is taken from Zirwes et al. (2023) and is only included as a visual aid to assess plausibility.

Questions/Remarks

• I had some issues with local testing due to allocation errors. These seem to be caused by the Multi-Threading already on the main branch for MacOS. I would appreciate some feedback or pointers on how to ideally test the code efficiently.
• Furthermore, this is my first PR for Trixi, and it turned out to be larger than I had hoped. Although I tried to comply with style and coding conventions, I am happy for feedback and improvements.

Disclaimer

LLMs have been used to create the PR.

Funding Statement

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project Number 237267381 – TRR 150.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

❌ Patch coverage is 77.39726% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.26%. Comparing base (e6a0a8f) to head (891b71a).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
src/callbacks_step/amr.jl	0.00%	23 Missing ⚠️
src/solvers/dgsem_p4est/dg.jl	43.75%	9 Missing ⚠️
src/solvers/dgsem/indicators.jl	95.45%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2880      +/-   ##
==========================================
- Coverage   97.05%   96.26%   -0.78%     
==========================================
  Files         606      608       +2     
  Lines       47371    48378    +1007     
==========================================
+ Hits        45972    46571     +599     
- Misses       1399     1807     +408     

Flag	Coverage Δ
unittests	96.26% <77.40%> (-0.78%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[DanielDoehring]

Thanks for working on this, MPI readyness for NSF has been for long on my list.

We need you to break this PR down into multiple ones

1. 2D Conforming (No Mortars)
2. 2D Non-Conforming (Mortars)
3. 3D Conforming (No Mortars)
4. 3D Non-Conforming (Mortars)

Furthermore, I would ask you to carefully test this - I tried adding some more MPI tests in #2710 for the flux-differencing and shock-capturing routines, which were back then not really realiable.

[TJP-Karpowski]

Thank you for the feedback, I can split this into multiple PRs. Note that currently 2D MPI treatment is not implemented as for my usecase and understanding 2D cases often still fit on one machine. So if it is alright I would only implement 3D for now and split this into possibly these PRs:

1. IndicatorPositional: Add IndicatorPositional with MortarTestcase (no new feature only Indicator and Testcase in parabolic)
2. Cache_parabolic: Change cache_parabolic to the new layout without adding mpi treatment to ensure nothing breaks
3. MPI treatment 3D: Baseline MPI implementation for static grids.
4. MPI+AMR: MPI + AMR/Mortar grids.

Does that sound like a reasonable split for you? If so I will adapt it accordingly. Do you have some tips for testing, or ideas what you would like to see?

[DanielDoehring]

Hm, so while most 2D cases should still fit on one machine, some high resolution simulations (I am thinking about shock-boundary, shock- bubble/multiphase interaction) might quickly get limited due to memory access (we see essentially that multithreaded maybe scales good until 16 cores, but flattens off at that point). So I am very much in favor of adding a 2D implementation first, as this is much easier to debug.

What is IndicatorPositional all about? This is probably completely independent from MPI, right?

Regarding cache_parabolic: If it is needed to bring back this guy we should do it in a separate PR - but can we maybe restrict this to the MPI only case? For the current multi-threaded implementation, we managed to reduce this cache a lot.

[TJP-Karpowski]

IndicatorPositional is indeed indepentend it is just a AMR indicator which takes a f(x,t) and can then be used to do AMR in specific locations. In the testcase I use it to force some uneven refinment to test the mortars. But for example to refine a specific location e.g. near walls, I think it is generally useful. I can restrict the cache_parabolic to the parallel p4est cases for now. Then I can try:

1. Indicator and testcase (just tooling)
2. 2D MPI conforming (introducing cache adaptations for P4estMeshParallel{2/3} only)
3. 2D MPI mortar
4. 3D MPI conforming
5. 3D MPI mortar

Does that sound good?

[DanielDoehring]

Yeah sounds good!
The indicator positional sounds interesting - a good generalization of refinement_patches 👍

[TJP-Karpowski]

The PR has been split into #2881, #2886, #2888, and #2894. I close this one for now.