MueLu: Nondeterministic aggregation is the default for OpenMP builds

[GrahamBenHarper]

Bug Report

@trilinos/muelu

This has actually appeared multiple times in the past few days, for example, some tedious debugging finally showed nondeterministic aggregation was preventing #10916 from merging, but also there was a brief amount of time where nondeterministic aggregation somehow merged into MueLu and started blocking PRs as noted in #11019.

I'm not sure exactly how this is happening, but it seems like a good idea to make sure the OpenMP build defaults to something deterministic. I'm guessing there was a reason for this decision originally (performance?), but we should look into whether we want a default behavior in MueLu to be non-reproducible.

This might also be a good standup discussion topic.

[brian-kelley]

The history of that is mostly because non-deterministic aggregation (based on D2 coloring) was implemented first. When deterministic was added, I imagine performance was the reason the default wasn't updated.

Now, if you want both parallel performance (significantly faster than the non-deterministic default) and determinism, and don't care about setting min/max agg sizes, use "aggregation: coloring algorithm" = "mis2 aggregation". Generally the aggregates are very similar to non-Kokkos serial agg with default settings.

[GrahamBenHarper]

Awesome, thanks! I've been doing the dumb thing with "aggregation: deterministic" = true, but that's good to know that mis2 is also an appropriate choice. It's also good to know that performance was likely the reason... the unfortunate thing is that changing default behavior is really backwards-incompatible, but OpenMP build's default behavior is, from what I understand, different from the default behavior in other configurations. I want to know if anybody else thinks it's really something worth fixing/changing.

[brian-kelley]

I don't think the default behavior is backend-dependent actually. Yes, KokkosKernels has different default coloring algos per backend, but the MueLu master list sets "aggregation: coloring algorithm" to "serial", overriding the KokkosKernels default.

(obviously though, if you're running on Serial backend, even with "aggregation: deterministic" = false, it will end up being deterministic. So that's a differerence)

[cgcgcg]

I think the coloring itself is by default deteministic. Our defaults are "aggregation: deterministic"=false, but also "aggregation: coloring algorithm"=serial. We use the value of "aggregation: deterministic" also in the different phases of aggregation, and that's where the non-deterministic behavior comes in.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[rppawlo]

Was this ever completed? It would really be helpful to have a deterministic default from an application's viewpoint.

[GrahamBenHarper]

@rppawlo I've been hesitant to pull the trigger because a change in backwards compatibility will probably cause a bit of confusion, even though I believe it's the correct thing to do. Let me double-check with the rest of @trilinos/muelu and make sure it's fine

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.