UCP: Tune Alltoall pairwise posting depth

[jeffnvidia]

What

Replace the binary automatic choice of posting either one or all peer transfers with a coarse posting-depth selector. Explicit ALLTOALL_PAIRWISE_NUM_POSTS values remain authoritative.

if total_bytes <= 66000 or N <= 8: P = N
else if peer_bytes <= 64 KiB:       P = min(N, 32)
else if peer_bytes <= 1 MiB:        P = min(N, 16)
else if peer_bytes <= 4 MiB:        P = N for N <= 16, otherwise 8
else if peer_bytes <= 8 MiB:        P = min(N, 4)
else:                               P = 4 for N <= 32, otherwise 1

The existing 66,000-byte aggregate-message behavior is retained. The exact 64 KiB boundary is a separate per-peer threshold. Per-peer bytes use the same calculation as the progress path.

Why

The current automatic selector uses one outstanding send and receive for every team larger than 32 ranks once the aggregate message exceeds 66,000 bytes, while teams of at most 32 ranks post the full team. The former serializes small and medium peer transfers too aggressively; the latter can over-post medium and large transfers.

NUM_POSTS=P limits incomplete sends and receives independently, so intermediate depths provide useful admission control without changing peer order or the UCP data path.

Evidence

Retrospective streaming scoring over the existing tuning corpus gives:

• 55 matched cells at 4/8/16/32/64 ranks with the stable UCX protocol: +23.5% geometric mean versus the current selector, with no matched regression; geometric-mean oracle coverage improves from 76.5% to 94.4%.
• 44 matched cells with the default UCX protocol: +22.0%, with no matched regression. Eleven cells are excluded because the required candidate depth did not complete in that older crash-affected sweep.
• Two newer 32/64-rank streaming corpora: +34.5% and +36.1%, with no matched regression.

The selector intentionally preserves full posting for teams of eight ranks or fewer: throttling those teams did not provide a robust cross-protocol benefit, and it caused 5--25% regressions above 8 MiB per peer.

Fresh held-out Gaia validation used PPN8, streaming cadence, the default UCX protocol selection, and nine non-power-of-two sizes from 48 KiB through 12 MiB per peer. Performance scoring compares the explicit depth selected by this rule against the explicit depth selected by the old rule; the focused unit tests independently verify the automatic mapping at every boundary.

• At 64 ranks, the selected depth improves geometric-mean bandwidth by 104.1% over the old selector across nine cells, with no regression (individual changes: 0.0% to +429.3%).
• At 32 ranks, the initial six-way sequence was strongly position/bimodality sensitive. Four focused six-repeat scouts rotated every relevant configuration through every sequence position. Replacing those four noisy cells with the position-balanced measurements gives +22.3% geometric mean across nine cells, with no regression. The rotated 48/96/384/1536-KiB-per-peer cells were 0.0%, +9.1%, +28.8%, and +10.9% versus the old selector.

One optional P4 stress control at 64 ranks and 12 MiB per peer timed out; both the old and proposed selectors use P1 there and completed. Two of 66 position-rotated launches hit an existing UCX lanes_distance assertion before producing a timing; the other 64 launches completed and those two infrastructure failures are excluded.

Older Doris measurements have positive aggregate scores but mixed individual cells and anomalously low values in two 64-rank streaming cases. They are not included in the no-regression statements above, and fresh Doris validation was not available.

Impact

This changes only automatic posting-depth selection in the TL/UCP pairwise Alltoall algorithm. It does not change Alltoallv, topology-aware peer ordering, or user-configured posting depths.

Checks

• Added focused tests for every inclusive threshold, the 8/9, 16/17, and 32/33-rank transitions, valid range, and monotonicity.
• Exact-head full UCC/UCX/CUDA build and both focused tests pass.
• make dist passes, and the resulting source tarball contains the new selector header.
• The new header is listed in TL/UCP's Automake source manifest.
• All 16 exact-head GitHub checks pass: normal/ASAN builds and tests, codestyle, docs, standard/CUDA/ROCm clang-tidy, UCC perftest, and OMPI build/IMB.
• Compiled the changed source separately with UCC warning-as-error flags.
• Ran the selector boundary harness locally.
• git diff --check origin/master...HEAD.