Conversation
SUPA support was only wired into rdma_context and cuda_alike; every other vendor-conditional omitted USE_SUPA, and gpu_vendor/supa.h mapped CUDA names to non-existent `supa*` symbols, so it never compiled against a real SUPA SDK. Transfer Engine: - Discover GPU topology and classify device memory for SUPA (topology.cpp, memory_location.cpp). - Enable the TCP GPU staging path for SUPA. - Build examples and the TCP transport test with SUPA. - Rewrite gpu_vendor/supa.h against the real SUPA API: runtime symbols use the `su` prefix, driver symbols use `sudrv`, driver enums use their `su*` names, and suMemcpyAsync takes the stream before the kind (wrapped to preserve the CUDA argument order). Mooncake Store: - Add AcceleratorVendor::kSupa and register the CUDA-like accelerator device. Build: - Include the SUPA SDK as a system include directory so its generic headers (e.g. version.h) do not shadow Mooncake headers. - Discover the runtime library by name (libsupa/libsupa-runtime) instead of hardcoding it, and link the resolved libraries. Docs: document -DUSE_SUPA and BIREN_HOME in build.md, quick-start.md and supported-protocols.md. Tested on 8x Biren166M + 8x RoCE: full TE and Store build and link with -DUSE_SUPA=ON against the SUPA SDK (GPU allocation verified at runtime); the default build without USE_SUPA is unchanged.
Dashener2
requested review from
ShangmingCai,
UNIDY2002,
XucSh,
YiXR,
alogfans,
chestnut-Q,
doujiang24,
staryxchen,
stmatengss and
ykwd
as code owners
September 24, 2026 12:15
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SUPA support was only wired into rdma_context and cuda_alike; every other vendor-conditional omitted USE_SUPA, and gpu_vendor/supa.h mapped CUDA names to non-existent
supa*symbols, so it never compiled against a real SUPA SDK.Transfer Engine:
suprefix, driver symbols usesudrv, driver enums use theirsu*names, and suMemcpyAsync takes the stream before the kind (wrapped to preserve the CUDA argument order).Mooncake Store:
Build:
Docs: document -DUSE_SUPA and BIREN_HOME in build.md, quick-start.md and supported-protocols.md.
Tested on 8x Biren166M + 8x RoCE: full TE and Store build and link with -DUSE_SUPA=ON against the SUPA SDK (GPU allocation verified at runtime); the default build without USE_SUPA is unchanged.
Description
Biren (SUPA) support existed only as a skeleton:
rdma_context.cppandcuda_alike.hknew aboutUSE_SUPA, but every other vendor-conditional omittedit, and
gpu_vendor/supa.hmapped the CUDA API ontosupa*symbols that do notexist in the real SUPA SDK (
suGetDeviceCount,suMallocDevice,sudrvMemCreate,...). As a result the
-DUSE_SUPA=ONbuild could not compile, and even if it had,GPU topology discovery, device-memory classification, the TCP staging path, and
the Store accelerator device would all have fallen back to CPU paths.
This change completes the Biren integration so
-DUSE_SUPA=ONbuilds and worksend to end:
topology.cpp: includeUSE_SUPAin the GPU topology guard sosupa:Nentries are discovered and NIC/GPU affinity applies.
memory_location.cpp: classify Biren device memory viacuPointerGetAttributesinstead of treating it as host memory.
tcp_transport.cpp/tcp_transport_session_impl.h: enable GPU-awarestaging for SUPA.
example/*andtests/tcp_transport_test.cpp: build the GPU paths under SUPA.gpu_vendor/supa.h: rewrite against the actual SUPA SDK. Runtime API usesthe
suprefix, driver API usessudrv, driver enums use theirsu*names,NVIDIA-only fabric/VMM concepts are dropped, and
suMemcpyAsync(which takesthe stream before the kind) is wrapped to keep the CUDA call order.
AcceleratorVendor::kSupa, register the CUDA-like accelerator device, andenable the cuda-like registry path for SUPA.
${BIREN_HOME}/supa/includeas a system include dir: the SDK shipsgeneric headers (notably
version.h) that otherwise shadow Mooncake's owngenerated
version.h.libsupafor older SDKs,libsupa-runtimefor current ones) and link the resolved library set.build.md,quick-start.mdandsupported-protocols.md, documenting-DUSE_SUPA=ONandBIREN_HOME.No related issue; the work aligns with the "Domestic GPU Support" milestone of
the overall roadmap.
Module
mooncake-transfer-engine)mooncake-store)mooncake-conductor)mooncake-reshard)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Tested inside a Biren SUPA dev container on a host with 8x Biren166M (64 GB each),
Hygon C86-4G CPU and 8x RoCE NICs, using SUPA SDK 1.10.
Test commands:
Test results:
Unit tests pass
Integration tests pass (if applicable)
Manual testing done (describe below)
-DUSE_SUPA=ON: 0 compiler errors.libtransfer_engine.aandlibmooncake_store.alink, and all examples/tests link; the binaries resolvelibsupa-runtime.so.1at runtime.Runtime smoke test: the target allocates a 256 MiB buffer in Biren VRAM
("VRAM is used" / "Allocating memory on GPU 0") and the TCP transport starts
cleanly, confirming the
su*API mapping and the newmemory_location/topologypaths compile and run.Default build (
USE_SUPA=OFF): 0 compiler errors, no regression toexisting backends.
Not covered here: RDMA end-to-end transfer could not be exercised inside the
container because the
zrdma/dinghai10eNIC userspace provider is notinstalled there (
ibv_devicesis empty), andrdma_transport_test/tcp_transport_testwere built but not executed. A follow-up run on a barehost (or an image with the vendor RDMA provider) is recommended before
merging.
Checklist
./scripts/code_format.shNotes:
pre-commit run --files ...passes on all changed files(trailing-whitespace, end-of-file-fixer, check-merge-conflict,
check-added-large-files, clang-format, codespell and cmake-format); the Python
and Rust hooks are skipped because no Python/Rust files are touched. No new unit
tests were added; the change is compile-gated enablement, and the existing
tcp_transport_test/rdma_transport_test/ examples now build underUSE_SUPA. The change is ~264 insertions, under the 500 LOC RFC threshold.AI Assistance Disclosure
An AI coding assistant (opencode, deepseek-flash) was used to locate the missing
USE_SUPAbranches, rewritegpu_vendor/supa.hagainst the real SUPA SDKsymbols, adjust the CMake, and run the containerized build/regression
validation. All changed lines were reviewed by the submitter, who can defend the
change end to end.