[Feature] Add MESH expert residency with io_uring direct I/O for MoE inference

[RaQiu]

What does this PR do?

This PR introduces MESH, an experimental memory-tiered expert residency system for KTransformers MoE inference.

MESH is designed for heterogeneous CPU-GPU MoE serving when the full expert working set cannot stay comfortably resident in host DRAM. The current KT path works well when expert weights are available from normal memory-backed storage, but constrained-memory deployments expose a difficult systems problem: expert weights are sparse at runtime, but the full expert set must remain accessible. Relying only on mmap leaves expert residency mostly under OS page-cache control.

MESH adds an explicit runtime-managed expert residency layer for AMXINT4 MoE inference. It can load expert weights from NVMe through io_uring + O_DIRECT into NUMA-local CPU buffers, manage a bounded resident expert slot pool, and preserve the existing KT AMX compute path once the required experts are resident.

The goal is not to replace KT's AMX kernels. The goal is to make expert residency explicit: which experts are in CPU memory, which experts are cold, when cold experts are read, and how prefill/decode should share the resident slot pool.

Why this is useful

On constrained-memory machines, mmap makes MoE expert residency hard to control:

• The OS page cache can hold a hidden copy of expert weights.
• NUMA-local CPU execution may require a second application-managed copy.
• Page-cache reclaim timing is not visible to the runtime.
• Page faults happen synchronously on the critical path.
• The runtime cannot directly express that some experts should remain hot while others can be evicted.

MESH gives KTransformers a runtime-level mechanism to manage this explicitly:

• expert weights can be read directly into NUMA-local buffers,
• resident experts are tracked by layer and expert id,
• cold expert reads are issued through async io_uring,
• promotion and demotion are visible to the runtime,
• GPU-resident experts remain outside CPU resident-slot management,
• existing AMX compute kernels are reused after residency is resolved.

This is useful for local single-node MoE serving, workstation deployments, and memory-constrained CPU-GPU systems where CPU AMX, GPU attention, NUMA locality, and NVMe-backed expert storage need to work together.

Main components

1. io_uring direct-I/O expert loading

MESH adds an io_uring-based loading path for expert weights. With O_DIRECT, expert tensors can be read from storage directly into application-owned buffers, avoiding the OS page cache on the MESH path.

The async reader tracks request completion explicitly and validates read results. It also includes retry/validation logic for failed or incomplete reads.

2. NUMA-local resident expert slots

MESH adds a resident slot pool for CPU-managed experts. Each resident expert is associated with slot-owned buffers for its expert tensors. The slot metadata tracks expert state, slot state, and active readers so that a resident expert is not evicted while it is being used by the AMX forward path.

The slot pool is virtual: a slot is not permanently tied to a fixed expert id. It can be rebound to different experts as promotion and eviction happen.

3. Batched cold expert promotion

For a layer forward, MESH can collect the CPU experts needed by that forward pass, identify which ones are already resident, submit reads for the cold ones, wait for those reads to complete, bind the completed buffers into slots, and then call the existing KT AMX Base::forward() path.

This keeps the compute path close to KT's original implementation while moving expert residency decisions into an explicit runtime layer.

4. Cache policy and heat-aware residency

MESH keeps a bounded resident set and uses a policy-driven eviction path. The current implementation supports a SIEVE-style base policy and a heat/lookahead signal derived from router scores. The intent is to retain experts that are likely to be reused while allowing cold or low-value experts to be demoted.

The implementation also supports full-gate score observation and skips unnecessary observation when the effective resident capacity already covers all CPU-managed experts.

5. Deferred-expert aware decode behavior

MESH integrates with KT's deferred expert execution. Decode can distinguish resident experts from cold experts and issue prefetches for cold deferred experts. The accounting separates normal hits, cold misses, and in-flight misses so that async prefetch behavior is not mistaken for a fully cold miss.

6. Prefill residency support

MESH includes prefill-specific residency support. The current branch contains a prefill layer-window mode and transition logic back into decode hot-cache mode. This allows prefill and decode to interpret the resident slot pool differently while still sharing the same underlying slot abstraction.

The implementation also supports configurable early-layer residency through KT_MESH_EARLY_LAYER_EXPERTS, because early MoE layers can have different miss behavior from deeper layers.

7. GPU expert compatibility

MESH does not treat GPU experts as CPU resident-slot candidates. CPU-side residency logic checks the actual per-layer GPU expert mask and skips experts that are already assigned to GPU execution.

This matters because GPU experts are not necessarily a fixed prefix of expert ids and may vary by layer or under dynamic expert placement.

Compatibility

MESH is intended to be opt-in through the IOURING backend. The existing mmap/default path should continue to behave as before.

The implementation is designed to preserve KT's existing execution assumptions:

• AMX compute kernels are not replaced.
• The quantized expert layout is preserved.
• Existing CPU/GPU expert separation is respected.
• Existing KT forward computation is reused after expert residency is resolved.
• Non-IOURING paths should not depend on MESH residency state.

Current validation status

This work has been validated at two levels.

Automated/unit-level tests

The branch includes unit-level coverage for the async I/O layer, including:

• basic async reads,
• batch reads,
• multiple requests belonging to the same expert,
• timeout behavior,
• IO backend enum exposure,
• Python-to-C++ config conversion,
• short-read rejection.

The short-read test verifies that a completed-but-incomplete read is not treated as a successful request.

Manual/system-level validation

MESH has also been tested manually on a dual-socket AMX server with Qwen3.5-35B AMXINT4 weights, NVMe-backed expert storage, NUMA execution, and SGLang/KTransformers serving.

The manual validation included:

• IOURING backend startup and model serving.
• io_uring direct-read behavior under AMXINT4 expert loading.
• Deferred-expert decode behavior.
• hit / cold-miss / in-flight-miss accounting.
• per-layer and per-token expert hit-rate analysis.
• prefill behavior under multiple scheduling variants.
• early-layer full-residency configuration.
• GPU-expert mask compatibility with CPU resident-slot management.
• prefill timing and storage pressure analysis with iostat.
• server-side smoke tests for current MESH code paths.

Some of these experiments are saved under local paper/experiment artifact directories and are not suitable as normal CI tests.

The full MESH prefill/decode system test is not currently included as an automated per-commit test because it is hardware- and model-dependent: it requires AMX-capable CPUs, the AMXINT4 expert weight layout, NVMe-backed weights, NUMA configuration, and a running SGLang/KTransformers server.

This PR should be reviewed as an experimental systems path rather than a fully production-hardened default backend.

Fixes # (issue)

Before submitting

• Did you read the contributor guideline?
• Did you write any new necessary tests?

Note: automated tests exist for the async I/O layer. MESH has also been manually validated on the target AMX/NVMe/model-weight infrastructure, but those full system tests are not included in per-commit CI.

[gemini-code-assist]

Code Review

This pull request introduces a new Electron-based frontend for KTransformers, providing a GUI for model management, server control, and system monitoring. It also includes significant backend enhancements, such as io_uring support for expert weight loading, NUMA-aware memory management, and a tiered weight residency strategy to optimize performance when model sizes exceed physical RAM. Several security and performance issues were identified in the review, including insecure Electron settings, command injection risks in IPC handlers, and inefficient resource management in the hot path of the inference engine.

[gemini-code-assist]

Disabling webSecurity is a significant security risk in Electron applications as it bypasses the Same-Origin Policy. This allows the renderer process to make requests to any domain, which can be exploited if the application ever loads untrusted content. It is recommended to keep webSecurity enabled and handle cross-origin requirements via proper CORS configuration on the server or by using a proxy in the main process.

[gemini-code-assist]

Using shell: true with spawn can lead to command injection vulnerabilities if any part of the arguments is user-controlled. Since args is already an array, shell: true is unnecessary and should be set to false (which is the default) to execute the command directly without a shell. This improvement should be applied to all spawn calls in this file and other service files where shell: true is used.

Suggested change

	const proc = spawn('kt', args, { shell: true })
	const proc = spawn('kt', args, { shell: false })

[gemini-code-assist]

Performing heap allocations (posix_memalign or _aligned_malloc) inside the forward method for every prefill pass is a significant performance bottleneck and can lead to memory fragmentation. These buffers should be pre-allocated during initialization or managed through a reusable memory pool to ensure optimal performance in the hot path.

[gemini-code-assist]

There is a potential race condition in ExpertCacheStats. The configure_experts method reallocates the expert_access_count array while holding expert_mu, but the note_expert_access method (and others like note_expert_hit) accesses this array without any synchronization. If configure_experts is called while inference threads are active, it could lead to use-after-free or out-of-bounds access. Consider using a synchronization mechanism that protects the array pointer itself during access, or ensure that configuration only happens when no inference is running.

[gemini-code-assist]

Hardcoding python3 may cause the application to fail on Windows systems where the executable is typically named python. Consider detecting the platform or using a configurable path for the Python interpreter.

Suggested change

	const proc = spawn('python3', [scriptPath], { shell: false })
	const proc = spawn(process.platform === 'win32' ? 'python' : 'python3', [scriptPath], { shell: false })

[gemini-code-assist]

Reading environment variables using std::getenv inside the forward method is inefficient as it is called on every inference pass. These values should be read once during initialization and stored as member variables to avoid repeated string lookups in the hot path.

[gemini-code-assist]

Calling std::getenv in the hot path (forward_one) should be avoided. It is recommended to cache the value of environment variables during initialization.