CMAP: Tiered Memory Research Platform

A research platform for exploring tiered memory management on Intel Xeon Max (HBM + DDR) and CXL systems.

Overview

CMAP provides an execution layer for tiered memory research. It doesn't replace existing observation mechanisms (TPP, DAMON) or policies, but accelerates the aggregation, scanning, and migration stages.

┌─────────────────────────────────────────────────────────────┐
│                    Existing Policies                        │
│              (TPP, DAMON, Chrono, Alto/SOAR)                │
└─────────────────────┬───────────────────────────────────────┘
                      │ hotness data
                      ▼
┌─────────────────────────────────────────────────────────────┐
│                    CMAP Library                             │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────────┐    │
│  │ Ingest  │→ │  Merge  │→ │  Scan   │→ │  Migrate    │    │
│  │         │  │ (SIMD)  │  │ (SIMD)  │  │ (move_pages)│    │
│  └─────────┘  └─────────┘  └─────────┘  └─────────────┘    │
│                                                             │
│  Columnar Metadata Store (HBM-pinned)                       │
│  ┌────────┬────────┬────────┬────────┐                     │
│  │ count[]│ idle[] │ tier[] │ pfn[]  │                     │
│  └────────┴────────┴────────┴────────┘                     │
└─────────────────────────────────────────────────────────────┘

Current Status

Phase 1 Complete - Core library implementation

Component	Status	Description
Columnar Store	✅	Hash-based PFN lookup, HBM-pinned buffers
SIMD Scan	✅	AVX-512 vectorized predicate evaluation
Migration	✅	move_pages() with rate limiting
TPP Adapter	✅	NUMA balancing integration
DAMON Adapter	✅	sysfs interface integration
Accelerators	✅	IAA/DSA abstraction with emulation fallback

Building

Requirements

• Linux kernel 5.15+ (for DAMON sysfs)
• GCC 10+ with AVX-512 support
• libnuma-dev
• CMake 3.16+

Build

mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

Build Options

cmake .. \
  -DCMAP_ENABLE_AVX512=ON \
  -DCMAP_ENABLE_IAA=OFF \
  -DCMAP_ENABLE_DSA=OFF \
  -DCMAP_BUILD_TESTS=ON

Quick Test (Standalone)

Platform-independent tests can run on any system:

cd tests/standalone
gcc -O2 -o test_standalone test_standalone.c
./test_standalone

Project Structure

cmap/
├── include/cmap/
│   ├── cmap.h          # Public API
│   ├── types.h         # Core types (tiers, columns, predicates)
│   └── error.h         # Error codes and logging
├── src/
│   ├── core/           # Column store, NUMA, buffer management
│   ├── scan/           # SIMD scan engine, bitset operations
│   ├── merge/          # Event aggregation
│   ├── migrate/        # Page migration, rate regulation
│   ├── ingest/         # Multi-source event ingestion
│   └── accel/          # IAA/DSA abstraction
├── adapters/
│   ├── tpp/            # TPP/NUMA balancing adapter
│   └── damon/          # DAMON adapter
└── tests/
    ├── unit/           # Unit tests (Linux)
    └── standalone/     # Portable tests

Research Direction

This platform enables research into:

1. Observation accuracy: How well do TPP/DAMON capture actual hotness?
2. HBM characteristics: When does HBM actually help vs DDR?
3. Migration overhead: What is the true cost of page migration?
4. Policy effectiveness: Where do existing policies fail?

See COMPETITIVE_ANALYSIS.md for detailed gap analysis.

API Example

#include <cmap/cmap.h>

// Initialize with HBM pinning
cmap_ctx_t *ctx = cmap_init(CMAP_FLAG_HBM_PIN | CMAP_FLAG_FALLBACK_SIMD);

// Scan for hot pages: count >= 10 AND idle <= 5
cmap_condition_t conds[] = {
    { CMAP_COL_COUNT, CMAP_OP_GE, 10 },
    { CMAP_COL_IDLE, CMAP_OP_LE, 5 }
};
cmap_predicate_t pred = { conds, 2, false };
cmap_bitset_t result = {0};

cmap_scan_columns(ctx->columns, &pred, &result, &ctx->stats);

// Migrate hot pages to HBM
cmap_migrate_pages(ctx, &result, CMAP_TIER_DDR, CMAP_TIER_HBM);

cmap_destroy(ctx);

Related Work

• TPP (ASPLOS 2023): Transparent Page Placement via hint faults
• DAMON (Linux): Data Access MONitor with region sampling
• HeMem (SOSP 2021): Software-managed tiered memory
• Memtis (SOSP 2023): Per-page access tracking
• Alto/SOAR: Migration regulation and AOL metrics

License

Apache-2.0

Authors

Research platform developed at ETRI.