MemBase
_{^{memory data mapping}}

Ultra-high-performance memmap library for Rust

     

MemBase is a cross-platform, memory-mapped I/O library engineered for performance-critical systems. Optimized to deliver maximum throughput for extreme memory mapping operations and heavy data manipulation with minimal overhead.

Performance Benchmarks

Sequential Read Performance:

• Standard I/O: 2,784 MB/s
• MemBase: 26,277 MB/s
• 9.4x faster than traditional I/O

Random Access Performance:

• Standard I/O: 397,634 ops/s
• MemBase: 1,509,185 ops/s
• 3.8x faster random access

Key Features

Zero-Copy Architecture

Direct memory access eliminates buffer copying and system call overhead, delivering raw performance for data-intensive applications.

Advanced Memory Management

Intelligent memory mapping strategies optimize for both sequential and random access patterns, automatically adapting to workload characteristics.

Cross-Platform Compatibility

Native support across Windows, macOS, and Linux with platform-specific optimizations under a unified API.

Safety-First Design

Memory-safe operations with Rust's ownership model, preventing common pitfalls like buffer overflows and use-after-free errors.

Integrated Database Operations

Built-in database functionality with optimized query execution and in-place file modifications for persistent data structures.

Use Cases

• High-Frequency Trading Systems - Microsecond-critical market data processing
• Database Engines - Storage layer optimization for OLTP and OLAP workloads
• Real-Time Analytics - Stream processing with persistent state management
• Game Engines - Asset loading and world state persistence
• Scientific Computing - Large dataset manipulation and analysis
• Log Processing - High-throughput log ingestion and analysis

Architecture

MemBase leverages OS-level memory mapping primitives with intelligent prefetching and cache-aware algorithms. The library automatically handles:

• Page fault optimization
• Memory pressure management
• Multi-threaded access coordination
• Atomic operations for concurrent modifications

Usage

Add this to your Cargo.toml:

[dependencies]
MemMap = "0.1.0"

Basic Example

use MemMap::{MmapOptions, Mmap};
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open a file
    let file = File::open("data.bin")?;

    // Create a memory map with default options
    let map = unsafe { MmapOptions::new().map(&file)? };

    // Access the memory map
    if map.len() >= 8 {
        let value = unsafe { *(map.as_ptr() as *const u64) };
        println!("First 8 bytes as u64: {}", value);
    }

    Ok(())
}

Advanced Example

use MemMap::{MmapOptions, Mmap, HugePageSize, PrefetchStrategy};
use MemMap::platform::Advice;
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Open a file
    let file = File::open("large_data.bin")?;

    // Create a memory map with advanced options
    let map = unsafe {
        MmapOptions::new()
            .read(true)
            .write(true)
            // Use huge pages if available
            .huge_pages(HugePageSize::TwoMB)
            // Use sequential prefetching
            .prefetch(PrefetchStrategy::Sequential)
            // Populate the mapping immediately
            .populate(true)
            // Map the file
            .map_mut(&file)?
    };

    // Advise the kernel about our access pattern
    map.advise(Advice::Sequential)?;

    // Use the memory map
    // ...

    // Flush changes to disk
    map.flush()?;

    Ok(())
}

---

📌 License

Licensed under the Apache License, version 2.0 (the "License"); you may not use this software, including, but not limited to the source code, media files, ideas, techniques, or any other associated property or concept belonging to, associated with, or otherwise packaged with this software except in compliance with the License.

You may obtain a copy of the License at: http://www.apache.org/licenses/LICENSE-2.0.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the LICENSE file included with this project for the specific language governing permissions and limitations under the License.

^{COPYRIGHT © 2025 JAMES GOBER.}