v-storage

A flexible storage abstraction library for the Veda platform. Provides unified interface for different storage backends including memory, LMDB, MDBX, Tarantool, and remote storage.

🚀 Features

• Multiple Storage Backends: Memory, LMDB, MDBX, Tarantool (TTStorage), Remote storage
• Unified API: Common Storage trait for all backends
• 🔥 Zero-Copy Operations: Avoid unnecessary data copying with Cow<[u8]> for LMDB and MDBX
• ⚡ Optimized Primitives: Direct byte conversion without intermediate buffers for i64, u64, i32, u32
• 🔗 Unified Transaction API: Identical interface for LMDB and MDBX via ZeroCopyStorage trait
• Three Architecture Patterns: Dynamic dispatch, generic containers, enum dispatch
• Individual Support: Native support for Veda platform Individual objects
• Zero-cost Abstractions: Compile-time optimization options with minimal overhead
• Factory Patterns: Builder, Provider, and Config patterns for easy construction
• Error Handling: Comprehensive error types and result handling
• Backward Compatibility: Support for legacy API methods

📦 Installation

Add this to your Cargo.toml:

[dependencies]
v-storage = "0.1.0"

# Optional features
v-storage = { version = "0.1.0", features = ["tt_2", "tokio_0_2"] }

Available Features

• tt_2 - Tarantool 2.x support
• tt_3 - Tarantool 3.x support
• tokio_0_2 - Tokio 0.2 runtime support
• tokio_1 - Tokio 1.x runtime support

🏃 Quick Start

use v_storage::*;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create memory storage
    let storage = VStorage::builder()
        .memory()
        .build()?;
    let mut storage = VStorage::new(storage);

    // Store Individual with semantic predicates
    storage.put_value(StorageId::Individuals, "user:1", r#"{
        "@": "user:1",
        "rdf:type": [{"type": "Uri", "data": "foaf:Person"}],
        "rdfs:label": [{"type": "String", "data": "John Smith - Software Engineer"}],
        "foaf:name": [{"type": "String", "data": "John"}],
        "foaf:familyName": [{"type": "String", "data": "Smith"}],
        "foaf:age": [{"type": "Integer", "data": 30}],
        "foaf:mbox": [{"type": "String", "data": "john.smith@example.com"}],
        "veda:hasPosition": [{"type": "Uri", "data": "position:software_engineer"}],
        "org:memberOf": [{"type": "Uri", "data": "org:development_team"}]
    }"#)?;
    
    // Retrieve Individual
    if let StorageResult::Ok(data) = storage.get_value(StorageId::Individuals, "user:1") {
        println!("User Individual: {} bytes", data.len());
    }

    Ok(())
}

🏗️ Architecture

Storage Types

Type	Description	Dispatch	Use Case
VStorage	Dynamic dispatch with trait objects	Runtime	Maximum flexibility, runtime type selection
VStorageGeneric	Compile-time typed containers	Static	Type safety, known storage type
VStorageEnum	Static dispatch through enum	Static	Applications preferring enum dispatch

Storage Backends

• Memory Storage - In-memory HashMap-based storage
• LMDB Storage - Lightning Memory-Mapped Database (heed 0.22.0)
• MDBX Storage - Modern fork of LMDB with improved performance (libmdbx 0.6.3)
• Tarantool Storage - In-memory NoSQL database
• Remote Storage - Network-based storage client

StorageId Types

• Individuals - Main Individual objects for Veda platform
• Tickets - System tickets and tasks
• Az - Authorization and permission data

📖 Usage Examples

Basic Operations

use v_storage::*;

// Create storage using Builder pattern
let storage = VStorage::builder()
    .memory()
    .build()?;
let mut storage = VStorage::new(storage);

// Basic operations with semantic Individual data
storage.put_value(StorageId::Individuals, "person:example", r#"{
    "@": "person:example",
    "rdf:type": [{"type": "Uri", "data": "foaf:Person"}],
    "rdfs:label": [{"type": "String", "data": "Example Person"}],
    "foaf:name": [{"type": "String", "data": "Example"}]
}"#)?;
storage.get_value(StorageId::Individuals, "person:example")?;
storage.remove_value(StorageId::Individuals, "person:example")?;
storage.count(StorageId::Individuals)?;

Factory Patterns

// Builder Pattern - LMDB
let storage = VStorage::builder()
    .lmdb("/path/to/db", StorageMode::ReadWrite, None)
    .build()?;

// Builder Pattern - MDBX (recommended)
let storage = VStorage::builder()
    .mdbx("/path/to/db", StorageMode::ReadWrite, None)
    .build()?;

// Provider Pattern  
let storage = StorageProvider::memory();
let storage = StorageProvider::mdbx("/path/to/db", StorageMode::ReadWrite, None);

// Config Pattern
let config = StorageConfig::Memory;
let storage = VStorage::from_config(config)?;

// Config Pattern - MDBX
let config = StorageConfig::Mdbx {
    path: "/path/to/db".to_string(),
    mode: StorageMode::ReadWrite,
    max_read_counter_reopen: None,
};
let storage = VStorage::from_config(config)?;

Working with Individual Objects

use v_individual_model::onto::individual::Individual;

let mut individual = Individual::default();
let result = storage.get_individual(
    StorageId::Individuals, 
    "person:john", 
    &mut individual
);

match result {
    v_result_code::ResultCode::Ok => {
        println!("Individual loaded: {}", individual.get_id());
    },
    v_result_code::ResultCode::NotFound => {
        println!("Individual not found");
    },
    _ => println!("Error loading individual"),
}

Static Dispatch Usage

// Use VStorageEnum for static dispatch
let mut storage = VStorageEnum::memory();

// Batch operations with semantic Individual data
for i in 0..1000 {
    let individual_data = format!(r#"{{
        "@": "person:{}",
        "rdf:type": [{{"type": "Uri", "data": "foaf:Person"}}],
        "rdfs:label": [{{"type": "String", "data": "Person {}"}}],
        "foaf:name": [{{"type": "String", "data": "Person{}"}}],
        "veda:index": [{{"type": "Integer", "data": {}}}]
    }}"#, i, i, i, i);
    
    storage.put_value(
        StorageId::Individuals, 
        &format!("person:{}", i), 
        &individual_data
    )?;
}

🎯 Design Patterns

Strategy Pattern

Unified Storage interface allows switching between different storage implementations:

fn process_data(storage: &mut dyn Storage) {
    // Works with any storage implementation
    storage.put_value(StorageId::Individuals, "person:demo", r#"{
        "@": "person:demo",
        "rdf:type": [{"type": "Uri", "data": "foaf:Person"}],
        "rdfs:label": [{"type": "String", "data": "Demo Person"}]
    }"#);
}

Builder Pattern

Fluent API for configuring storage:

let storage = VStorage::builder()
    .memory()
    .build()?;

Abstract Factory

Multiple ways to create storage instances:

// Through provider
let storage1 = StorageProvider::memory();

// Through config
let storage2 = VStorage::from_config(StorageConfig::Memory)?;

// Through builder
let storage3 = VStorage::builder().memory().build()?;

📊 Architecture Comparison

Theoretical dispatch overhead (in practice, database I/O dominates):

Storage Type	Dispatch	Memory	Flexibility
VStorageEnum	Static (enum match)	Stack	Fixed set of types
VStorageGeneric	Static (monomorphization)	Direct	Compile-time known
VStorage	Dynamic (vtable)	Heap	Runtime selection

Note: In real applications, storage backend (LMDB, Tarantool, network) dominates timing.

🔧 Configuration

Memory Storage

let storage = VStorage::builder()
    .memory()
    .build()?;

LMDB Storage

let storage = VStorage::builder()
    .lmdb("/path/to/database", StorageMode::ReadWrite, Some(1000))
    .build()?;

MDBX Storage

// Modern LMDB alternative with better performance
let storage = VStorage::builder()
    .mdbx("/path/to/database", StorageMode::ReadWrite, None)
    .build()?;

// Or use generic version for static dispatch
let storage = StorageProvider::mdbx_generic(
    "/path/to/database", 
    StorageMode::ReadWrite, 
    None
);

Tarantool Storage

// Requires tt_2 or tt_3 feature
let storage = VStorage::builder()
    .tarantool("127.0.0.1:3301", "username", "password")
    .build()?;

Remote Storage

let storage = VStorage::builder()
    .remote("127.0.0.1:8080")
    .build()?;

Zero-Copy API (LMDB & MDBX) ⚡

For maximum performance, use the zero-copy API that avoids data copying:

use v_storage::lmdb_storage::LmdbInstance;
use v_storage::mdbx_storage::MdbxInstance;
use v_storage::common::{StorageMode, ZeroCopyStorage};

// Works identically for both LMDB and MDBX!
let mut storage = MdbxInstance::new("/path/to/db", StorageMode::ReadWrite);

// Put data
storage.put("key", b"value");

// Zero-copy read using transaction
let txn = storage.begin_ro_txn()?;
if let Some(data) = storage.get_with_txn(&txn, "key") {
    // data is Cow<[u8]> - usually Borrowed (no copying!)
    match data {
        std::borrow::Cow::Borrowed(slice) => {
            println!("Zero-copy read: {} bytes", slice.len());
            // Work with data without copying
        },
        std::borrow::Cow::Owned(vec) => {
            println!("Had to copy (data was modified)");
        }
    }
}
// Transaction drops, references are no longer valid

Benefits:

• LMDB: Always returns Cow::Borrowed - pure zero-copy
• MDBX: Returns Cow::Borrowed when possible, Cow::Owned only for modified data
• Unified: Same API for both databases using ZeroCopyStorage trait
• Optimized: Primitive types (i64, u64, i32, u32) read without intermediate buffer allocation

Generic Zero-Copy API:

Both LMDB and MDBX implement the ZeroCopyStorage trait for unified access:

fn read_data<S: ZeroCopyStorage>(storage: &S, key: &str) {
    if let Ok(txn) = storage.begin_ro_txn() {
        if let Some(data) = storage.get_with_txn(&txn, key) {
            // Works with both LMDB and MDBX!
            println!("Read {} bytes", data.len());
        }
    }
}

// Use with either database
let lmdb = LmdbInstance::new("/path/lmdb", StorageMode::ReadWrite);
let mdbx = MdbxInstance::new("/path/mdbx", StorageMode::ReadWrite);

read_data(&lmdb, "key");  // Same code
read_data(&mdbx, "key");  // Same code

📚 Examples

The examples/ directory contains comprehensive examples:

• zero_copy_api.rs - Zero-copy operations with LMDB/MDBX ⚡
• unified_api_simple.rs - Unified API for both databases 🔗
• basic_usage.rs - Basic operations and API usage
• storage_types.rs - Comparison of different storage types
• factory_patterns.rs - Various construction patterns
• individual_operations.rs - Working with Individual objects
• mdbx_usage.rs - MDBX storage usage patterns

Run examples:

# New zero-copy examples
cargo run --example zero_copy_api
cargo run --example unified_api_simple

# Other examples
cargo run --example basic_usage
cargo run --example storage_types
cargo run --example factory_patterns
cargo run --example individual_operations
cargo run --example mdbx_usage

🛠️ Development

Building

# Basic build
cargo build

# Build all examples
cargo build --examples

# With all features
cargo build --features "tt_2 tokio_0_2"

# Release build
cargo build --release

Testing

# Run tests
cargo test

# Run tests with features
cargo test --features "tt_2 tokio_0_2"

# Run integration tests
cargo test --test integration_tests

Documentation

# Generate documentation
cargo doc --open

# Generate with examples
cargo doc --examples --open

🔍 Error Handling

The library uses comprehensive error types:

match storage.get_value(StorageId::Individuals, "key") {
    StorageResult::Ok(value) => println!("Value: {}", value),
    StorageResult::NotFound => println!("Key not found"),
    StorageResult::Error(e) => println!("Error: {}", e),
}

🧪 Testing

# Unit tests
cargo test

# Integration tests  
cargo test --test integration_tests

# Example tests
cargo test --examples

# All tests with release optimization
cargo test --release

📋 Requirements

• Rust: 1.70 or higher
• Operating System: Linux, macOS, Windows
• Dependencies: See Cargo.toml

Optional Requirements

• LMDB: System LMDB libraries for LMDBStorage (heed 0.22.0)
• MDBX: libmdbx 0.6.3 for MDBXStorage - modern LMDB fork with better performance
• Tarantool: Running Tarantool instance for TTStorage
• Network: Connectivity for RemoteStorage

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

1. Clone the repository
2. Install Rust 1.70+
3. Run cargo build
4. Run cargo test
5. Check examples with cargo run --example basic_usage

Code Style

• Use cargo fmt for formatting
• Use cargo clippy for linting
• Add tests for new features
• Update documentation

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Related Projects

• Veda Platform - Semantic data management platform
• v-individual-model - Individual object model
• LMDB - Lightning Memory-Mapped Database
• libmdbx - Modern LMDB fork with better performance and reliability
• Tarantool - In-memory database and application server

💡 FAQ

Q: Which storage type should I choose? A: Use VStorageEnum for applications preferring static dispatch, VStorageGeneric for type safety with known storage types, and VStorage for maximum flexibility.

Q: Can I switch storage backends at runtime? A: Yes, with VStorage dynamic dispatch. Generic types require compile-time selection.

Q: Is the library thread-safe? A: Storage instances are not thread-safe by default. Use appropriate synchronization for concurrent access.

Q: How do I handle network failures with RemoteStorage? A: The library returns StorageResult::Error for network issues. Implement retry logic in your application.

Q: Should I use LMDB or MDBX? A: MDBX is a modern fork of LMDB with improved performance, better reliability, and additional features. For new projects, MDBX is recommended. LMDB remains supported for compatibility with existing systems. Both support zero-copy operations with identical API.

Q: What are the performance optimizations? A: The library includes several optimizations:

• Zero-copy reads using Cow<[u8]> for LMDB/MDBX (avoid data copying when possible)
• Direct byte conversion for primitive types without intermediate buffer allocation
• Transaction-based API for efficient batch operations
• Unified interface allows code reuse between LMDB and MDBX

---