v1.2

VeloxDB

This is the FINAL version of using SstFileManager to handle data management cross memory and disk/ssd.

In this version, the B-plus tree structure has been well tested. Each Page instance has a fixed size (4 KB as default). Each Internal Node in the B+ Tree has been serialized, and written into a single InternalPage. All the kv pairs (Leaf Nodes in B+ Tree) will be stored sequentially into LeafPage.

In the incoming version, LSM-Tree will be used for integrating Memtable and SSTs.

Database Operations

VeloxDB::Open(string db_name)

Initializes the database system, setting up the necessary files and directories (including SSTs and related data). Can be initialized with a custom Memtable size or default size of 1e3.

#include "VeloxDB/VeloxDB.h"
/*
 *  Initialize with default value : 
 *       Memtable::size == 1e3
 */ 
auto MyDBDefault = std::make_unique<VeloxDB>();
auto MyDBDefault = std::make_unique<VeloxDB>(int memtableSize);

MyDBDefault->Open("database_name");

VeloxDB::Close()

Closes the database, flushing any data in memory (Memtable) to disk and storing it in SSTs.

#include "VeloxDB/VeloxDB.h"
// Close the database and flush the Memtable to disk
auto MyDB = std::make_unique<VeloxDB>();
MyDB->Open("database_name");
MyDB->Close();

Data Operations

Template<typename K, typename V> VeloxDB::Put(K key, V value)

Inserts a key-value pair into the database, where both the key and value can be of various types (int, double, string, etc.).

#include "VeloxDB/VeloxDB.h"
// Example of inserting different data types
auto MyDB = std::make_unique<VeloxDB>();
MyDB->Open("database_name");
MyDB->Put(1, 100);             // int -> int
MyDB->Put(1.5, 'A');           // double -> char
MyDB->Put("Hello", 1e8LL);     // string -> long long

VeloxDB::Get(const KeyValueWrapper& key)

Retrieves a value from the database based on the key. Supports multiple data types.

#include "VeloxDB/VeloxDB.h"
#include "kv/KeyValue.h"
// Example of retrieving values
auto MyDB = std::make_unique<VeloxDB>();
MyDB->Open("database_name");
MyDB->Put(1, 100);
MyDB->Put(1.5, 'A');
MyDB->Put("Hello", 1e8LL);

// Retrieve the value by key
auto result1 = MyDB->Get("Hello");
long long value1 = result1.kv.long_value(); // 1e8
string key1 = result1.kv.string_key(); // "Hello"

// Retrieve the value by `KeyValueWrapper` instance
auto result1 = MyDB->Get(KeyValueWrapper("Hello", "")); 
// Expected result1: { key: "Hello", value: 1e8LL }
long long value1 = result1.kv.long_value(); // 1e8
string key1 = result1.kv.string_key(); // "Hello"

// e.g.2
auto result2 = MyDB->Get(1);
int value2 = result2.kv.int_value();
int key2 = results.kv.int_key();

// check if not found using : bool KeyValueWrapper::isEmpty() const;
if(result.isEmpty()){
    // Key not found :-(
} else {
    // Found :-D
}

VeloxDB::Scan(KeyValueWrapper smallestKey, KeyValueWrapper largestKey)

Scans the database for key-value pairs within a specified key range. The results are returned in sorted key order.

#include "VeloxDB/VeloxDB.h"
// Scan for key-value pairs within a range
auto MyDB = std::make_unique<VeloxDB>();
MyDB->Open("database_name");
// Scan by key
std::set<KeyValueWrapper> results = MyDB->Scan(1, 10);
// Scan by `KeyValueWrapper` instance
std::set<KeyValueWrapper> results = MyDB->Scan(KeyValueWrapper(1, ""), KeyValueWrapper(10, ""));

VeloxDB::Update(KeyValueWrapper KeyToUpdate) (TBA)

This will allow the updating of key-value pairs within the database.

VeloxDB::Delete(KeyValueWrapper Key) (TBA)

This will allow the deletion of key-value pairs from the database.

Buffer Pool Operation

setBufferPoolParameters(size_t capacity, EvictionPolicy policy)

Set/reset buffer pool size_t:: capacity and EvictionPolicy:: policy (LRU, CLOCK, RANDOM)

EvictionPolicy newPolicy = EvictionPolicy::LRU;
EvictionPolicy newPolicy = EvictionPolicy::CLOCK;
EvictionPolicy newPolicy = EvictionPolicy::RANDOM;

Warning

This method will clear all the previous cache in the buffer pool.

#include "VeloxDB/VeloxDB.h"
#include "Memory/BufferPool/BufferPool.h"
// Open the database
auto MyDB = std::make_unique<VeloxDB>();
MyDB->Open("database_name");

// Set buffer pool parameters
size_t Capacity = 1e3;
EvictionPolicy Policy = EvictionPolicy::CLOCK;
MyDB->SetBufferPoolParameters(Capacity, Policy);

// Reset 
size_t newCapacity = 1e4;
EvictionPolicy newPolicy = EvictionPolicy::LRU;
MyDB->SetBufferPoolParameters(newCapacity, newPolicy);

// Perform database operations
MyDB->Put(1, "value1");
KeyValueWrapper value = MyDB->Get(1);

// Close the database
MyDB->Close();

printCacheHit()

print total number of cache hit in the buffer pool during the database operations.

#include "VeloxDB/VeloxDB.h"

int memtableSize = 1e4; 
auto db = std::make_unique<VeloxDB>(memtableSize);
db->Open("test_db");

const int numEntries = 1e6;  // Insert 1e6 key-value pairs

// Insert key-value pairs
for (int i = 0; i < numEntries; ++i) {
    db->Put(i, "value_" + std::to_string(i));
}

std::set<KeyValueWrapper> resultSet = db->Scan(KeyValueWrapper(1, ""), KeyValueWrapper(50000, ""));

db->printCacheHit(); // this will print the total number of cache hit in buffer pool

// Clean up
db->Close();

Benchmark

Note

Hardware Resources

System:     macOS Sonoma Version 14.3.1
Chip:       Apple M3 Max
Memory:     48 GB

SST Files Layout

Supported Data Types

Supported Language

Supported Platforms and Compilers

The KV-Store system has been tested across multiple platforms and compilers. Below is the current support status:

Platform	Compiler	Status
MacOS ARM64	GCC	✅
Ubuntu ARM64	GCC	✅
Ubuntu ARM64	Clang	✅
Windows x86	MSVC (cl)	✅

Legacy Repo

KvDB

KV-Store