Legend: ✅ implemented · 🟡 partial · ❌ not implemented yet.
| Area | Status | Where |
|---|---|---|
Frame head, rstr magic, cid pipelining |
✅ | include/rawstor/protocol.h |
Response hash (xxh3 with libxxhash) |
✅ | librawstd/src/hash.c |
Session: SET_OBJECT |
✅ | OST ost/src/client.cpp, MDS mds/src/client.cpp |
Data: READ WRITE DISCARD ALLOCATE RELEASE FLUSH WRITE_ZEROES |
✅ | OST server + src/ost_backend.cpp |
Metadata: META SET_SYNC_STATE CREATE_VERSION LIST_VERSIONS |
✅ | OST server + src/ost_backend.cpp |
LIST, LOCATION_INFO |
✅ | OST and MDS servers, both clients |
Object: OBJ_CREATE OBJ_OPEN OBJ_RESIZE OBJ_REMOVE |
✅ | MDS server + src/mds_client.cpp |
Object versions: OBJ_COMMIT_VERSION OBJ_REMOVE_VERSION OBJ_LIST_VERSIONS |
✅ | MDS server + src/mds_client.cpp |
idempotency_key on mutating OBJ_* |
✅ | mds/src/store.cpp (applied_mutations) |
-ENOSYS for commands outside a server's role |
✅ | ost/src/client.cpp, mds/src/client.cpp |
Protocol version + feature bits in the SET_OBJECT handshake |
❌ | planned in MDS design |
Explicit len field in the response ({res, len, hash}) |
❌ | res still carries the payload size |
map_epoch in the IO frame (epoch-fence) |
❌ | planned in MDS design |
| Auth / capabilities | ❌ | open question |
One binary protocol is spoken by every rawstor server role: rawstor-ost
(object storage) and rawstor-mds (metadata server). The authoritative
definition is include/rawstor/protocol.h;
this page describes the same layouts for a reader.
- Transport: a stateful TCP connection.
- Byte order: host order, i.e. little-endian on every supported platform.
- Structs: packed (no padding, no reserved fields), with every multi-byte field on its natural alignment inside the struct. Every struct's size is checked at compile time.
- Magic:
0x72737472("rstr"in ASCII) opens every frame, as a sanity and endianness check. - Names:
RawstorFrame*for frames and payloads any role may use,RawstorFrameObj*for the payloads of the object (OBJ_*) commands.
In the diagrams below every row is 8 bytes: the left column is the row's
byte offset, the ruler on top each byte's position within the row. A
field's height is its size (a 64-bit field takes one row, a 16-byte id
two), fields smaller than 8 bytes share a row, and a ~ edge marks data
that follows the struct.
A client may pipeline requests: each carries a cid (command id) and its
response echoes it back, so responses are matched by cid, not by order.
The first request on a connection is SET_OBJECT. On an OST it binds the
connection to one object (one chunk of it, one version of it); the I/O
commands then act on that object. On the MDS it is a plain handshake. A
server answers any command outside its role with res = -ENOSYS.
Every request and every response starts with the same 8-byte head.
+0 +1 +2 +3 +4 +5 +6 +7
+-------------------------------------------+---------------------+---------------------+
0 | uint32_t magic | uint16_t cmd | uint16_t cid |
| | | |
+-------------------------------------------+---------------------+---------------------+
One command space for every role, grouped into ranges: 0x00 session (every
role), 0x01..0x1f data (OST), 0x20..0x3f metadata shared by OST and MDS,
0x40..0x5f object (MDS). SET_SYNC_STATE and META predate the grouping and
keep 0x0b/0x0c.
| cmd | name | role | request payload | response payload |
|---|---|---|---|---|
0x00 |
SET_OBJECT |
all | Basic (val = open flags, e.g. RAWSTOR_READONLY) |
— |
0x01 |
READ |
OST | IO | data (res bytes) |
0x02 |
WRITE |
OST | IO, then len bytes of data |
— |
0x03 |
DISCARD |
OST | IO | — |
0x04 |
ALLOCATE |
OST | Allocate | — |
0x05 |
RELEASE |
OST | Basic (non-nil version_id: that version only) |
— |
0x06 |
LIST |
OST, MDS | List | List rows |
0x08 |
LOCATION_INFO |
OST / MDS | Basic (unused) | RawstorLocationInfo |
0x09 |
FLUSH |
OST | Basic (unused) | — |
0x0a |
WRITE_ZEROES |
OST | IO (flags: SYNC, UNMAP) |
— |
0x0b |
SET_SYNC_STATE |
OST | SyncState | — |
0x0c |
META |
OST | Basic (offset = chunk offset, version_id, nil = live) |
Meta |
0x0d |
CREATE_VERSION |
OST | Basic (version_id = new version) |
— |
0x0e |
LIST_VERSIONS |
OST | Basic (offset = chunk offset) |
Version rows |
0x40 |
OBJ_CREATE |
MDS | ObjCreate | ObjCreated |
0x41 |
OBJ_OPEN |
MDS | Basic (version_id, nil = live) |
ObjDescriptor + chunks |
0x42 |
OBJ_RESIZE |
MDS | ObjOp (val = new size) |
ObjResized |
0x43 |
OBJ_REMOVE |
MDS | ObjOp | ObjDescriptor + chunks (the removed map) |
0x45 |
OBJ_COMMIT_VERSION |
MDS | ObjCommitVersion + members | ObjVersionCommitted |
0x46 |
OBJ_REMOVE_VERSION |
MDS | ObjOp (version_id) |
ObjVersionMember records |
0x47 |
OBJ_LIST_VERSIONS |
MDS | Basic | Version rows |
Ids (object_id, version_id, ost_id, ...) are 16-byte UUIDs; a nil
version_id means the live version. Object and version ids are generated by
the client.
Every mutating OBJ_* request (OBJ_CREATE, OBJ_RESIZE, OBJ_REMOVE,
OBJ_COMMIT_VERSION, OBJ_REMOVE_VERSION) carries an idempotency_key: a UUID the client
generates once per operation and keeps across its retries. The MDS records
the result of the first request that applies an idempotency_key and answers any
repeat of it with that same result instead of applying it again, so a
request resent after a lost reply is safe (see
MDS design, "Idempotent mutations"). Replies that a retry needs
in order to finish the client's side of the operation carry it:
OBJ_RESIZE the chunk count it grew from, OBJ_REMOVE the map the object
had.
Every response is the head followed by a 12-byte body; a payload, if any, follows right after.
+0 +1 +2 +3 +4 +5 +6 +7
+-------------------------------------------+---------------------+---------------------+
0 | uint32_t magic | uint16_t cmd | uint16_t cid |
| | | |
+-------------------------------------------+---------------------+---------------------+
8 | uint64_t hash |
| |
+-------------------------------------------+-------------------------------------------+
16 | int32_t res | payload (res bytes)... |
| | |
+-------------------------------------------+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
res < 0is-errno; no payload follows.res >= 0is the payload size in bytes (forREAD/WRITE/DISCARD/WRITE_ZEROES, the number of bytes handled);0means no payload.hashcovers the payload (the data, forREAD) and is checked by the receiver: xxh3 when built with libxxhash, 0 otherwise and whenever the sender doesn't compute one (the MDS never does).
Shared by every command that only names an object: SET_OBJECT, RELEASE,
CREATE_VERSION, META, LOCATION_INFO, FLUSH and the MDS's OBJ_OPEN. offset is the chunk offset of
the object object_id names (0 for a plain object); version_id binds
a version for SET_OBJECT, RELEASE, CREATE_VERSION, META and OBJ_OPEN
(nil = live); val is command-specific.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t object_id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t offset |
| |
+---------------------------------------------------------------------------------------+
24 | |
| uint8_t version_id[16] |
| |
32 | |
| |
+---------------------------------------------------------------------------------------+
40 | uint64_t val |
| |
+---------------------------------------------------------------------------------------+
READ, WRITE, DISCARD, WRITE_ZEROES on the bound object. hash
covers the data that follows a WRITE (same hash as a response's), 0
otherwise. flags:
RAWSTOR_FLAG_SYNC (durable before the response; WRITE, WRITE_ZEROES),
RAWSTOR_FLAG_UNMAP (may deallocate; WRITE_ZEROES).
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | uint64_t offset |
| |
+---------------------------------------------------------------------------------------+
8 | uint64_t hash |
| |
+-------------------------------------------+----------+--------------------------------+
16 | uint32_t len | flags | WRITE data... |
| | | |
+-------------------------------------------+----------+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
limit ids per page, starting after token_id (nil = from the start).
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t token_id[16] |
| |
8 | |
| |
+-------------------------------------------+-------------------------------------------+
16 | uint32_t limit |
| |
+-------------------------------------------+
Sets one copy's mirror consistency state (see mirroring).
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t object_id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t chunk_offset |
| |
+---------------------------------------------------------------------------------------+
24 | uint64_t epoch |
| |
+---------------------------------------------------------------------------------------+
32 | uint64_t sync_id |
| |
+---------------------------------------------------------------------------------------+
40 | uint64_t sync_id_history[0] |
| |
+---------------------------------------------------------------------------------------+
48 | uint64_t sync_id_history[1] |
| |
+---------------------------------------------------------------------------------------+
56 | uint64_t sync_id_history[2] |
| |
+---------------------------------------------------------------------------------------+
64 | uint64_t sync_id_history[3] |
| |
+----------+----------------------------------------------------------------------------+
72 | state |
| |
+----------+
Creates one copy of one chunk. chunk_shift is log2(chunk_size), 0 for an
unchunked object; stripe_width/failure_domain/width/member_role are the
chunk's placement identity, stored with it.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t object_id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t chunk_offset |
| |
+---------------------------------------------------------------------------------------+
24 | uint64_t size |
| |
+---------------------------------------------------------------------------------------+
32 | uint64_t stripe_width |
| |
+----------+----------+----------+----------+-------------------------------------------+
40 | chunk_ | failure_ | width | member_ |
| shift | domain | | role |
+----------+----------+----------+----------+
One row per (id, chunk offset); an id with several chunks spans consecutive
rows. The MDS answers one row per mds:// object, chunk offset 0 (the
object is addressed whole). The last row is always the resume cursor for the next page (a nil id
once nothing is left), never a result. An empty payload means the far end is
already exhausted.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t chunk_offset |
| |
+---------------------------------------------------------------------------------------+
LIST_VERSIONS' and OBJ_LIST_VERSIONS' answer: one row per version id,
in no particular order. An empty payload means the object has no versions
(always, on a backend without them).
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t version_id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
Everything about one stored copy: its size, its placement identity and its mirror consistency state.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | uint64_t size |
| |
+---------------------------------------------------------------------------------------+
8 | uint64_t epoch |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t sync_id |
| |
+---------------------------------------------------------------------------------------+
24 | uint64_t sync_id_history[0] |
| |
+---------------------------------------------------------------------------------------+
32 | uint64_t sync_id_history[1] |
| |
+---------------------------------------------------------------------------------------+
40 | uint64_t sync_id_history[2] |
| |
+---------------------------------------------------------------------------------------+
48 | uint64_t sync_id_history[3] |
| |
+----------+----------+----------+----------+-------------------------------------------+
56 | state | chunk_ | width | member_ |
| | shift | | role |
+----------+----------+----------+----------+
uint64_t used, uint64_t total (see <rawstor/location.h>).
Embedded in ObjCreate and ObjDescriptor, at an 8-byte offset in both; the
chunk_shift that follows it there rounds its 3 trailing bytes out to 4, so
ObjDescriptor's nchunks stays aligned.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | uint64_t stripe_width |
| |
+---------------------------------------------------------------------------------------+
8 | uint64_t placement_seed |
| |
+----------+----------+----------+------------------------------------------------------+
16 |redundancy| width | failure_ |
| | | domain |
+----------+----------+----------+
chunk_shift is log2(chunk_size); an mds:// object's chunk_size is always
a nonzero power of two, so 0 (and anything from 64 on) is rejected.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | |
| uint8_t idempotency_key[16] |
| |
24 | |
| |
+---------------------------------------------------------------------------------------+
32 | uint64_t logical_size |
| |
+---------------------------------------------------------------------------------------+
40 | policy.stripe_width |
| |
+---------------------------------------------------------------------------------------+
48 | policy.placement_seed |
| |
+----------+----------+----------+----------+-------------------------------------------+
56 |redundancy| width | failure_ | chunk_ |
| | | domain | shift |
+----------+----------+----------+----------+
ObjCreated and ObjVersionCommitted are a single uint64_t map_epoch.
OBJ_RESIZE (val = the new size), OBJ_REMOVE and OBJ_REMOVE_VERSION
(version_id = the version to remove); fields a command doesn't use are
0/nil.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | |
| uint8_t idempotency_key[16] |
| |
24 | |
| |
+---------------------------------------------------------------------------------------+
32 | |
| uint8_t version_id[16] |
| |
40 | |
| |
+---------------------------------------------------------------------------------------+
48 | uint64_t val |
| |
+---------------------------------------------------------------------------------------+
OBJ_RESIZE's reply: the new map_epoch and the chunk count the object grew
from, so a retried resize still knows which chunks it has to create.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | uint64_t map_epoch |
| |
+-------------------------------------------+-------------------------------------------+
8 | uint32_t old_nchunks |
| |
+-------------------------------------------+
OBJ_OPEN's reply (and OBJ_REMOVE's: the map the object had): the
descriptor, then nchunks chunk entries, each a
uint8_t width followed by width slots. A slot is 19 bytes followed by
location_len bytes of its OST's location URI (e.g. ost://host:7777, not
null-terminated); location_len is 0 when the topology no longer lists that
OST.
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | uint64_t logical_size |
| |
+---------------------------------------------------------------------------------------+
24 | uint64_t map_epoch |
| |
+---------------------------------------------------------------------------------------+
32 | policy.stripe_width |
| |
+---------------------------------------------------------------------------------------+
40 | policy.placement_seed |
| |
+----------+----------+----------+----------+-------------------------------------------+
48 |redundancy| width | failure_ | chunk_ | uint32_t nchunks |
| | | domain | shift | |
+----------+----------+----------+----------+-------------------------------------------+
Each chunk entry:
+0 +1 +2 +3 +4 +5 +6 +7
+----------+----------------------------------------------------------------------------+
0 | width | width slots... |
| | |
+----------+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
Each slot:
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t ost_id[16] |
| |
8 | |
| |
+---------------------+----------+------------------------------------------------------+
16 |uint16_t location_len|slot_index| location... |
| | | |
+---------------------+----------+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+
Registers a version: nmembers ObjVersionMember records follow, one per
chunk copy that holds it. OBJ_REMOVE_VERSION replies with the same records (the
copies to destroy).
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | |
| uint8_t id[16] |
| |
8 | |
| |
+---------------------------------------------------------------------------------------+
16 | |
| uint8_t version_id[16] |
| |
24 | |
| |
+---------------------------------------------------------------------------------------+
32 | |
| uint8_t idempotency_key[16] |
| |
40 | |
| |
+-------------------------------------------+-------------------------------------------+
48 | uint32_t nmembers |
| |
+-------------------------------------------+
ObjVersionMember — 24 bytes:
+0 +1 +2 +3 +4 +5 +6 +7
+---------------------------------------------------------------------------------------+
0 | uint64_t logical_index |
| |
+---------------------------------------------------------------------------------------+
8 | |
| uint8_t ost_id[16] |
| |
16 | |
| |
+---------------------------------------------------------------------------------------+