Distributed State Machine Replication – Replicated Robot Factory

A distributed primary–backup replicated robot factory implemented in C++ using TCP sockets and multithreading. The system maintains consistent customer records across multiple servers using state machine replication (SMR) and continues operating despite server failures.

This project demonstrates key distributed systems concepts including:

• state machine replication
• primary–backup fault tolerance
• log-based recovery
• concurrent request processing
• latency and throughput measurement

---

System Overview

The system consists of clients and replicated servers.

Clients

Clients simulate customers placing robot orders or querying customer records.

Each client program can spawn multiple customer threads, each issuing requests to the server.

Supported request types:

• Robot Order – Creates a robot order for a customer (write operation)
• Read Record – Retrieves the latest order for a customer
• Print Records – Prints all customer records

Servers

Servers maintain replicated state using primary–backup replication.

Each server maintains:

• A replicated operation log
• A customer record table
• Replication metadata tracking the primary and committed operations

The system separates responsibilities across worker threads:

Worker	Responsibility
Engineer Worker	Handles client requests
Production Factory Administrator (PFA) Worker	Primary administrator responsible for replication
Idle Factory Administrator (IFA) Worker	Receives replication requests on backup servers

---

Replication Protocol

Write requests follow this workflow:

Client → Engineer Worker → PFA Worker → Replication → Backup Servers

1. The primary appends the operation to its replicated log.
2. The operation is sent to all backup servers.
3. Backup servers append the operation to their logs.
4. Once replication completes, the operation is committed and applied to the state machine.

This ensures that all replicas process operations in the same order.

Primary Selection and Read Handling

The server that receives write requests acts as the primary server for the replication protocol.
All write operations are coordinated by this primary and replicated to the backup servers.

Read requests do not require replication and can be served by any server, since all replicas maintain the same committed state.

---

Failure Handling

The system tolerates both primary and backup failures.

Backup Failure

If a backup server fails during replication:

• The primary closes the failed connection
• The server is temporarily ignored
• Requests continue to be processed
• Client connections to backup terminate gracefully

Primary Failure

If the primary fails:

• Backups detect the failure through socket errors
• Client connections to primary terminate gracefully
• Another server can be promoted to primary

---

Server Recovery

When a failed server restarts, the primary automatically attempts to reconnect.

If reconnection succeeds:

1. The primary replays committed log entries
2. The recovering server rebuilds its state
3. Normal replication resumes

This allows servers to rejoin the system without manual state reconstruction.

---

Performance Measurement

The client measures: **- average latency (microseconds)

• minimum latency (microseconds)
• maximum latency (microseconds)
• throughput (operations per second)**

using a std::chrono::high_resolution_clock.

Two workloads were evaluated:

• Write workload – Robot order requests
• Read workload – Customer record queries

Write performance decreases as the number of replicas increases due to replication overhead, while read throughput improves when requests are distributed across servers.

---

Building the Project

Compile the system using:

make

This produces two binaries:

server
client

---

Running Servers

Server command format:

./server <port> <server_id> <num_peers> <peer_id ip port>...

Example with 3 servers:

./server 5000 0 2 1 127.0.0.1 5001 2 127.0.0.1 5002
./server 5001 1 2 0 127.0.0.1 5000 2 127.0.0.1 5002
./server 5002 2 2 0 127.0.0.1 5000 1 127.0.0.1 5001

Alternatively, the helper script can launch multiple servers:

./run_servers.sh 3

---

Running Clients

Client command format:

./client <server_ip> <server_port> <num_customers> <num_requests> <request_type>

Example:

./client 127.0.0.1 5000 256 10000 1

Arguments:

Parameter	Description
server_ip	IP address of server
server_port	Server port
num_customers	Number of customer threads
num_requests	Requests per thread
request_type	Request type (1 = order, 2 = read, 3 = print)

---

Example Workload

Write Workload

Simulates customers placing robot orders (write operations).

./client 127.0.0.1 5000 8 100000 1

This launches 8 customer threads, each issuing robot order requests.

Read Workload

Simulates concurrent clients reading customer records.

./client 127.0.0.1 5000 128 10000 2

This launches 128 customer threads issuing read requests.

Print Workload

Prints the current customer records stored on the server.

./client <server_ip> <server_port> 1 <num_customers_to_read> 3

Arguments:

• 1 - The print workload uses a single client thread
• <num_customers_to_read> - Number of customer records to print

Example:

./client 127.0.0.1 5000 1 256 3

This prints the most recent order number for each customer stored on the server.

---

Key Concepts Demonstrated

This project demonstrates several important distributed systems concepts:

• Primary–backup replication
• State machine replication
• Log-based recovery
• Concurrent server architectures
• Network fault tolerance
• Performance measurement of distributed systems

---

Technologies

• C++17
• POSIX sockets
• Multithreading (std::thread)
• Synchronization primitives (mutex, condition_variable)
• TCP networking