License

Copyright

(c) 2021-2024 Espresso Systems. HotShot was developed by Espresso Systems.

HotShot Consensus Module

HotShot is a BFT consensus protocol based off of HotStuff, with the addition of proof-of-stake and VRF committee elections.

Disclaimer

DISCLAIMER: This software is provided "as is" and its security has not been externally audited. Use at your own risk.

Usage

Please see the rustdoc for API documentation, and the examples directory for usage.

Dependencies

Unix-like

Nix (macos and linux)

nix develop

Brew (macos)

brew install cmake protobuf

Apt-get (linux)

apt-get install cmake protobuf

Windows

Chocolatey

choco install cmake protoc

Scoop

scoop bucket add extras
scoop install protobuf cmake

Building

Once dependencies have been installed, to build everything:

just build

Static linking

HotShot supports static linking for its examples:

# Nix-shell is optional but recommended
nix develop .#staticShell

just build

Testing

To test:

RUST_LOG=$ERROR_LOG_LEVEL RUST_LOG_FORMAT=$ERROR_LOG_FORMAT just test

• RUST_LOG=$ERROR_LOG_LEVEL: The basic levels of logging include warn, error, info.
• RUST_LOG_FORMAT=$ERROR_LOG_FORMAT: The types of logging include full, json, and compact.
• Internally, the inclusion of the --nocapture flag indicates whether or not to output logs.
• Internally, we run at --test-threads=1 because the tests spawn up a lot of file handles, and unix based systems consistently run out of handles.

To stress test, run the ignored tests prefixed with test_stress:

RUST_LOG=$ERROR_LOG_LEVEL RUST_LOG_FORMAT=$ERROR_LOG_FORMAT just run_test test_stress

Careful

To double-check for UB:

nix develop .#correctnessShell
just careful

Testing on CI

To test as if running on CI, one must limit the number of cores and ram to match github runners (2 core, 7 gig ram). To limit the ram, spin up a virtual machine or container with 7 gigs ram. To limit the core count when running tests:

ASYNC_STD_THREAD_COUNT=1 RUST_LOG=$ERROR_LOG_LEVEL RUST_LOG_FORMAT=$ERROR_LOG_FORMAT just tokio test

Tokio-console

To use tokio-console, drop into the console shell:

nix develop .#consoleShell

Then, run an example.

On a separate terminal, also drop into the console shell and start tokio-console:

nix develop .#consoleShell -c tokio-console

This second window should now display task usage.

Open Telemetry + Jaeger Integration

To view distributed logs with just the centralized server and one client, first edit the centralized_server/orchestrator file to include have a threshold and num_nodes of 1.

Then open three separate terminals.

# Terminal 1
# Start the jaeger instance to view spans
docker run -d -p6831:6831/udp -p6832:6832/udp -p16686:16686 -p14268:14268 jaegertracing/all-in-one:latest

# Terminal 2
# Start the CDN

# Terminal 3
# Start the client

Resource Usage Statistics

To generate usage stats:

• build the test suite
• find the executable containing the test of interest
• run profiling tools

The executable cargo uses is shown in the output of cargo test.

For example, to profile test_stress_dht_many_round:

# bring profiling tooling like flamegraph and heaptrack into scope
nix develop .#perfShell

# show the executable we need run
# and build all test executables (required for subsequent steps)
cargo test --verbose --release --lib --bins --tests --benches --workspace -- --test-threads=1
# the output cargo test contains the tests path:
#       Running `/home/jrestivo/work/crosscross/target/release/deps/counter-880b1ff53ee21dea test_stress --test-threads=1 --ignored`
#       running 7 tests
#       test test_stress_dht_many_rounds ... ok
#       ...

# a more detailed alternative to flamegraph
# NOTE: only works on linux
heaptrack $(fd -I "counter*" -t x | rg release) --ignored -- test_stress_dht_many_round --nocapture
# palette provides memory statistics, omission will provide cpu cycle stats as colors
# NOTE: must be run as root on macos
flamegraph --palette=mem $(fd -I "counter*" -t x | rg release) --ignored -- test_stress_dht_one_round
# code coveragte statistics
cargo-llvm-cov llvm-cov --test=test_stress_dht_many_round --workspace --all-targets --release --html --output-path lcov.html

This will output:

• heaptrack.counter-$HASH which is viewable by heaptrack. This provides a plethora of useful statistics about memory and cpu cycles.
• flamegraph.svg which is a (moderately) less detailed version of heaptrack.
• lcov.html generates a summary of code coverage.

Debugging

A debugging config file is provided for vscode and vscodium in .vscode/launch.json. This is intended to be used with vadimcn/vscode-lldb but may work with other rust debuggers as well.

To bring lldb into scope with nix, run nix develop .#debugShell.

Git Workflow

For espresso developers we have written up a description of our workflow here.

Extra Editor Configuration

Choose an async runtime to use before launching a text editor. This may be done by setting the environment RUSTFLAGS. For example:

nvim # launch text editor of choice. We choose neovim in this example
unset RUSTFLAGS # Unset rustflags so we may continue to use the justfile. The justfile sets these particular config options

Debugging

We support the CodeLLDB Debugger.

Neovim

Install dap and rust-tools. Install the CodeLLDB debugger listed above. Follow the instructions here to configure the adapter. To add our project-local configurations, run:

lua require('dap.ext.vscode').load_launchjs(nil, { ["codelldb"] = {"rust"} })

Finally, place a breakpoint and run :DapContinue to begin debugging.

NOTE: Do NOT configure dap at all with rust-tools. Do it manually.

Example configuration.

Vscode

Install the extension and load the launch.json file. Then run the desired test target.