Skip to content

Commit

Permalink
feat: add parallel builder (#219)
Browse files Browse the repository at this point in the history
## 📝 Summary
This PR introduces a new building algorithm which parallelizes the
process of building blocks. In particular, it moves the largely
sequential process of processing orders, resolving conflicts, and
building blocks into modular and parallel process. Each of these
components now runs in parallel, allowing for continuous intake of
orderflow, flexible resolution of conflicts, and building blocks with
the latest flow and best conflict resolution results. It is a
significant architecture change.

See
[run_parallel_builder](https://github.com/flashbots/rbuilder/blob/parallel-builder/crates/rbuilder/src/building/builders/parallel_builder/mod.rs#L180)
for a good code entry point and the
[readme.md](https://github.com/flashbots/rbuilder/blob/parallel-builder/crates/rbuilder/src/building/builders/parallel_builder/readme.md)
for more architecture details.

Key Changes:
- Introduced a new parallel_builder module, replacing the
merging_builder. This adds several components for orderflow intake,
conflict identification, conflict resolution, and merging of resolved
conflicts.
- Added a simulation cache to optimize repeated simulations of similar
order sequences.
- Updated the builder configuration to use the new Parallel Builder
instead of the Merging Builder.
- Implemented backtest functionality for the Parallel Builder.

## 💡 Motivation and Context

- Concurrency driven efficiency improvements: running significant
processes in parallel is more efficient, and should lead to better
blocks. Processes aren't sequential, so they aren't blocked on each
other anymore.
- Reuses previous work: Instead of repeating conflict resolution, we are
able to reuse old resolved conflict results.
- Better conflict management: We can prioritize "fast" conflict
resolution tasks to get results for a conflict group to the building
assembler fast, but then also queue "slow" tasks (like evolutionary
algorithms) which might produce better results but are slower.

## Testing
Finally, I've added a range of tests to the new algorithm. I further
have tested it extensively on a local node using mempool txs.

- ---

## ✅ I have completed the following steps:

* [✅ ] Run `make lint`
* [✅ ] Run `make test`
* [ ✅] Added tests (if applicable)
  • Loading branch information
bertmiller authored Oct 23, 2024
1 parent 3159c3f commit ccd9cb5
Show file tree
Hide file tree
Showing 23 changed files with 3,486 additions and 891 deletions.
30 changes: 23 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 3 additions & 4 deletions config-backtest-example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@ failed_order_retries = 1
drop_failed_orders = true

[[builders]]
name = "merging"
algo = "merging-builder"
name = "parallel"
algo = "parallel-builder"
discard_txs = true
num_threads = 5
merge_wait_time_ms = 100
num_threads = 25
7 changes: 3 additions & 4 deletions config-live-example.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,7 @@ failed_order_retries = 1
drop_failed_orders = true

[[builders]]
name = "merging"
algo = "merging-builder"
name = "parallel"
algo = "parallel-builder"
discard_txs = true
num_threads = 5
merge_wait_time_ms = 100
num_threads = 25
3 changes: 3 additions & 0 deletions crates/rbuilder/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,9 @@ shellexpand = "3.1.0"
async-trait = "0.1.80"

eth-sparse-mpt = { git = "https://github.com/flashbots/eth-sparse-mpt", rev = "5d0da73" }
crossbeam = "0.8.4"
parking_lot = "0.12.3"
dashmap = "6.1.0"

[build-dependencies]
built = { version = "0.7.1", features = ["git2", "chrono"] }
Expand Down
174 changes: 0 additions & 174 deletions crates/rbuilder/src/building/builders/merging_builder/combinator.rs

This file was deleted.

Loading

0 comments on commit ccd9cb5

Please sign in to comment.