Skip to content

Commit

Permalink
AA using mev plugin (#3641)
Browse files Browse the repository at this point in the history
* Add MevBudler

* Use trigger approach for bundler instead of harcoded timer

* Rename PeriodicMevBundler to just MevBundler

* Add head block as bundle event arg

* Add OnNewBlockBundleTrigger trigger

* Add flashbotsBundler based on ContinuousBundleSender

* Rename BundleEventArgs in Bundler to BundleUserOpsEventArgs

* Add MevBundler Test

* Delete PeriodicBundleTrigger

* Change the way we detect if MevPlugin is enabled

* Move MevBundler to InitRpcModules hook

`MevBundler` depends on `InitBlockchain` step, which still hasn't been run when
`Init` hook is called.

* Add some logging in MevBundler

* Change debug logs to info just for debugging ;)

* fix MevBundler instantiation MevBundlerTest

* Changed blockCount parameter in eth_feeHistory to long. (#3670)

* Updating Fast Sync config files (#3671)

Co-authored-by: matilote <matilote@users.noreply.github.com>

* Make FielterStore and id's thread safe

* Fixing the incorrect path

* CI: Build solution files on GHA (#3652)

* CI: Build baseline.sln on GHA

* CI: Build cortex.sln on GHA

* CI: Build Benchmarks.sln on GHA

* Fix some build errors after merge from master

* Fix test failing at ...Evm.Test/Tracing/AccessTxTracerTests.cs(50,40)

* Adding additional instructions for Ubuntu 21.10

* Add additional symlink for Fedora 35 (#3678)

This fixes usage with Fedora 35. Same as Ubuntu 21.10 fix. Matched
"find" usage with existing Fedora symlinks

* testing rocksdb latest builds

* fixing path

* adding file execution permissions

* trying with windows one more time

* checking the other way around

* testing rocksdb windows build

* something wrong with chdir on gha

* trying different way for executing script

* another way

* define full path

* Complete path to sh

* Update the bash path

* Random Attempt

* Update build-rocksdb-libs.yml

* Update build-rocksdb-libs.yml

* Update the path of bash

* Update build-rocksdb-libs.yml

* Update build-rocksdb-libs.yml

* Fixed path

* Fixed path in ubuntu and mac flows

* fixing uplaod paths

* fixing linux-x64 path

* trying with full path

* updating rocksdb shared libraries to 6.26.1

* trying diff path

* another try

* fixing path

* Update to mainnet_pruned in readme

* Updating Fast Sync config files (#3683)

Co-authored-by: matilote <matilote@users.noreply.github.com>

* fix pool tests

* Added PeriodicBundleTrigger again (very usefull for testing locally)

* Fix null signature in BundleTransaction

* Add spaceneth with aa enabled config

* Enable PeriodicBundleTrigger

* Logs for trigger

* logs

* More logs

* Enable autoreset in periodic bundle trigger

* fix

* logs =)

* Add try catch to trigger

* Add exception handling to OnNewBlockBundleTrigger

* update rocksdb

* Log stack trace

* log

* Log only block author

* added configs

* fix mev?

Co-authored-by: Klaudia Jazgar <67744705+kjazgar@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: matilote <matilote@users.noreply.github.com>
Co-authored-by: lukasz.rozmej <lukasz.rozmej@gmail.com>
Co-authored-by: 1swaraj <36474630+1swaraj@users.noreply.github.com>
Co-authored-by: Harsh Mishra <erbeusgriffincasper@gmail.com>
Co-authored-by: Mateusz Jędrzejewski <33068017+matilote@users.noreply.github.com>
Co-authored-by: Dave <13626476+DaveWK@users.noreply.github.com>
Co-authored-by: Mateusz Jędrzejewski <mateusz.jedrzejewski@yahoo.pl>
Co-authored-by: kristofgazso <kristof.gazso@gmail.com>
  • Loading branch information
11 people authored Dec 23, 2021
1 parent bc62530 commit 624d7a0
Show file tree
Hide file tree
Showing 68 changed files with 783 additions and 349 deletions.
51 changes: 10 additions & 41 deletions .github/workflows/build-rocksdb-libs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,13 @@ jobs:
steps:
- name: Cloning rocksDB repository
run: |
git clone git://github.com/warrenfalk/rocksdb-sharp-native.git
- name: Checking out the latest rocksdb version for Windows
if: matrix.os == 'windows-latest'
run: |
sed -i "31s/.*/ROCKSDBVERSION=$(git ls-remote --tags --refs --sort="v:refname" git://github.com/facebook/rocksdb.git | tail -n1 | sed 's/.*\///')/" /d/a/nethermind/nethermind/rocksdb-sharp-native/build-rocksdb.sh
shell: bash
git clone https://github.com/NethermindEth/rocksdb-sharp-1.git
- name: Building rocksDB for Windows
if: matrix.os == 'windows-latest'
run: |
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
call "C:\Program Files\Git\bin\bash.exe"
sh /d/a/nethermind/nethermind/rocksdb-sharp-native/build-rocksdb.sh
(bash /d/a/nethermind/nethermind/rocksdb-sharp-1/build-native/build-rocksdb.sh)
shell: cmd
- uses: actions/upload-artifact@v1
name: Uploading Windows artifact
Expand All @@ -35,50 +30,24 @@ jobs:
- name: Building rocksDB for Linux
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt-get update
sudo apt install libsnappy-dev libzstd-dev liblz4-dev zlib1g-dev
git clone https://github.com/facebook/rocksdb.git
cd rocksdb/
git fetch --tags
latestTag=$(git describe --tags `git rev-list --tags --max-count=1`)
git checkout $latestTag
TAG="${latestTag:1}"
PORTABLE=1 make -j8 shared_lib
strip librocksdb.so.${TAG}
mv librocksdb.so.${TAG} librocksdb.so
cd rocksdb-sharp-1/build-native
chmod +x build-rocksdb.sh
./build-rocksdb.sh
- uses: actions/upload-artifact@v1
name: Uploading Linux artifact
if: matrix.os == 'ubuntu-latest'
with:
name: linux_artifact_rocksdb
path: rocksdb/librocksdb.so
path: rocksdb-sharp-1/build-native/runtimes/linux-x64/native/librocksdb.so
- name: Building rocksDB for OSX
if: matrix.os == 'macOS-latest'
run: |
git clone git://github.com/Cyan4973/lz4.git
brew install gcc
brew install snappy
brew install zstd
make -C lz4/lib
cp -L lz4/lib/liblz4.dylib ./liblz4_64.dylib
make -C lz4/lib clean
CFLAGS="-arch x86_64" CXXFLAGS="-arch x86_64" LDFLAGS="-arch x86_64" make -C lz4/lib
lipo -create ./liblz4_64.dylib -output ./liblz4.dylib
cp -v ./liblz4.dylib lz4/lib/$(readlink lz4/lib/liblz4.dylib)
touch lz4/lib/liblz4
make -C lz4/lib install
git clone https://github.com/facebook/rocksdb.git
cd rocksdb/
git fetch --tags
latestTag=$(git describe --tags `git rev-list --tags --max-count=1`)
git checkout $latestTag
TAG="${latestTag:1}"
sed -i.bak '14s/CXXFLAGS += ${EXTRA_CXXFLAGS}/CXXFLAGS += -w/' Makefile
PORTABLE=1 make -j8 shared_lib
mv librocksdb.${TAG}.dylib librocksdb.dylib
cd rocksdb-sharp-1/build-native
chmod +x build-rocksdb.sh
./build-rocksdb.sh
- uses: actions/upload-artifact@v1
name: Uploading Darwin artifact
if: matrix.os == 'macOS-latest'
with:
name: darwin_artifact_rocksdb
path: rocksdb/librocksdb.dylib
path: rocksdb-sharp-1/build-native/runtimes/osx-x64/native/librocksdb.dylib
36 changes: 36 additions & 0 deletions .github/workflows/build-solution.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Build Solution

on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]

defaults:
run:
working-directory: src/Nethermind

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
submodules: recursive

- name: Setup .NET
uses: actions/setup-dotnet@v1
with:
dotnet-version: 6.0.x

- name: Install dependencies
run: |
dotnet restore Baseline.sln
dotnet restore Cortex.sln
dotnet restore Benchmarks.sln
- name: Build
run: |
dotnet build --configuration Release --no-restore Baseline.sln
dotnet build --configuration Release --no-restore Cortex.sln
dotnet build --configuration Release --no-restore Benchmarks.sln
16 changes: 10 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
[Linux x64/arm64](http://downloads.nethermind.io)<br/>
[MacOS](http://downloads.nethermind.io)<br/>

It syncs fully on:
It syncs fully on:
* `Mainnet`
* `Goerli`
* `Rinkeby`
Expand All @@ -40,15 +40,15 @@ It syncs fully on:
1. To execute the launcher
``nethermind``
1. To execute the runner
``nethermind --config mainnet``
``nethermind --config mainnet_pruned``

**Homebrew**
1. `brew tap nethermindeth/nethermind`
1. `brew install nethermind`
1. To execute the launcher
``nethermind-launcher``
1. To execute the runner
``nethermind --config mainnet``
``nethermind --config mainnet_pruned``

# Build from Source

Expand All @@ -68,8 +68,11 @@ It syncs fully on:
* Install dependencies
```sh
sudo apt-get install libsnappy-dev libc6-dev libc6

# Link libraries (only for Ubuntu 21.10)
sudo ln -s /usr/lib/x86_64-linux-gnu/libdl.so.2 /usr/lib/x86_64-linux-gnu/libdl.so
```
*Tested on Ubuntu 21.04, 20.04 and 18.04 LTS*
*Tested on Ubuntu 21.04, 20.04 and 18.04 LTS and 21.10*

#### Debian
* [Install .NET](https://docs.microsoft.com/en-gb/dotnet/core/install/linux-debian)
Expand Down Expand Up @@ -100,6 +103,8 @@ sudo yum install -y glibc-devel snappy libzstd
# Link libraries
sudo ln -s `find /usr/lib64/ -type f -name "libbz2.so.1*"` /usr/lib64/libbz2.so.1.0 && \
sudo ln -s `find /usr/lib64/ -type f -name "libsnappy.so.1*"` /usr/lib64/libsnappy.so
# also required for Fedora 35
sudo ln -s `find /usr/lib64/ -type f -name "libdl.so.2*"` /usr/lib64/libdl.so
```
*Tested on Fedora 32*

Expand All @@ -118,7 +123,7 @@ git clone https://github.com/NethermindEth/nethermind --recursive
cd nethermind/src/Nethermind
dotnet build Nethermind.sln -c Release
cd Nethermind.Runner
dotnet run -c Release --no-build -- --config mainnet
dotnet run -c Release --no-build -- --config mainnet_pruned
```

## Docker Image
Expand Down Expand Up @@ -163,4 +168,3 @@ Nethermind client can be used in your projects, when setting up private Ethereum

# License
[![GitHub](https://img.shields.io/github/license/nethermindeth/nethermind.svg)](https://github.com/NethermindEth/nethermind/blob/master/LICENSE)

1 change: 0 additions & 1 deletion scripts/deployment/publish-github.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ GIT_SHORT_TAG_FIRST3="${GIT_SHORT_TAG:0:3}"
echo The TAG is $GIT_SHORT_TAG_FIRST3

cd $RELEASE_DIRECTORY
cd plugins && PUB_PLUGINS_FILE="$(basename *.zip)" && cd ..
cd $LIN_RELEASE && PUB_LIN_FILE="$(basename nethermind-linux-amd64-*)" && cd ..
cd $OSX_RELEASE && PUB_OSX_FILE="$(basename nethermind-darwin-amd64-*)" && cd ..
cd $WIN_RELEASE && PUB_WIN_FILE="$(basename nethermind-windows-amd64-*)" && cd ..
Expand Down
1 change: 1 addition & 0 deletions scripts/execution.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,6 @@ then
sudo /usr/share/nethermind/Nethermind.Runner $@
else
echo 'Executing Nethermind Launcher'
cd /usr/share/nethermind
sudo /usr/share/nethermind/Nethermind.Launcher
fi
2 changes: 1 addition & 1 deletion src/Nethermind/Baseline.sln
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Network.Stats",
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.PubSub", "Nethermind.PubSub\Nethermind.PubSub.csproj", "{A913DB39-865B-4162-B279-16654B20D949}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.WebSockets", "Nethermind.WebSockets\Nethermind.WebSockets.csproj", "{3C731BA7-F932-4771-A890-7FB4AB10F80F}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Sockets", "Nethermind.Sockets\Nethermind.Sockets.csproj", "{3C731BA7-F932-4771-A890-7FB4AB10F80F}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Network", "Nethermind.Network\Nethermind.Network.csproj", "{B34B5EC6-1810-4961-9CF4-4A96B286B6E1}"
EndProject
Expand Down
3 changes: 0 additions & 3 deletions src/Nethermind/Cortex.sln
Original file line number Diff line number Diff line change
Expand Up @@ -76,8 +76,6 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Nethermind.Merkleization",
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Nethermind.Ssz.Benchmark", "Nethermind.Ssz.Benchmark\Nethermind.Ssz.Benchmark.csproj", "{CF819EE6-60D9-43FA-84A1-77D41FD20541}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Nethermind.Core2.Cryptography.Benchmark", "Nethermind.Core2.Cryptography.Benchmark\Nethermind.Core2.Cryptography.Benchmark.csproj", "{88DCDFAA-62D7-4E7D-9F02-D3F1F0328585}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -247,6 +245,5 @@ Global
{359CB477-96DF-4A84-83D6-1A6B8E82410F} = {4FFCC84D-DC26-4429-84D4-8809DD52B851}
{D013F1F4-1C24-4ED0-83A5-86E338746104} = {4FFCC84D-DC26-4429-84D4-8809DD52B851}
{5792C470-A4E7-447B-87DD-D7269723EB13} = {4FFCC84D-DC26-4429-84D4-8809DD52B851}
{88DCDFAA-62D7-4E7D-9F02-D3F1F0328585} = {86A6990C-81A6-4D2A-B2D5-269F0905EB6C}
EndGlobalSection
EndGlobal
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
using System;
using System.Linq;
using System.Collections.Generic;
using Nethermind.AccountAbstraction.Bundler;
using Nethermind.Consensus.Transactions;
using Nethermind.Core;
using Nethermind.Int256;
using Nethermind.Mev.Data;
using Nethermind.Mev.Source;
using NSubstitute;
using NUnit.Framework;
using Nethermind.Logging;

namespace Nethermind.AccountAbstraction.Test
{
[TestFixture]
public class MevBundlerTests
{
private ITxSource GetTxSource(params Transaction[] transactions)
{
var txSource = Substitute.For<ITxSource>();
txSource.GetTransactions(Arg.Any<BlockHeader>(), Arg.Any<long>()).Returns(transactions);
return txSource;
}

private IBundlePool GetBundlePool(List<MevBundle> bundles)
{
var bundlePool = Substitute.For<IBundlePool>();
bundlePool.AddBundle(Arg.Any<MevBundle>()).Returns(true).AndDoes(info => bundles.Add(info.Arg<MevBundle>()));
bundlePool.GetBundles(Arg.Any<long>(), Arg.Any<UInt256>()).Returns(bundles);
return bundlePool;
}

private IBundleTrigger GetBundleTrigger()
{
return Substitute.For<IBundleTrigger>();
}

[Test]
public void adds_bundles_to_mev_pool_when_mev_plugin_is_enabled()
{
var transactions = new Transaction[] { };

var bundleTrigger = GetBundleTrigger();
var txSource = GetTxSource(transactions);

var bundles = new List<MevBundle>();
var bundlePool = GetBundlePool(bundles);

var bundler = new MevBundler(bundleTrigger, txSource, bundlePool, NullLogger.Instance);
var bundleEventArgs = new BundleUserOpsEventArgs(Core.Test.Builders.Build.A.Block.TestObject);

bundleTrigger.TriggerBundle += Raise.EventWith(this, bundleEventArgs);

var bundledTxs = bundles.SelectMany(bundle => bundle.Transactions);
Assert.That(bundledTxs.SequenceEqual(transactions));
}
}
}
Loading

0 comments on commit 624d7a0

Please sign in to comment.