These instructions are up to date wrt the following releases.
nethermind
1.26.0lighthouse
5.3.0mev-boost
1.8.0
You'll need to be comfortable with Linux. The goal here is to gain an understanding of how each node process is configured directly so we're not using any higher level containerisation stuff.
Before you start:
- (Optional) Update the firmware in your router, factory reset, and reconfigure, out of an abundance of caution.
- Figure out some stuff up front:
- Hostname: replace
node01
below with this. - Username on the host: replace
[username]
below with this. - Static IP address: replace
192.168.20.51
below with this. - SSH port to open up: replace
60001
below with this.
- Hostname: replace
First decide between Intel and ARM for your hardware. If you want a quiet, fanless machine, choose ARM, but beware the issues I had below with the Radxa Rock 5B.
Only buy the big drive after reading this awesome "hall of blame" Gist.
- Grab an Intel NUC. Specs for mine:
- i5-1135G7, 4 cores. Only two slots (one M.2, one 2.5" SATA)
- 32 GB RAM (but 16GB is fine)
- Small drive (OS) is a Samsung SSD 980 250 GB M2
- Big drive (data) is a Crucial BX500 2TB 2.5" SSD. "On the box" it says "SATA 6GB/s - Up to 540MB/s Read - Up to 500MB/s Write". The speed of this drive will affect your performance and re-sync time more than anything else. I don't recommend this drive, which takes about 30 hours to sync. Wish I'd got an NVMe.
- Set the machine to restart after a power failure.
F2
during boot to get into BIOS settings- Power > Secondary power settings
- After power failure: Power on
F10
to save and exit
- Install Ubuntu Server 22.04 LTS
- F10 on boot to boot from USB for Intel NUC
Ubuntu Server (minimized)
- Check ethernet interface(s) have a connection, use DHCP for now
- Use an entire disk, 250GB SSD, LVM group, no need to encrypt
- Take photo of file system summary screen during installation
- Hostname:
node01
. Set a username. - Install OpenSSH server.
- Import SSH identity: yes, GitHub, don’t allow password auth over SSH
- Use your SSH public key from GitHub
- No extra snaps
- Finish installation and reboot
- Disable
cloud-init
sudo touch /etc/cloud/cloud-init.disabled
sudo reboot
- Make sure you can still log in as your new user.
sudo dpkg-reconfigure cloud-init
and choosenone of the above
.sudo apt-get purge cloud-init
sudo rm -rf /etc/cloud/ && sudo rm -rf /var/lib/cloud/
sudo reboot
- Again, make sure you can still log in as your new user.
- Grab a Radxa Rock 5B board and parts. Specs for mine:
- 16 GB RAM
- Small drive (OS) is a 64 GB eMMC
- Big drive (data) is a Crucial P3 Plus PCIe 4.0 NVMe M.2 2280. Do NOT get an NVMe drive with a massive heat sink on it, cause it won't fit. Clearance is limited between the bottom of the board and the case.
- This drive claims "Up to 4,800MB/s Read - Up to 4,100MB/s Write" on the box.
- But hewre's the
fio
output (see below for the command line used)
READ: bw=316MiB/s (331MB/s), 316MiB/s-316MiB/s (331MB/s-331MB/s), io=3070MiB (3219MB), run=9722-9722msec WRITE: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=1026MiB (1076MB), run=9722-9722msec
- Get the Radxa case/heatsink. No need for a fan.
- Get the eMMC to USB adapter for flashing the eMMC from the host machine.
- There is no BIOS!
- ARM boards are not PCs and do not have a BIOS. Don't bother connecting a keyboard and monitor if you don't plan to use one long term.
- The board powers on again if it ever loses power, so, nothing to set up there.
- To shut it down and have it stay off, use
sudo poweroff
. If you usesudo shutdown -r now
it will just reboot. - The RTC is managed from some binaries - see below.
- Flash the OS to the MMC.
- Download the latest Ubuntu image from Radxa's releases repo. It'll be something like
rock-5b_ubuntu_jammy_cli_b36.img.xz
. - Note that this is NOT an installer image, it's the actual OS that you're going to run.
- Make sure you can uncompress
xz
files:sudo apt install xz-utils
unxz rock-5b_ubuntu_jammy_cli_b36.img.xz
- Plug the USB adapter in and find out what
sd*
device it got withsudo dmesg | tail -20
. Mine wassda
. - Write the image with
sudo dd if=./rock-5b_ubuntu_jammy_cli_b36.img of=/dev/sda bs=1M
sudo umount /dev/sda
- Remove the USB adapter, take the little eMMC module off it and stick it on the underside of the board.
- Connect the ethernet and USB cables and the board should power on. It seems OK to run it without the heatsink briefly. The LED should go green, then blue.
- Watch the web interface of your router for the machine joining the LAN. Grab the IP address. Mine was
192.168.20.51
. You might instead be able to userock-5b.local
if your router creates it. - SSH in:
ssh rock@192.168.20.51
, password:rock
- Check the OS version is what you expect:
lsb_release -a
- Create a new user:
sudo adduser [username]
. Set the password. - Add the user to the group that can be root:
sudo adduser [username] sudo
- Disconnect from SSH as
rock
and reconnect as your new user - Remove the existing
rock
user:sudo deluser rock
- Change the hostname. Mine is
node01
.sudo nano /etc/hostname
and changesudo nano /etc/hosts
and change
- Get network time working
sudo nano /etc/systemd/timesyncd.conf
- Add
time.google.com
sudo timedatectl set-ntp true
systemctl restart systemd-timesyncd
- Check time with
timedatectl status
and check forSystem clock synchronized: yes
- TODO: Not working. Fix.
- Download the latest Ubuntu image from Radxa's releases repo. It'll be something like
- Switch to SSH-ing in from another machine now, for ease of copy/pasting stuff from this guide.
- Grab the IP address from the router.
ssh [username]@[ip]
- If you restored your GitHub SSH key during installation and you have it on your local machine, you should be straight in.
- Update packages and get some stuff we're going to need below.
sudo apt update
sudo apt upgrade
(make coffee)- Agree to restarting all the suggested services.
sudo apt install nano unzip net-tools netplan.io ufw fail2ban parted fio ccze smartmontools speedtest-cli
- Configure a static IP address.
- Get the interface name for the Ethernet:
ip link
. Mine wasenP4p65s0
on the ARM board,enp88s0
on the Intel NUC. - Figure out whether you're using
netplan
orNetworkManager
. If both are installed you might have/etc/netplan/something.yaml
simply delegating toNetworkManager
withrenderer: NetworkManager
. The below assumes you want to usenetplan
. - Paste the below block into a new
netplan
config file:sudo nano /etc/netplan/01-config.yaml
.
network: version: 2 renderer: networkd ethernets: enp88s0: dhcp4: no addresses: - 192.168.20.51/24 routes: - to: default via: 192.168.20.1 nameservers: addresses: [8.8.8.8, 1.1.1.1, 1.0.0.1]
.yaml
files use spaces for indentation (either 2 or 4), not tabs.- A subnet mask of
/24
means only the last octet (8 bits) in the address changes for each device on the subnet. - The DNS servers here are Google's and Cloudflare's.
- You might also consider using 9.9.9.9 (Quad9, does filtering of known malware sites).
sudo chmod 600 /etc/netplan/01-config.yaml
sudo netplan try
to check your config file.sudo netplan apply
. You'll lose your SSH connection at this point.- Try to SSH in on the new IP. Then confirm it's changed with
ip a
.
- Get the interface name for the Ethernet:
- Tighten up ssh
sudo nano /etc/ssh/sshd_config
- Change the ssh port from the default. Uncomment the
Port
line. Pick a memorable port number, eg. 60001, and make a note of it. - Only allow ssh'ing in using a key from now on. Set
PasswordAuthentication no
. - Also change
systemd
which may be the one listening on port 22 because it's "socket activated". sudo mkdir /etc/systemd/system/ssh.socket.d
sudo nano /etc/systemd/system/ssh.socket.d/port.conf
and put:
[Socket] ListenStream= ListenStream=[60001]
- Reboot and reconnect, but this time use the
-p 60001
arg forssh
.
- Configure the firewall
- Confirm
ufw
is installed:which ufw
- Run these:
sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow 60001/tcp comment 'ssh' sudo ufw allow out from any to any port 123 comment 'ntp' sudo ufw allow 30303 comment 'execution client' sudo ufw allow 9000 comment 'consensus client' sudo ufw allow 8545/tcp comment 'execution client health' sudo ufw allow 8551/tcp comment 'execution client health' sudo ufw enable
- Note that
http
andhttps
are absent above. - Check which ports are accessible with
sudo ufw status
- Also block pings:
sudo nano /etc/ufw/before.rules
, find the line readingA ufw-before-input -p icmp --icmp-type echo-request -j ACCEPT
and changeACCEPT
toDROP
sudo ufw reload
- Confirm
- Make sure that
unattended-updates
works for more than just security updates and includes non-Ubuntu PPAs.- On ARM only,
sudo touch /etc/apt/apt.conf.d/50unattended-upgrades
first because this won't exist. sudo nano /etc/apt/apt.conf.d/50unattended-upgrades
- Add this anywhere:
// Allow non-Ubuntu PPAs Unattended-Upgrade::Origins-Pattern { "o=*"; };
- On ARM only,
- (Optional and only required if you didn't restore SSH keys from GitHub during installation) Set up ssh keys for all client machines from which you'll want to connect.
- If re-installing, restore the
~/.ssh/authorized_keys
file you backed up earlier, usingscp
.- Try connecting using
ssh
first - You'll get an error about the host key changing, including a command to run to forget the old host key. Run it.
- Now do the
scp
copy:scp -P 1035 ./authorized_keys [username]@[ip]:/home/[username]/.ssh/authorized_keys
- Try connecting using
- Otherwise:
- You might like to set an alias in
~/.bashrc
such asalias <random-name>="ssh -p 60001 [username]@[server IP]"
- Similarly for scp:
alias <random-name>="scp -P 60001 $1 [username]@[server IP]:/home/[username]"
ssh-keygen -t rsa -b 4096 -C "[client nickname]"
- No passphrase.
- Accept the default path. You'll get both
~/.ssh/id_rsa.pub
(public key) and~/.ssh/id_rsa
(private key). - Copy the public key to the server:
scp -P 60001 ~/.ssh/id_rsa.pub [username]@[server IP]:/home/[username]/.ssh/authorized_keys
- Verify the file is there on the server.
- Verify you can ssh in to the server and you're not prompted for a password. Use the alias you created earlier.
- You might like to set an alias in
- If re-installing, restore the
- Ban any IP address that has multiple failed login attempts using
fail2ban
- In order for
fail2ban
to work, thesshd
service needs to be running, not just the "socket activated" version.sudo systemctl enable ssh.service
(Note thessh
here, NOTsshd
)sudo systemctl start ssh.service
sudo cp /etc/fail2ban/fail2ban.conf /etc/fail2ban/fail2ban.local
sudo nano /etc/fail2ban/fail2ban.local
and add:
[sshd] enabled = true port = 60001 filter = sshd logpath = /var/log/auth.log maxretry = 3 bantime = -1
sudo systemctl enable fail2ban.service
sudo systemctl start fail2ban.service
- TODO: Fails on ARM because
/var/log/auth.log
doesn't exist. - Make a note to come back periodically and check for any banned IPs with
sudo fail2ban-client status sshd
- In order for
- Partition and mount the big drive
lsblk
and confirm the big drive isn't mounted yet. It might be calledsda
ornvme0n1
.sudo parted --list
and confirm it's not partitioned yetsudo fdisk /dev/nvme0n1
n
for new partitionp
for primary- default partition number
- default first sector
- default last sector
p
to print (check)- If your disk is larger than 2TB, don't worry about
fdisk
only supporting sizes up to 2TB. We'll deal with that next. w
to write.
- (Optional) if your disk is biger than 2TB, give it a GPT label
sudo parted /dev/nvme0n1
mklabel gpt
Yes
mkpart primary 0GB 4001GB
(for a 4TB drive)quit
.
- Format the partition:
sudo mkfs -t ext4 /dev/nvme0n1
- Get the UUID for the drive from
sudo blkid
- Append to
/etc/fstab
:sudo nano /etc/fstab
- Add
/dev/disk/by-uuid/YOUR_DISK_UUID /data ext4 defaults 0 2
- The
0
here means thedump
backup program should skip the disk and the2
is the order in whichfsck
will check disks.
- The
sudo mkdir /data
,sudo mount -a
and confirm the drive is mounted withls -lah /data
- Make the drive writable by your user with
sudo chown -R [username]:[username] /data
df -H
and confirm the drive is there and mostly free space- Reboot and make sure the drive mounts again
- Check for firmware updates for the drive
sudo smartctl -a /dev/nvme0n1
and get the current firmware version.- Do some Googling and figure out if there's an update you need.
- Test the performance of the big drive
cd /data
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=150G --readwrite=randrw --rwmixread=75
- Output is explained here
- Here's what I got with a
Crucial P3 Plus NVMe M.2
. The pertinent bit is 137k IOPS for read and 45.2k IOPS for write
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 153600MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=535MiB/s,w=177MiB/s][r=137k,w=45.2k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=1538: Wed Apr 24 11:28:31 2024
read: IOPS=55.5k, BW=217MiB/s (227MB/s)(113GiB/531276msec)
bw ( KiB/s): min=148360, max=631704, per=99.93%, avg=221899.05, stdev=27043.69, samples=1062
iops : min=37090, max=157926, avg=55474.71, stdev=6760.91, samples=1062
write: IOPS=18.5k, BW=72.3MiB/s (75.8MB/s)(37.5GiB/531276msec); 0 zone resets
bw ( KiB/s): min=50416, max=207504, per=99.93%, avg=73954.19, stdev=9002.97, samples=1062
iops : min=12604, max=51876, avg=18488.48, stdev=2250.73, samples=1062
cpu : usr=5.60%, sys=17.07%, ctx=21177496, majf=2, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=29492326,9829274,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=217MiB/s (227MB/s), 217MiB/s-217MiB/s (227MB/s-227MB/s), io=113GiB (121GB), run=531276-531276msec
WRITE: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=37.5GiB (40.3GB), run=531276-531276msec
Disk stats (read/write):
dm-1: ios=29446674/9827674, merge=0/0, ticks=32826092/970688, in_queue=33796780, util=99.92%, aggrios=29492341/9842875, aggrmerge=0/259, aggrticks=32827649/1050440, aggrin_queue=33878550, aggrutil=99.91%
nvme0n1: ios=29492341/9842875, merge=0/259, ticks=32827649/1050440, in_queue=33878550, util=99.91%
- Forward ports from the router to the host:
- Any execution client: 30303 (both TCP and UDP)
- Lighthouse (consensus client): 9000 (both TCP and UDP)
- Only while travelling, SSH: 60001 TCP
- Add the PPA and install the package. See Nethermind's docs.
- Create a directory for the Rocks DBs and logs:
mkdir /data/nethermind
- Create a
nethermind
user but do NOT create a home directory and this user should never log in, so they should not have a shell:sudo useradd -M -s /bin/false nethermind
- Create a
systemd
unit file as followssudo nano /etc/systemd/system/nethermind.service
:
[Unit]
Description=Nethermind Ethereum Client
After=network.target
Wants=network.target
[Service]
User=nethermind
Group=nethermind
Type=simple
Restart=always
RestartSec=5
TimeoutStopSec=180
WorkingDirectory=/data/nethermind
EnvironmentFile=/data/nethermind/.env
ExecStart=/usr/share/nethermind/Nethermind.Runner \
--datadir /data/nethermind \
--config /usr/share/nethermind/configs/mainnet.cfg
[Install]
WantedBy=default.target
- Create an
.env
file as followsnano /data/nethermind/.env
:
DOTNET_BUNDLE_EXTRACT_BASE_DIR = /data/nethermind
NETHERMIND_JSONRPCCONFIG_ENABLED = true
NETHERMIND_JSONRPCCONFIG_JWTSECRETFILE = /data/jwtsecret
NETHERMIND_JSONRPCCONFIG_HOST = 0.0.0.0
NETHERMIND_JSONRPCCONFIG_PORT = 8545
NETHERMIND_JSONRPCCONFIG_ENGINEHOST = 0.0.0.0
NETHERMIND_JSONRPCCONFIG_ENGINEPORT = 8551
NETHERMIND_JSONRPCCONFIG_ENABLEDMODULES = [Eth, Subscribe, Trace, TxPool, Web3, Personal, Proof, Net, Parity, Health, Rpc, Admin]
NETHERMIND_JSONRPCCONFIG_ENGINEENABLEDMODULES = [Net, Eth, Subscribe, Web3]
NETHERMIND_HEALTHCHECKSCONFIG_ENABLED = true
NETHERMIND_HEALTHCHECKSCONFIG_UIENABLED = true
# Not working. See bug: https://github.com/NethermindEth/nethermind/issues/5738.
# NETHERMIND_PRUNINGCONFIG_FULLPRUNINGTRIGGER = VolumeFreeSpace
# NETHERMIND_PRUNINGCONFIG_MODE = Full
# NETHERMIND_PRUNINGCONFIG_FULLPRUNINGTHRESHOLDMB 307200
NETHERMIND_JSONRPCCONFIG_ADDITIONALRPCURLS = [http://127.0.0.1:8555|http|admin]
- Make
nethermind
own the file:sudo chown nethermind /data/nethermind/.env
- (Optional and only required if you already started running as root): Change ownership of all data and logs to the
nethermind
user:sudo chown -R nethermind /data/nethermind
sudo chown -R nethermind /usr/share/nethermind
- Start the service and enable it on boot:
sudo systemctl daemon-reload
sudo systemctl start nethermind.service
sudo systemctl status nethermind.service
sudo systemctl enable nethermind.service
- Follow the logs for a bit to check it's working:
journalctl -u nethermind -f
- The (commented) config above will prune the database once the remaining space on the drive falls below 300 GB. Othwerwise pruning will be manual and you'll have to watch your disk space.
- Once up and running, check health with:
curl http://192.168.20.51:8545/health
- Or if you have a GUI and browser: http://192.168.20.51:8545/healthchecks-ui
- To test the websockets subscriptions using
wscat
:npm install -g wscat
wscat -c ws://192.168.20.51:8545
{"method":"eth_subscribe","params":["newHeads"],"id":1,"jsonrpc":"2.0"}
- You should get some JSON back every block.
- Something to note on the executables.
/usr/bin/nethermind
is just a shell script that runs either:/usr/share/nethermind/Nethermind.Runner
, if there are command line args, or/usr/share/nethermind/Nethermind.Launcher
, if there aren't
- There seems to be no need for this. If you just run
/usr/share/nethermind/Nethermind.Runner
, it works. See this bug.
- Go to https://github.com/sigp/lighthouse/releases and find the latest (non-portable) release, with suffix
x86_64-unknown-linux-gnu
. Download, extract and delete it on the host.wget https://github.com/sigp/lighthouse/releases/download/v4.0.1/lighthouse-v4.0.1-x86_64-unknown-linux-gnu.tar.gz
tar -xvf lighthouse-*.tar.gz
rm lighthouse-*.tar.gz
- Make sure it runs:
./lighthouse --version
- Move the binary out of your home dir:
sudo mv ./lighthouse /usr/bin
- Create two users but do NOT create home directories for them and they should never log in, so they should not have a shell:
sudo useradd -M -s /bin/false lighthouse-bn
sudo useradd -M -s /bin/false lighthouse-vc
- Create two
systemd
unit files as follows:sudo nano /etc/systemd/system/lighthouse-bn.service
:
[Unit] Description=Lighthouse Beacon Node Wants=network-online.target After=network-online.target [Service] Type=simple User=lighthouse-bn Group=lighthouse-bn Restart=always RestartSec=5 ExecStart=/usr/bin/lighthouse bn \ --network mainnet \ --datadir /data/lighthouse/mainnet \ --execution-endpoint http://localhost:8551 \ --execution-jwt /data/jwtsecret \ --http \ --http-address 192.168.20.51 \ --http-allow-origin "*" \ --builder http://localhost:18550 \ --graffiti eliotstock \ --suggested-fee-recipient <ADDRESS> \ --checkpoint-sync-url https://mainnet.checkpoint.sigp.io/ [Install] WantedBy=multi-user.target
sudo nano /etc/systemd/system/lighthouse-vc.service
:
[Unit] Description=Lighthouse Validator Client Wants=network-online.target After=network-online.target [Service] User=lighthouse-vc Group=lighthouse-vc Type=simple Restart=always RestartSec=5 ExecStart=/usr/bin/lighthouse vc \ --network mainnet \ --datadir /data/lighthouse/mainnet \ --beacon-nodes http://192.168.20.51:5052 \ --builder-proposals \ --graffiti eliotstock \ --suggested-fee-recipient <ADDRESS> [Install] WantedBy=multi-user.target
- Don't forget to replace
<ADDRESS>
with the Ethereum address to which you want rewards paid. - To open up the Beacon Node API locally:
- Omit
--http-address
and--http-allow-origin
from thebn
file and--beacon-nodes http://192.168.20.51:5052
from thevc
file if you don't need access to the Beacon Node API on your local network. - You can now use the Beacon Node API on port
5052
but only on the local network. Do not NAT this through to the internet or you'll get DoS'ed.
- Omit
- Note that
localhost
is correct on thebn
file, even though the EL client used192.168.20.51
. - You may wish to add
--debug-level warn
to each file later on to reduce log noise. Start with the default ofinfo
though. - Create data directories and change ownership of all data and logs to the
lighthouse
users:sudo mkdir -p /data/lighthouse sudo mkdir -p /data/validator_keys sudo chown -R lighthouse-bn /data/lighthouse sudo chgrp -R lighthouse-bn /data/lighthouse sudo chown -R lighthouse-vc /data/validator_keys sudo chgrp -R lighthouse-vc /data/validator_keys
- Start the services and enable them on boot:
sudo systemctl daemon-reload sudo systemctl start lighthouse-bn.service sudo systemctl status lighthouse-bn.service sudo systemctl enable lighthouse-bn.service sudo systemctl start lighthouse-vc.service sudo systemctl status lighthouse-vc.service sudo systemctl enable lighthouse-vc.service
- Follow the logs for a bit to check it's working:
journalctl -u lighthouse-bn -f
journalctl -u lighthouse-vc -f
- There are no Ubuntu packages for MEV-Boost. Download the latest binary from https://github.com/flashbots/mev-boost/releases.
- Extract and delete the tarball:
tar -xvf mev* && rm mev*.tar.gz
- Move the binary:
mv mev-boost /data
- Pick one or more relays to use.
- If you have no strong opinions about this, just use Ultra Sound:
https://0xa1559ace749633b997cb3fdacffb890aeebdb0f5a3b6aaa7eeeaf1a38af0a8fe88b9e4b1f61f236d2e64d95733327a62@relay.ultrasound.money
- If you do, however, pick one or more from the eth-educators list. You also might like to check https://www.relayscan.io/.
- Create an
mev-boost
user but do NOT create a home directory and this user should never log in, so they should not have a shell:sudo useradd -M -s /bin/false mev-boost
- Create a
systemd
unit file.sudo nano /etc/systemd/system/mev-boost.service
, then paste in the one from the repoREADME
, except:- Change the working directory to
WorkingDirectory=/data
- Change the path to the binary to
ExecStart=/data/mev-boost \
- Put your relay in.
- Change the working directory to
- Change ownership of the binary. This isn't strictly necessary but keeps things tidy.
sudo chown mev-boost /data/mev-boost
sudo chgrp mev-boost /data/mev-boost
- Start the service and enable it on boot:
sudo systemctl daemon-reload
sudo systemctl start mev-boost.service
sudo systemctl status mev-boost.service
sudo systemctl enable mev-boost.service
- Follow the logs for a bit to check it's working:
journalctl -u mev-boost -f
- Generate a JWT token to be used by the clients:
openssl rand -hex 32 | tr -d "\n" > "/data/jwtsecret"
- This file needs to be readable by both the
nethermind
user and thelighthouse-bn
user, so leave it as world readable.
- The first time you sync only, or if you've fallen far behind, use a checkpoint sync endpoint for the beacon node:
lighthouse --network mainnet --datadir /data/lighthouse/mainnet bn --execution-endpoint http://localhost:8551 --execution-jwt /data/jwtsecret --checkpoint-sync-url https://beaconstate.ethstaker.cc
- Get the checkpoint sync URL from https://eth-clients.github.io/checkpoint-sync-endpoints/
- See this thread in the Lighthouse Discord for more details o checks: https://discord.com/channels/605577013327167508/605577013331361793/1019755522985050142
- Do the key management stuff for Lighthouse: https://lighthouse-book.sigmaprime.io/key-management.html
- Create a password file for this network:
nano stake.pass
andchmod 600 ./stake.pass
lighthouse --network mainnet account wallet create --name stake --password-file stake.pass
- Write down mnemonic -> sock drawer (not really obvs)
lighthouse --network mainnet account validator create --wallet-name stake --wallet-password stake.pass --count 1
- Create a password file for this network:
- Take a note of how long the initial sync takes. The bottleneck for me is SSD speed. If you ever need to re-sync, you'll feel the pain of potentially missing a block proposal the longer this takes. I had to re-sync when I forgot to set any pruning command line args for NM and filled up my disk. As of NM v1.19, a re-sync takes two to three days for me with this drive.
- To dig deeper on I/O performance:
sudo apt install sysstat
sudo nano /etc/default/sysstat
and changefalse
totrue
- Reboot and restart all processes
sar
and check the%iowait
columniostat
(oriostat -x 1
for repeated sampling) and check%util
for the SSD.
- Get yourself a new address to use as the fee recipient address. Should be on a hardware wallet, seed phrase secure etc. Don't worry about the withdrawal address at this point.
- On the staking machine:
- Download, extract and tidy up the staking deposit CLI.
- Go to https://github.com/ethereum/staking-deposit-cli/releases/ and copy the URL of the latest version of the CLI.
cd /data
wget <paste URL here>
tar -xvf staking*
rm staking*.tar.gz
mv staking*/deposit .
rmdir staking*
- Go offline before generating the mnemonic. In a perfect world you do this on an air-gapped machine with a fresh OS installation that's never been online. But having the validator keys on the staking machine itself at the end is convenient, so simply doing it on the staking machine while offline is acceptable, imo. Reboot before and after generating the mnemonic.
- Run it and record the mnemonic. We'll generate two keys but use only one for now.
./deposit new-mnemonic --num_validators 2 --chain mainnet
- The password that secures your validator keystore(s) doesn't need to be super secure. Someone with these keys can sabotage your validator performance but can't withdraw your stake. It will be written in plain text to the filesystem when Lighthouse imports the keys anyway.
- This will generate:
./validator_keys/deposit_data-*.json
./validator_keys/keystore-m_12381_3600_0_0_0-1663727039.json
- Remember this mnemonic can be used to regenerate both the signing key and the withdrawal key for later after Shanghai, although you'll get a slightly different keystore file if you do, even if you use the same password.
- Import only the first of the two keystores into the validator:
lighthouse --network mainnet --datadir /data/lighthouse/mainnet account validator import --keystore /data/validator_keys/keystore-m_12381_3600_0_0_0-*.json
and enter the password for the keystore.
- Back the keystore up onto a USB drive
- First format the drive:
lsblk
, plug the drive in, 'lsblk' again to spot the name of the device. Might be/dev/sda
.- Unmount any paritions if they're mounted:
sudo umount /dev/sda1
,sudo umount /dev/sda2
. - Make a new partition table:
sudo fdisk /dev/sda
,o
,n
,p
,1
, default, default,w
- Format the new partition:
sudo mkfs.vfat -F 32 -n 'keys' /dev/sda1
- Eject:
sudo eject /dev/sda
- Reinsert the drive
sudo mkdir /media/usb
sudo mount -t vfat /dev/sdb1 /media/usb
sudo cp -r /data/validator_keys /media/usb
sudo eject /media/usb
- First format the drive:
- Download, extract and tidy up the staking deposit CLI.
- On the machine where you have MetaMask and your hardware wallet connected:
- Run through the checklist at https://launchpad.ethereum.org/en/checklist and make sure everything is tickety-boo.
- Get to https://launchpad.ethereum.org/en/upload-deposit-data where you upload your deposit data.
- Plug in the USB drive and mount it.
- Get as far as https://launchpad.ethereum.org/en/generate-keys in the Launchpad flow. This is where you upload your deposit data JSON file, connect using the account you have on your hardware wallet, and pay the 32 ETH.
- If you generated more keys than you needed:
- Edit the deposit data JSON file down to just the ones you're funding, and
- Edit the
/data/lighthouse/mainnet/validators/validator_definitions.yml
file to disable the other validators.
- This transaction cost 50,634 gas, which was 0.00065 ETH at the time when I last did it. Having 0.001 ETH in your accouint to cover gas should be more than enough.
- Copy the public keys of the validator(s) you're funding from
/data/lighthouse/mainnet/validators/validator_definitions.yml
so you can paste them into beaconcha.in later (see Monitoring below)
- Create a user on https://beaconcha.in/.
- Got to https://beaconcha.in/user/notifications and add your validator.
- Get the mobile app.
- Sign in on the mobile app.
- Consider turning off the missed attestation notification after a week or so of smooth running. They're quite noisy and if you get too many notifications, you risk missing a more important one such as being offline.
- Run
tmux
first, to get output from multiple services on the screen at once.tmux
refresher:- Create four panes with
C-b "
- Enter command prompt mode with
C-b :
, then:- Make the panes evenly sized:
select-layout even-vertical
- Give the panes titles of their index and the command running in them:
set -g pane-border-format "#{pane_index} #{pane_current_command}"
- Make the panes evenly sized:
- Move around the panes with
C-b [arrow keys]
- Kill a pane with
C-b C-d
- Dettach from the session with
C-b d
- Create four panes with
- Tail the logs for each running
systemd
service, one per pane (ccze
is a logs colouriser):journalctl -u nethermind -f | ccze
journalctl -u mev-boost -f | ccze
journalctl -u lighthouse-bn -f | ccze
journalctl -u lighthouse-vc -f | ccze
- Once you know your validator node index, you can get the current balance of your validator with
curl http://localhost:5052/eth/v1/beacon/states/head/validators/{index}
. - Check disks have space:
df -h
- To prune Nethermind, call the
admin_prune
RPC endpoint, like this. - Expect this to take a day or two. Definitely don't do this if you're about to be in the sync committee.
- To prune Nethermind, call the
- Check CPU load average:
htop
. Should be 0.70 max, times the number of cores. So on an 8-core machine, a load average of 5.6 is the threshold at which the machine is getting overloaded. - Check RAM available:
htop
, seeMem
. - Check internet connectivity and speed:
sudo apt install speedtest-cli
speedtest --secure
- My results: ~250 Mbit/s down, ~90 Mbit/s up.
- Minumum according to some Googling: 10 Mbit/s either way.
- If your router is a bit rubbish, like mine, you might want to preemptively reboot it once a month rather than have it go down in the middle of the night.
- Check logs:
- Execution client: No log levels in logs. Just
grep rror /data/nethermind/logs/mainnet.logs.txt
- Beacon Node:
grep -e WARN -e ERRO -e CRIT /data/lighthouse/mainnet/beacon/logs/beacon.log
- Validator:
grep -e WARN -e ERRO -e CRIT /data/lighthouse/mainnet/validators/logs/validator.log
- Google any errors.
- Upgrade to latest stable versions if necessary.
- Ask in the Discord server for the client about any errors if they persist after upgrade.
- Execution client: No log levels in logs. Just
- Check the ports you're listening on:
sudo lsof -nP -iTCP -sTCP:LISTEN +c0 | grep IPv4
- Ignoring the OS services such as
sshd
, you should have:
Port | Process |
---|---|
8545 |
EL client, JSON RPC for general use |
8551 |
EL client, JSON RPC for the CL client only |
9000 |
CL client, for the EL client |
5052 |
CL client, Beacon Node API for general use |
18550 |
MEV Boost |
- Subscribe to the repos to get an email on a new release. For each of these, drop down 'Watch', go to 'Custom' and check 'Releases'.
- If using a PPA, the update to the binary will happen automatically on new releases, but there's no automated restart of the process after that AFACT. It might also take some time to install. If you're in a rush to upgrade:
sudo apt update
apt list --upgradable
and expect to seenerthermind
in there.- If it's very soon after the release the binary may still be being built. Check https://launchpad.net/~nethermindeth/+archive/ubuntu/nethermind/+packages.
sudo apt upgrade
- If you followed the above instructions, here's what you're using:
nethermind
: PPAlighthouse
: binarymev-boost
: binary
- To check which version you're currently running, run these. But be aware that in the case of a PPA, the running process may in fact still be the older version. You'll need to restart to get the new version to start executing.
nethermind --version
lighthouse --version
/data/mev-boost --version
- On each new release:
- Follow the instructions above again to get a new binary.
- Overwrite the existing binary as root.
- If it's executable as anyone, there's no need to
chown
andchgrp
it back to the process owner as root. - Restart the process with
sudo systemctl restart [service]
- Check the logs in the tmux session for a successful restart.
To stop staking, which is different to withdrawal:
lighthouse account validator exit
- Open up your chosen SSH port (eg. 60001) on your NATs before you go.
- Make sure the laptop you take has the SSH keys for the staking machine - test connecting from outside, using a mobile internet connection.
- Disable the SSH port NAT when you get back.
- Don't panic about losing your stake. The withdrawal private key is not on the machine and can be generated from the mnemonic only. We haven't even generated that private key yet, anywhere.
- Don't go and build a new staking machine, regenerate the same keys and start running again. You'll risk slashing by double signing if:
- You didn't turn on Lighthouse's doppelganger protection AND
- The thief powers on the machine AND
- The machine manages to find the router on the thief's network because the subnet is the same or the thief cared enough to detect the expected subnet AND
- The processes are all set up to start on boot using
systemd
OR - The thief got the user password and started all the services
- So is it worth turning on doppelganger protection?
- Probably not. If the machine password is secure and you didn't set up
systemd
services, the chances of the machine ever validating again are low and the cost of DP is two or three epochs of rewards on an upgrade or restart.
- Probably not. If the machine password is secure and you didn't set up
- Instead, wait a month or two and monitor your validator index.
- If it never starts running again, it's low enough risk to build a new machine and start validating again with regenerated keys.
- There's little point in the thief extracting the validator key from the machine, because it can't be used without the password anyway.
- See also this discussion
- Follow everything above again on the new machine but only up to the point of creating the validator keys. The goal is to avoid slashing for double signing, so you only want one machine running at a time with the keys on it.
- You can get the new machine sync'ed to the point of one of the earlier phases in snap sync and be ready for the validator to work. You don't need to wait for the later phases (old headers, old bodies, old receipts).
- You'll sleep better at night knowing you've tested restoring from only the seed phrase so we'll use that approach here but other options are available:
- You can physically move the
/data/lighthouse/mainnet/validators
directory to the new machine, assuming the old one is still around. - Lighthouse has a
move
command.
- You can physically move the
- Stop all the services on the new machine:
sudo systemctl stop lighthouse-vc
sudo systemctl stop lighthouse-bn
sudo systemctl stop mev-boost
sudo systemctl stop nethermind
- Unplug the Ethernet cable. You'll need to be on the machine itself from here on, obvs, not SSH-ing in.
curl google.com
to check you're really offline (maybe you have WiFi?). You probably don't haveping
installed.- Recreate the validator keys from the mnemonic, making sure you can write to the
/data/validator_keys
directory first:
cd /data
sudo chown [username]:[username] validator_keys
deposit --language en existing-mnemonic
- At the
Create a password
step, you are NOT entering the old password from the old keystore. You're creating new keystores with the same keys in them, with a new password. - Compare the
keystore*
files contents to the ones on the old machine, if you still have access to it. Only thepubkey
value needs to match though. - Compare the
deposit_data-*.json
files. At least thepubkey
,withdrawal_credentials
andsignature
values should match. - If everything looks good, decomission the old machine:
- Stop all services (see above)
cd /data/lighthouse/mainnet
sudo mv validators validators.do_not_use
- Start all services
tmux attach
and expect to see the validator NOT running. The VC process should be loggingNo validators present
.- Unplug the Ethernet cable on the old machine to be safe.
- Wait two epochs to be safe.
- Recreate the validator on the new machine:
lighthouse --network mainnet account validator import --directory /data/validator_keys --datadir /data/lighthouse/mainnet
sudo chown -R lighthouse-vc:lighthouse-vc /data/lighthouse/mainnet/validators
- If you're not running all of the validators you created keys for, edit
/data/lighthouse/mainnet/validators/validator_definitions.yml
and setfalse
on the ones you don't want. - Start all the services and tail the logs
sudo systemctl stop nethermind
sudo systemctl stop mev-boost
sudo systemctl stop lighthouse-bn
sudo systemctl stop lighthouse-vc
tmux attach
- Once the only issue in the logs is that you're offline, plug in the Ethernet cable.
- Delete the NATs to the old machine and create new ones to the new machine. See above.
- Lighthouse BN logs should start to say
INFO Syncing
. - Check https://beaconcha.in/. In time, you should be so back.
- See "Unstaking" above.
sudo ufw allow 5052/tcp comment 'beacon node api'
sudo ufw reload
- Get the local IP of your host with
ip a
(eg.http://192.168.20.51:5052/
) - You won't be able to use the Swagger UI version hosted by the Ethereum Foundation repo at https://ethereum.github.io/beacon-APIs, because it uses
https
. When the browser makes requests to your beacon node, they'll be over straighthttp
and you'll get aMixed content
error on the browser console. You could serve the API overhttps
but it's easier not to. - Instead just run Swagger UI locally.
git clone git@github.com:ethereum/beacon-APIs.git
cd beacon-APIs
python3 -m http.server 8080
- Open http://localhost:8080 and set the version to
dev
(release number version will fail because we haven't built thereleases
directory). - Set the
server_url
tohttp://192.168.20.51:5052/
- Test some API endpoints.
If this seems like a ton of work, you can forget most of the above and just install and run sedge
: https://docs.sedge.nethermind.io/docs/quickstart/install-guide
- Expect
segde
to use about 1TB per month in bandwidth. It'll be more while sync'ing, then decrease. - To start the
sedge
containers once installed:sudo docker compose -f docker-compose-scripts/docker-compose.yml up -d execution consensus
- To stop them:
sudo docker compose -f docker-compose-scripts/docker-compose.yml down
- Running as root was necessary on my host, but can be avoided (Google it).
Rough notes on setting up a separate machine for Juno.
sudo apt install git gcc make tmux lsof
- Don't use the Ubuntu APK for
golang
- it's not recent enough.sudo apt remove golang-1.18 golang-1.18-doc golang-1.18-go golang-1.18-src golang-doc golang-go golang-src
- Grab the tarball URL from https://go.dev/dl/
wget https://go.dev/dl/go1.22.0.linux-amd64.tar.gz
(for example)- Follow instructions at https://go.dev/doc/install
- You'll also need a Rust toolchain. See https://www.rust-lang.org/tools/install.
cd && git clone https://github.com/NethermindEth/juno.git && cd juno
make juno
- Grab the latest snapshot URL from https://github.com/NethermindEth/juno,
wget
it onto the node and extract it to/data/juno/mainnet
. - Get your local Ethereum node running, sync'ed and with the RPC interface up. Take a note of the IP and port for RPC, eg.
ws://192.168.20.41:8545
. - Open a
tmux
session so you can continue execution after you disconnect ssh. - Run Juno with:
./build/juno \ --db-path /data/juno/mainnet \ --http-port 6060 --eth-node ws://192.168.20.51:8545 --log-level DEBUG
- To run with p2p sync (as of 2024-09), get a Sepolia snapshot and add:
--network=sepolia \ --p2p \ --p2p-peers=/ip4/35.231.95.227/tcp/7777/p2p/12D3KooWNKz9BJmyWVFUnod6SQYLG4dYZNhs3GrMpiot63Y1DLYS
- If you can't build juno, you can fall back on running it as a container:
docker run -d \ --name juno \ -p 6060:6060 \ -v /data/juno/mainnet:/var/lib/juno \ nethermind/juno \ --http \ --http-port 6060 \ --http-host 0.0.0.0 \ --db-path /var/lib/juno \ --eth-node ws://192.168.20.51:8545
- Your RPC node is now available (even without waiting for sync to complete) on, eg.
http://192.168.20.53:6060
. You can test this with:curl --location 'http://192.168.20.53:6060' \ --data '{ "jsonrpc":"2.0", "method":"starknet_blockNumber", "id":1 }'
- Note that there are no log files yet. All logging simply goes to the console.
- Configure the firewall
- Confirm
ufw
is installed:which ufw
- Run these:
sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow 22/tcp comment 'ssh' sudo ufw allow out from any to any port 123 comment 'ntp' sudo ufw allow 6060 comment 'juno http' sudo ufw allow 6061 comment 'juno ws' sudo ufw enable
- Note that
http
andhttps
are absent above. - Check which ports are accessible with
sudo ufw status
sudo ufw reload
- Confirm
- Upgrades
cd ~/juno
git pull origin main
(or a release tag)rustup update
- Check https://go.dev/dl/ for the latest golang version.
wget https://go.dev/dl/go1.22.0.linux-amd64.tar.gz
(for example)- Follow instructions at https://go.dev/doc/install
make juno
tmux attach
Ctrl-C
to kill the process- Run the same command line as before the upgrade.