Prepare OOB to be usable as emergency server uplink

[Firefishy]

The out-of-band Raspberry Pi devices we have in Amsterdam and Dublin has their own independent uplink to the Internet via private 4G network (1NCE).

We should pre-configure these devices to be usable as a potential backup uplink in the event of a full ISP outage.

[Firefishy]

Likely pre-configure NAT and likely site to site wireguard over 4G connection.

[pnorman]

Do these have enough bandwidth to work as an uplink between sites?

[Firefishy]

Do these have enough bandwidth to work as an uplink between sites?

The Teltonika TRM240 4G modem we use supports up to 10 Mbps download / 5 Mbps upload.

Which should be sufficient for syncing a small amount of unsynced (tail) of postgres WAL logs, diffs and osmdbt state.

The new Teltonika TRM500 5G modem supports up to 3.4 Gbps, but we'd hit the 1nce 500MB data limit very quickly. Top-up is €10 per 500MB.

[pnorman]

If we're able to handle a complete site outage (e.g. power goes down or both switches go down) would we still need the OOB uplink?

[Firefishy]

If we're able to handle a complete site outage (e.g. power goes down or both switches go down) would we still need the OOB uplink?

Nothing runs if both power or both switches down. The out-of-band has dual feed power and has serial links to both PDUs and both Switches and can recover from complete config loss.

[pnorman]

Nothing runs if both power or both switches down. The out-of-band has dual feed power and has serial links to both PDUs and both Switches and can recover from complete config loss.

Yes, but the alternate site would be running. My point is about what scenarios we want to recover from. If we want to recover from a complete site outage, that scenario doesn't allow for OOB. We could decide that we're okay with a couple minutes of data loss on a complete outage but no data loss on an upstream network outage, in which case the OOB uplink would be useful.

[Firefishy]

I will be upgrading the out-of-band equipment in Amsterdam on 17th and 18th Feb 2025. I have a new Raspberry Pi 5. The device also has redundant partitions to allow failsafe upgrades ("tryboot")

A new Automatic Transfer Switch for powering the Raspberry Pi will also be installed.

[Firefishy]

I the OOB Raspberry PI was upgraded.

It is now a Raspberry PI 5 configured as follows:

1. Redundant Boot Partitions ("tryboot").
2. Backup Serial Connection to another system. Using built in UART.
3. Now using Modem Manager to manage the 4G connection. Custom UDEV rules to allow Modem Manager to quickly find modem.
4. Power is via Official Raspberry PI 5 power supply plugged into the APC Automatic Transfer Switch. A spare Raspberry PI 5 power supply is pre-installed, wired up and can be connected to Raspberry PI as needed.
5. Failback USB boot memory stick installed in the event the unlikely event the MicroSD card fails.
6. MicroSD card is a SanDisk MAX ENDURANCE 256GB. Claimed to be pSLC and supports an extremely high TBW.
7. Hardware watchdog configured in systemd
8. The root filesystem is mounted with commit=30 interval set.