security: Run services that process external data it their own tokio executors, to avoid denial of service attacks

[teor2345]

Motivation

Zebra services that process external data are vulnerable to denial of service attacks, which cause them to use a lot of CPU, or block other tasks. It's tricky to find all of these kinds of issues. So we can limit their impact by running some tasks independently.

Excess CPU usage or blocking could also enable timing attacks on the scanner. A timing attack guesses secret data by running secret and public tasks, and seeing how the timing is different when they run together.

Specifications

Launching:
https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.new_multi_thread
https://docs.rs/tokio/latest/tokio/runtime/struct.Builder.html#method.build

Running/Shutdown:
https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.block_on
https://docs.rs/tokio/latest/tokio/runtime/struct.Runtime.html#method.shutdown_timeout

Complex Code or Requirements

We need to shut down these extra executors when Zebra shuts down. Sometimes launching or shutdown needs an extra std::thread wrapper, to get out of the async context.

We already do this with the RPC server:

zebra/zebra-rpc/src/server.rs

Lines 286 to 320 in 4306a00

	/// Shut down this RPC server asynchronously.
	/// Returns a task that completes when the server is shut down.
	pub fn shutdown(&self) -> JoinHandle<()> {
	let close_handle = self.close_handle.clone();
	let span = Span::current();
	tokio::task::spawn_blocking(move || {
	span.in_scope(|| Self::shutdown_blocking_inner(close_handle))
	})
	}
	/// Shuts down this RPC server using its `close_handle`.
	///
	/// See `shutdown_blocking()` for details.
	fn shutdown_blocking_inner(close_handle: CloseHandle) {
	// The server is a blocking task, so it can't run inside a tokio thread.
	// See the note at wait_on_server.
	let span = Span::current();
	let wait_on_shutdown = move || {
	span.in_scope(|| {
	info!("Stopping RPC server");
	close_handle.clone().close();
	debug!("Stopped RPC server");
	})
	};
	let span = Span::current();
	let thread_handle = std::thread::spawn(wait_on_shutdown);
	// Propagate panics from the inner std::thread to the outer tokio blocking task
	span.in_scope(|| match thread_handle.join() {
	Ok(()) => (),
	Err(panic_object) => panic::resume_unwind(panic_object),
	})
	}

Suggested Fixes

Run these tasks in their own tokio executors:

• zebra-network and its tasks
• the mempool service and its tasks
• the block syncer and its tasks
• the inbound service and its tasks
• zebra-scan (security: Run the scanner it is own tokio executor, and use low-priority threads to avoid timing attacks #8043)
• zebra-rpc (already implemented)

Testing

We can block one executor and check the others still keep running.

Related Work

#8043