iomgr: dedicated sync-IO reactor for blocking drive I/O (iomgr::sync_wait)

[szmyd]

Why

v13 removed fibers (88e6f8d, d4f8aab) and, with them, the native DriveInterface::sync_write/sync_read
that let a reactor issue a blocking drive op by yielding the calling fiber while the reactor kept reaping
completions. The public drive surface is now async-only (async_* -> io_op, co_await).

Cold-path callers (superblock / chunk-info / metadata reads & writes during boot, checkpoint, recovery) still
need a synchronous drive op. Building that on a std::future cannot work on a reactor thread: the reactor
blocks in fut.get() and can never reap the io_uring completion that would unblock it → deadlock. (homestore
papered over this with a drive_bridge::sync_wait that HS_REL_ASSERT(!am_i_io_reactor()) — which just moved
the problem onto callers and aborted whenever a blocking op was reached from a reactor.)

What

A single, internal sync-IO reactor plus a blocking primitive that is safe to call from any thread:

• iomgr::sync_wait(io_op) -> io_result (io_op.hpp, drive.cpp) — runs the op on the dedicated sync
  reactor and blocks the caller until completion. Because the op is issued and reaped on a reactor that is
  never itself a sync_wait caller, the caller blocking can never deadlock the reactor driving the completion.
  Safe both off-reactor and on-reactor. (Debug-asserts it is not called on the sync reactor.)
• IOManager::start_sync_reactor() (iomgr.cpp) — creates the iomgr_sync_io reactor at the end of
  start(); IOManager::sync_io_reactor() accessor + m_sync_reactor member (iomgr.hpp). It is a user
  reactor, drive-capable like any other (the drive interface has all_io scope), kept off the worker pool.
• Isolation — match_regex() excludes the sync reactor from every reactor_regex (iomgr.cpp). Both
  message broadcasts (run_on_*) and scoped iodev/timer distribution (generic_interface::add_io_device) route
  through match_regex, so this keeps user closures and timer fds off it: no all_user/all_io broadcast can
  run a closure there (which could itself call sync_wait → self-deadlock), and broadcast counts (e.g.
  all_user) stay correct. It is addressed only directly by pointer — the sync_wait dispatch and the shutdown
  stop.
• Shutdown — the sync reactor increments m_yet_to_stop_nreactors at start but is hidden from the all_io
  stop broadcast, so IOManager::stop() stops it explicitly by pointer before the broadcast (otherwise shutdown
  would hang waiting on it).

Teardown robustness fix (surfaced by the above)

• IOReactorEPoll::remove_iodev_impl now tolerates EPOLL_CTL_DEL -> ENOENT (reactor_epoll.cpp). A
  reactor does not track which iodevs it actually ADDed; IOInterface::on_reactor_stop walks the whole iodev
  map and removes each regardless of scope, so it can DEL a scoped fd (e.g. an all_user/all_worker global
  timer) it never added, or one a prior cancel already removed. ENOENT means "already absent" — the desired
  post-state of a remove — so it returns success instead of LOGDFATAL/abort. (Pre-existing asymmetry, only
  reachable once blocking I/O stopped aborting first; reasonable to split into its own commit.)

Tests

• New test_sync_io (TestSyncIo-Epoll): off_reactor, on_reactor (the defining case — a blocking sync
  op issued from a worker reactor completes, no deadlock), concurrent_from_reactors.
• Full ctest suite green; test_sync_io + test_drive clean under TSAN (Sanitized-thread).

Reviewer notes

• This is the iomgr half. The homestore consumer change lands separately: drive_bridge::sync_wait becomes a
  thin delegate to iomgr::sync_wait (dropping its !am_i_io_reactor() assert), and homestore's interim
  BlockingIoExecutor is deleted.
• Files: io_op.hpp (+9), iomgr.hpp (+11), drive.cpp (+28), iomgr.cpp (+33), reactor_epoll.cpp (+11),
  test/CMakeLists.txt (+5), test/test_sync_io.cpp (+141).

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 50.00000% with 15 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (dev/v13.x@360ee83). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/lib/drive.cpp	40.00%	0 Missing and 9 partials ⚠️
src/lib/iomgr.cpp	66.66%	0 Missing and 4 partials ⚠️
src/lib/epoll/reactor_epoll.cpp	0.00%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff              @@
##             dev/v13.x     #126   +/-   ##
============================================
  Coverage             ?   45.42%           
============================================
  Files                ?       35           
  Lines                ?     1409           
  Branches             ?      628           
============================================
  Hits                 ?      640           
  Misses               ?      336           
  Partials             ?      433           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.