pageserver: startup: ensure local disk state is durable

[problame]

refs #6989

Problem

After unclean shutdown, we get restarted, start reading the local filesystem,
and make decisions based on those reads. However, some of the data might have
not yet been fsynced when the unclean shutdown completed.

Durability matters even though Pageservers are conceptually just a cache
of state in S3. For example:

• the cloud control plane is no control loop => pageserver responses
  to tenant attachmentm, etc, needs to be durable.
  • the storage controller does not rely on this (as much?)
• we don't have layer file checksumming, so, downloaded+renamed but not
  fsynced layer files are technically not to be trusted
  • Epic: Layer checksums #2683

Solution

syncfs the tenants directory during startup, before we start reading from it.

This is a bit overkill because we do remove some temp files (InMemoryLayer!)
later during startup. Further, these temp files are particularly likely to
be dirty in the kernel page cache. However, we don't want to refactor that
cleanup code right now, and the dirty data on pageservers is generally
not that high. Last, with direct
IO we're going to
have near-zero kernel page cache anyway quite soon.

[github-actions]

3766 tests run: 3660 passed, 0 failed, 106 skipped (full report)

---

Code coverage* (full report)

• functions: 32.2% (7255 of 22565 functions)
• lines: 50.3% (58802 of 116957 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
307ace2 at 2024-08-26T15:56:21.461Z :recycle:}