pageserver: refine how we delete timelines after shard split

[jcsp]

Problem

Previously, when we do a timeline deletion, shards will delete layers that belong to an ancestor. That is not a correctness issue, because when we delete a timeline, we're always deleting it from all shards, and destroying data for that timeline is clearly fine.

However, there exists a race where one shard might start doing this deletion while another shard has not yet received the deletion request, and might try to access an ancestral layer. This creates ambiguity over the "all layers referenced by my index should always exist" invariant, which is important to detecting and reporting corruption.

Now that we have a GC mode for clearing up ancestral layers, we can rely on that to clean up such layers, and avoid deleting them right away. This makes things easier to reason about: there are now no cases where a shard will delete a layer that belongs to a ShardIndex other than itself.

Summary of changes

• Modify behavior of RemoteTimelineClient::delete_all
• Add test_scrubber_physical_gc_timeline_deletion to exercise this case
• Tweak AWS SDK config in the scrubber to enable retries. Motivated by seeing the test for this feature encounter some transient "service error" S3 errors (which are probably nothing to do with the changes in this PR)

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2104 tests run: 2037 passed, 0 failed, 67 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_subscriber_restart: release

Code coverage* (full report)

• functions: 32.8% (7122 of 21684 functions)
• lines: 50.4% (57360 of 113737 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
b0fd1eb at 2024-08-01T18:32:13.796Z :recycle:}