Repository navigation
docs(iceberg): orphan sweep switch and what bounds the metadata directory (arc#835, arc#1106) - #105
Merged
Merged
Conversation
…tory (arc#835, arc#1106) Adds iceberg.orphan_sweep_enabled to the configuration reference, and a Limitations entry for metadata growth: a removal pass leaves superseded manifests behind, the orphan sweep reclaims the unreferenced ones, and a pass at 12 or more data manifests merges them into one. Also notes that a table the reconciler skips keeps its manifest count until it is written to again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents two shipped Arc changes that the Iceberg page did not mention.
Configuration reference — adds
iceberg.orphan_sweep_enabled(defaulttrue), from Arc #1107 (#835). Worth an entry rather than a footnote because it is the only thing the exporter deletes that nothing regenerates, which is why it has an off switch at all.Limitations — a new entry for metadata growth, from Arc #1124 (#1106). The page previously said nothing about what the metadata directory does over time, and the honest answer is that it grows with the removal history rather than with the data, bounded by two automatic mechanisms:
It also records the one case neither covers: a table whose file set has stopped changing is skipped by the reconciler, so it keeps whatever manifest count it last had until something writes to it again. That is a real thing an operator can observe and otherwise could not explain.
Text-only; no code or examples changed.
Refs Basekick-Labs/arc#835, Basekick-Labs/arc#1106, Basekick-Labs/arc#1107, Basekick-Labs/arc#1124.