Skip to content

fix(tiering): register spoke-namespace files; harden tier-drop rule - #688

Merged
xe-nvdk merged 2 commits into
mainfrom
fix/spoke-tiering-registration
Sep 2, 2026
Merged

xe-nvdk merged 2 commits into
mainfrom
fix/spoke-tiering-registration

Conversation

@xe-nvdk

@xe-nvdk xe-nvdk commented Sep 2, 2026

Copy link
Copy Markdown
Member

Summary

  • the tiering scanner errored on every spoke-namespace file (one path level deeper than the standard layout), leaving hub-side spoke data invisible to tiering; partition paths are now parsed by their validated date tail, the same approach hub compaction adopted for the identical reason in Edge sync: hub-side compaction of spoke namespaces #619
  • spoke files register under the query-visible split (database = spoke ID, measurement = spoke database), matching what FROM \"rocket-01\".telemetry resolves to, so registered metadata can never diverge from query-time tier assembly
  • spoke files are gated OUT of hot-to-cold migration (follow-up medium(tiering): cold migration of spoke-namespace files needs receipt-aware handling #687): legacy spoke-synced daily files carry sync receipts that a migration delete would make confirmPresent forget, re-accepting duplicates (the Edge sync: HubIndex.Forget MUST be wired before delete_after_sync or hub-side spoke-namespace retention ships #611 hazard class); the gate is driven by the same parser that registers files, after review showed both a content-based heuristic (numeric spoke measurement names) and metadata-based reconstruction (legacy rows) misclassify
  • multi-tier hardening: a verified-empty tier is dropped from a query only when another tier produced a positive pruning result, so spoke-shaped queries (whose generated paths sit one level shallow and existence-filter to empty) keep their full globs instead of losing a tier to a listing error

Process

Same pipeline as #681/#684/#686: adversarial plan review (which overturned both original load-bearing assumptions: the registration split and unconditional migration), independent implementation review (which caught the numeric-measurement gate bypass), licensed e2e before merge.

Verification

  • unit + integration tests: spoke/plain/deep parse shapes incl. numeric spoke measurements, spoke register-but-not-migrate, combineTierPruneResults truth table, spoke-shaped pruning outcome
  • full tiering/api/pruning/edgesync suites pass, including under TZ=America/Costa_Rica
  • licensed e2e with real S3: spoke + plain layout scanned with 0 errors (previously every spoke path errored), migration moved only the plain daily file, spoke files stayed hot, FROM \"rocket-01\".telemetry on the tiered hub returns correct rows, and the migrated plain daily file reads back from cold

Refs #686. Spoke cold migration ships separately via #687.

Ignacio Van Droogenbroeck added 2 commits September 1, 2026 19:26
The tiering scanner errored on every spoke-namespace file (one path
level deeper than the standard layout), so hub-side spoke data was
invisible to tiering. Parse partition paths by their date tail, the
approach hub compaction adopted for the same reason in #619, and
register spoke files under the query-visible split (database = spoke
ID). Spoke files are gated out of cold migration until receipt-aware
handling exists (#687): deleting a legacy spoke-synced daily file's hot
copy would make confirmPresent forget its receipt and re-accept a
duplicate. A verified-empty tier is now dropped from a multi-tier query
only when another tier produced a positive pruning result, so
spoke-shaped queries (whose generated paths sit one level shallow and
filter to empty) keep their full globs instead of losing a tier to a
listing error on the other one.

Refs #686
Review findings: the segment-content gate was defeated by 4-digit
numeric spoke measurement names, and both it and a metadata-based path
reconstruction misclassified legacy or synthetic rows (hour-level daily
files, PartitionTime not derived from the path). The tail parser that
registers files now also classifies them (SpokeNamespaced on
filePathInfo); unparseable paths skip migration conservatively. Year
floor aligned with compaction's isHourLevelFile (>= 1970).

Refs #687
@xe-nvdk
xe-nvdk merged commit 8ae1b8b into main Sep 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant