Skip to content

get_store_metrics' inventory is blind to 57% of the store it reports on: continuous aggregates and un-enumerated plain tables never appear, and the total is never reconciled #3582

Closed

Description

@erikdarlingdata

Summary

get_store_metrics reports the store's object inventory by three kinds — hypertable, dimension, background job — and on the largest production store that inventory accounts for 159 GiB of a 415 GiB database. The other 256 GiB is (a) the twenty continuous-aggregate materializations (~235 GiB), which are hypertables the tool never enumerates because timescaledb_information.hypertables lists them under their internal _materialized_hypertable_N names and the collector's catalog walk keys on collector target tables, and (b) plain tables the walk does not name — collect.query_store_text at 15 GiB (V74 stores statement text inline, by design), query_store_plan_map, the alert log, and siblings. The tool answered "here is the store" for 38% of it, and a growth investigation that trusted the tool first went looking for the missing bytes inside the one object it did measure correctly.

Fix shape

  1. Aggregates as first-class inventory rows: enumerate timescaledb_information.continuous_aggregates and size each via its materialization hypertable (hypertable_detailed_size on materialization_hypertable_schema.materialization_hypertable_name), reported under the aggregate's user-facing view name with a new object_kind (e.g. continuous_aggregate), carrying compression_enabled and its retention/compression policy presence — the fields the sibling issue (The continuous aggregates are 235 GiB and none are compressed: the rollups outweigh their 9x-compressed source, and they were never on the compression ladder #3581) needed and had to get by hand.
  2. A catch-all row for un-enumerated relations: everything in the database's user schemas not otherwise inventoried, summed, over a size floor, as object_kind = 'other' with a count — so the inventory's total is reconciled against pg_database_size and the tool can state its own coverage ("inventory covers 99.6% of the database; 1.7 GiB in 14 un-enumerated relations"). A reconciliation that does not close is itself a finding.
  3. Name the biggest un-enumerated plain tables explicitly where the product knows them (query_store_text, query_store_plan_map, config_alert_log): they are stable, product-owned tables, and the inventory should know them by name rather than lump them.
  4. Tool description + get_store_metrics note: state what the inventory covers and the reconciliation figure.

Same family as wave 1's payload-honesty work (#3541): a correct query producing a wrong conclusion because the envelope did not state its coverage.

Evidence: ground-truth pg_class size census across all schemas on the largest store vs the tool's inventory, 14:46Z, reconciled to the byte; bus record 20260918T144642Z.

Activity

  1. erikdarlingdata commented on Sep 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed in #3610 (merged 17:00Z, rides the next nightly). The hourly sweep now inventories every continuous aggregate under its view name (sized through its materialization, carrying compression_enabled and its refresh/compression/retention job ids read live), the named plain tables, and two catch-all rows (user-schema other / system) — and get_store_metrics reconciles each sweep's rows against that sweep's pg_database_size, stating enumerated and attributed coverage, the residual, a reconciled verdict against a 1% / 64 MiB bar, and the ten largest un-enumerated relations live. On the rig the rows summed exactly to Σpg_total_relation_size over every relation; the residual was the directory's non-relation files. Mechanism correction for the record: timescaledb_information.hypertables EXCLUDES materializations (mat_hypertable_id IS NULL) rather than listing them under internal names — same effect on the inventory, different reason. The same PR lands #3574's managed-mode self-proof (the owner-pool sweep persists its own job_history count; the block shows it beside the mcp role's verdict) and the two truth fixes (GUC-off arm: failed runs are written regardless; README: the mcp role is live whenever the store is managed).

  2. added a commit that references this issue on Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions