Skip to content

perf(wintermute): create bulk staging tables once per connection - #240

Open
afbase wants to merge 1 commit into
fix/wintermute-suppress-notice-floodfrom
perf/wintermute-hoist-bulk-staging-ddl
Open

perf(wintermute): create bulk staging tables once per connection#240
afbase wants to merge 1 commit into
fix/wintermute-suppress-notice-floodfrom
perf/wintermute-hoist-bulk-staging-ddl

Conversation

@afbase

@afbase afbase commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Stacked on the client_min_messages change; review that one first.

Each bulk path issued CREATE TEMP TABLE IF NOT EXISTS for its staging table on every
batch. Pools recycle with RecyclingMethod::Fast, which never discards temp tables, so
after the first batch on a connection that DDL did nothing but raise a duplicate_table
NOTICE and pay for a catalog lookup.

This hoists the eleven declarations into BULK_STAGING_DDL and runs them once per
connection from a post_create hook in a new config::create_pg_pool. The batch paths
now only TRUNCATE.

The invariant that introduces is that a pool whose connections reach indexer::bulk must
be built by create_pg_pool, or those paths fail with undefined_table. The indexer
pools, the indexer test pool, and the car_loader, direct_index and firehose_catchup
binaries are the pools that reach it, and all now route through it. The ingester pools do
not reach the bulk paths and are unchanged. A test asserts every truncated table is
declared, because drift there surfaces at runtime rather than in review.

Test plan

  • cargo clippy -p rsky-wintermute --all-targets --no-deps -- -D warnings
  • cargo test -p rsky-wintermute against a database with the appview schema — the
    indexer::tests suite is what actually exercises the staging tables
  • cargo build --release -p rsky-wintermute
  • direct_index indexes a repo end to end on a fresh connection
  • car_loader loads a CAR end to end
  • firehose_catchup drains a backlog
  • Indexer throughput unchanged, and no undefined_table errors in the logs

Each bulk path issued CREATE TEMP TABLE IF NOT EXISTS for its staging table on
every batch. Pools recycle with RecyclingMethod::Fast, which never discards temp
tables, so the DDL was redundant after the first batch on a connection and only
served to raise a duplicate_table NOTICE.

Hoist the eleven declarations into BULK_STAGING_DDL and run them once per
connection from a post_create hook in config::create_pg_pool. The batch paths
now only TRUNCATE.

Pools whose connections reach these functions must be built by create_pg_pool or
the staging tables will not exist, so the indexer pools, the indexer test pool,
and the car_loader, direct_index and firehose_catchup binaries all route through
it. A test asserts every truncated table is declared, since a mismatch would
surface as undefined_table at runtime rather than in review.
@afbase
afbase requested a review from rudyfraser August 24, 2026 04:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant