perf(wintermute): create bulk staging tables once per connection - #240
Open
afbase wants to merge 1 commit into
Open
perf(wintermute): create bulk staging tables once per connection#240afbase wants to merge 1 commit into
afbase wants to merge 1 commit into
Conversation
Each bulk path issued CREATE TEMP TABLE IF NOT EXISTS for its staging table on every batch. Pools recycle with RecyclingMethod::Fast, which never discards temp tables, so the DDL was redundant after the first batch on a connection and only served to raise a duplicate_table NOTICE. Hoist the eleven declarations into BULK_STAGING_DDL and run them once per connection from a post_create hook in config::create_pg_pool. The batch paths now only TRUNCATE. Pools whose connections reach these functions must be built by create_pg_pool or the staging tables will not exist, so the indexer pools, the indexer test pool, and the car_loader, direct_index and firehose_catchup binaries all route through it. A test asserts every truncated table is declared, since a mismatch would surface as undefined_table at runtime rather than in review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on the
client_min_messageschange; review that one first.Each bulk path issued
CREATE TEMP TABLE IF NOT EXISTSfor its staging table on everybatch. Pools recycle with
RecyclingMethod::Fast, which never discards temp tables, soafter the first batch on a connection that DDL did nothing but raise a
duplicate_tableNOTICE and pay for a catalog lookup.
This hoists the eleven declarations into
BULK_STAGING_DDLand runs them once perconnection from a
post_createhook in a newconfig::create_pg_pool. The batch pathsnow only
TRUNCATE.The invariant that introduces is that a pool whose connections reach
indexer::bulkmustbe built by
create_pg_pool, or those paths fail withundefined_table. The indexerpools, the indexer test pool, and the
car_loader,direct_indexandfirehose_catchupbinaries are the pools that reach it, and all now route through it. The ingester pools do
not reach the bulk paths and are unchanged. A test asserts every truncated table is
declared, because drift there surfaces at runtime rather than in review.
Test plan
cargo clippy -p rsky-wintermute --all-targets --no-deps -- -D warningscargo test -p rsky-wintermuteagainst a database with the appview schema — theindexer::testssuite is what actually exercises the staging tablescargo build --release -p rsky-wintermutedirect_indexindexes a repo end to end on a fresh connectioncar_loaderloads a CAR end to endfirehose_catchupdrains a backlogundefined_tableerrors in the logs