Repository navigation
The typed recompute plans its statements for the base it is given - #3
Merged
Merged
Conversation
… a generic plan cannot turn its source cleanup into hours. Signed-off-by: Bogdan Szabo <bogdan@re-cinq.com> (cherry picked from commit 02c10aa)
gedaiu
added a commit
to re-cinq/lore
that referenced
this pull request
Oct 18, 2026
…d by document) (#2810) re.13 = re.12 + the typed recompute planned for its base (re-cinq/utopia#3, deeplethe/utopia#1298), an index on fact_evidence(document_id) (#5, fork migration 9000), the utopia:temporal review fixes (#4) and re-dev CI (#1). Apply with terraform -target='kubernetes_deployment_v1.utopia[0]'.
Author
|
Upstream asked for a different shape, now deeplethe#1307: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Lore's deployment a
materialize_typedjob ran its source cleanup (DELETE FROM typed_fact_sources … NOT EXISTS (…), step 1 ofmaterialize_in_tx) for over three hours on a base of about 438,000 facts and 200,000 typed-fact sources. It held the database at its 4-CPU limit, and a second recompute of the same base waited on the advisory lock behind it.The statement is fast when planned for its base. On a copy of that base's tables (same rows, same indexes,
work_mem16MB):sqlx prepares statements, and after a few runs on a connection Postgres switches to a generic plan. That plan can't see how many rows
t.kb_id = $1selects. It guesses about four statements and picks a nested-loop anti-join that rescans about 356,000 materialized rows for each of the base's 200,000 sources.materialize_in_txnow starts withSET LOCAL plan_cache_mode = force_custom_plan. It is scoped to the transaction and covers both entry points, the worker'smaterializeand the job'stry_materialize. The new testa_recompute_plans_its_statements_for_the_base_it_is_givenchecks the recompute's transaction plans that way, and fails without the change; a timing test would be flaky.Upstream: proposed on deeplethe#1298, held until the maintainer agrees the shape. Not a decision record: it's a plan fix with no behaviour change.
🤖 Generated with Claude Code