feat: low-lock constraint reapplication via two-phase NOT VALID + VALIDATE#853
Open
marcus-kempe wants to merge 3 commits into
Open
feat: low-lock constraint reapplication via two-phase NOT VALID + VALIDATE#853marcus-kempe wants to merge 3 commits into
marcus-kempe wants to merge 3 commits into
Conversation
…IDATE When reapplying constraints on large, actively-used partition sets, the AccessExclusiveLock required by ALTER TABLE ADD CONSTRAINT blocks all concurrent operations for the duration of the constraint validation scan. On partitions with millions of rows this can block production traffic for extended periods. This change introduces three improvements to minimize lock contention: 1. Two-pass column loop in apply_constraints(): all min/max value scans are gathered first (AccessShareLock only), then all DDL statements are executed together, preventing one column's AccessExclusiveLock from being held during another column's potentially slow min/max scan. 2. NOT VALID constraint validation: when an existing partman constraint is NOT VALID and constraint_valid is true, apply_constraints() now validates it using ALTER TABLE VALIDATE CONSTRAINT which only requires SHARE UPDATE EXCLUSIVE lock (compatible with concurrent DML). 3. p_low_lock parameter on reapply_constraints_proc(): when true, each partition's constraints are applied in two phases: a) Add constraints as NOT VALID (brief AccessExclusiveLock, instant DDL) b) COMMIT to release the lock c) Validate constraints (SHARE UPDATE EXCLUSIVE, DML-compatible) This replaces the need for manual constraint_valid config toggling and separate proc calls. A p_force_not_valid parameter is also added to apply_constraints() to support the two-phase approach from the procedure level. Made-with: Cursor
- Add Dockerfile, docker-compose.yml, and run_tests.sh for running pgtap tests in a containerized PostgreSQL 17 environment - Fix plan count in low-lock part2 test (16 assertions, not 17) Made-with: Cursor
Verifies that the two-pass column loop in apply_constraints() correctly creates and validates constraints for multiple constraint columns (col1 and col2) via p_low_lock mode. Both columns get valid constraints, confirming the deferred DDL approach produces identical results. Made-with: Cursor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When reapplying constraints on large, actively-used partition sets, the
AccessExclusiveLockrequired byALTER TABLE ADD CONSTRAINTblocks all concurrent operations for the duration of the constraint validation scan. On partitions with millions of rows this can block production traffic for extended periods.This PR introduces a
p_low_lockparameter toreapply_constraints_procthat applies constraints in two phases per partition:NOT VALID(briefAccessExclusiveLockfor instant DDL only)COMMITto release the lockSHARE UPDATE EXCLUSIVE(compatible with concurrent DML)Additional improvements
Two-pass column loop in
apply_constraints(): when multipleconstraint_colsare configured, all min/max value scans are gathered first (AccessShareLockonly), then all DDL statements are executed together. This prevents one column'sAccessExclusiveLockfrom being held during another column's potentially slow min/max scan.NOT VALID validation: when
apply_constraints()finds an existing partman constraint that isNOT VALIDandconstraint_validistrue, it now validates it usingALTER TABLE ... VALIDATE CONSTRAINT(SHARE UPDATE EXCLUSIVE) instead of skipping it.p_force_not_validparameter onapply_constraints(): allows the caller to overrideconstraint_validand forceNOT VALIDconstraint creation, used internally by thep_low_lockpath.Usage
Background
This addresses the same class of locking issues as #780 but at a broader scope. PR #780 fixed the case where the last partition's lock was held during ANALYZE. This PR addresses:
apply_constraints()Tested against production partition sets with ~50 monthly partitions and continuous DML traffic. Without
p_low_lock, constraint reapplication regularly caused lock contention and had to be manually aborted. Withp_low_lock, the same operation completed without impacting concurrent operations.Test plan
test-time-procedure-weekly-low-lock-part1.sql/part2.sqlverify constraints are created and validated viap_low_lockmodetest-time-procedure-weekly-part1/part2tests continue to pass (default behavior unchanged)p_force_not_validcorrectly overridesconstraint_validconfigMade with Cursor