Drop IS NULL indices *after* the ALTER TABLE statement.

[reivilibre]

We made the assumption that the IS NULL index would help validate the addition NOT NULL constraint, but we didn't even give ourselves a chance because we dropped it before adding the NOT NULL constraint.

Follows: #15350

Base: develop

Original commit schedule, with full messages:

1. Move DROP INDEX to after ALTER .. SET NOT NULL

[squahtx]

Looks good, thanks for fixing it!

[reivilibre]

I'm afraid I have bad news, the presence of an index doesn't alter the timings of ALTER COLUMN .. SET NOT NULL. The docs don't mention it as a valid technique to improve performance and my testing shows the addition of the constraint takes the same amount of time without regard for the presence of the index, definitely not negligible.

The way forward is probably to add a NOT VALID constraint, which is enforced immediately for new rows, and then do a VALIDATE CONSTRAINT in the background.

Scanning a large table to verify a new foreign key or check constraint can take a long time, and other updates to the table are locked out until the ALTER TABLE ADD CONSTRAINT command is committed. The main purpose of the NOT VALID constraint option is to reduce the impact of adding a constraint on concurrent updates. With NOT VALID, the ADD CONSTRAINT command does not scan the table and can be committed immediately. After that, a VALIDATE CONSTRAINT command can be issued to verify that existing rows satisfy the constraint. The validation step does not need to lock out concurrent updates, since it knows that other transactions will be enforcing the constraint for rows that they insert or update; only pre-existing rows need to be checked. Hence, validation acquires only a SHARE UPDATE EXCLUSIVE lock on the table being altered. (If the constraint is a foreign key then a ROW SHARE lock is also required on the table referenced by the constraint.) In addition to improving concurrency, it can be useful to use NOT VALID and VALIDATE CONSTRAINT in cases where the table is known to contain pre-existing violations. Once the constraint is in place, no new violations can be inserted, and the existing problems can be corrected at leisure until VALIDATE CONSTRAINT finally succeeds.

— https://www.postgresql.org/docs/15/sql-altertable.html