KeyError on recent background updates

[Fizzadar]

Description

We're seeing a KeyError on the recent updates introduced by 11e15d7:

KeyError: 'current_state_events_stream_ordering_idx'
  File "synapse/storage/background_updates.py", line 405, in run_background_updates
    result = await self.do_next_background_update(sleep)
  File "synapse/storage/background_updates.py", line 528, in do_next_background_update
    update_info = self._background_update_handlers[self._current_background_update]

Specifically appears to be missing in BackgroundUpdates._background_update_handlers, but adding some logging indicates it is being registered as a background update:

{"message":"Registering background index update: current_state_events_stream_ordering_idx","namespace":"synapse.storage.background_updates","level":"info","time":1690998470.67,"request":"main","server_name":"beeper.com"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.9,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.97,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}
{"message":"Error doing update","namespace":"synapse.storage.background_updates","level":"error","time":1690998470.98,"request":"background_updates-0","server_name":"beeper.com","exc_type":"KeyError","exc_value":"'current_state_events_stream_ordering_idx'"}

If I run locally with sqlite there is no error, but I can't see any codepath that would change the behaviour for postgres 🤔

[Fizzadar]

Just checked - applies to all three of those updates only - the following update, event_forward_extremities_event_id_foreign_key_constraint_update, ran just fine after I manually created the indices and removed the update from the table.

[clokep]

Are you running the script to execute schema deltas while having a Synapse running? If the running Synapse doesn't yet have the new code which registers the background updates then you can hit this.

[Fizzadar]

No this is running v1.88, this is on the main process running a background job so the schema changes are applied already. Stumped on how this could be a KeyError given the logging indicating it registering as expected..

[reivilibre]

Are you running a split main and state database by any chance? Just a hunch, but could be that the background update is registered on the wrong database — there would be two different background updaters in that case.