Faster joins: check purge behavior is ok when in partial state

[MatMaul]

Purging a room with the admin API when we are in partial state should be handled gracefully.

We could reject the purge, or just wait for the full state and then do it.

Second approach is however not restart friendly: if we restart during a resync, the resync process will be resume but the purge will be ignored (? since purge v2 is async, to be checked).

relevant #synapse-dev conversation: https://matrix.to/#/!vcyiEtMVHIhWXcJAfl:sw1v.org/$vCxad2MgGfIiZYJOdscIrR6kY6H0aGc4jzc0MWYzXk4?via=matrix.org&via=element.io&via=envs.net

[DMRobertson]

Testing this locally:

synapse_main | 2023-02-13 16:47:51,354 - synapse.storage.SQL - 416 - DEBUG - DELETE-12 - [SQL] {purge_room-9b} DELETE FROM events WHERE room_id=?                                                                                                                                         │
│synapse_main | 2023-02-13 16:47:51,354 - synapse.storage.SQL - 421 - DEBUG - DELETE-12 - [SQL values] {purge_room-9b} ('!0-LEQs3hgewbNYq2FICg:host.docker.internal:38553',)                                                                                                               │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.storage.SQL - 438 - DEBUG - DELETE-12 - [SQL FAIL] {purge_room-9b} FOREIGN KEY constraint failed                                                                                                                                         │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.storage.SQL - 442 - DEBUG - DELETE-12 - [SQL time] {purge_room-9b} 0.000161 sec                                                                                                                                                          │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.storage.txn - 788 - DEBUG - DELETE-12 - [TXN FAIL] {purge_room-9b} FOREIGN KEY constraint failed                                                                                                                                         │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.storage.txn - 796 - DEBUG - DELETE-12 - [TXN END] {purge_room-9b} 0.006896 sec                                                                                                                                                           │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.storage.txn - 692 - DEBUG - user_directory.notify_new_event-12 - [TXN START] {get_users_in_dir_due_to_room-9c}                                                                                                                           │
│synapse_main | 2023-02-13 16:47:51,355 - synapse.http.server - 123 - ERROR - DELETE-12 - Failed handle request via 'RoomRestServlet': <SynapseRequest at 0x7f35e04cc430 method='DELETE' uri='/_synapse/admin/v1/rooms/%210-LEQs3hgewbNYq2FICg:host.docker.internal:38553' clientproto='HTT│
│synapse_main | Traceback (most recent call last):                                                                                                                                                                                                                                         │
│synapse_main |   File "/editable-src/synapse/http/server.py", line 306, in _async_render_wrapper                                                                                                                                                                                          │
│synapse_main |     callback_return = await self._async_render(request)                                                                                                                                                                                                                    │
│synapse_main |   File "/editable-src/synapse/http/server.py", line 516, in _async_render                                                                                                                                                                                                  │
│synapse_main |     callback_return = await raw_callback_return                                                                                                                                                                                                                            │
│synapse_main |   File "/editable-src/synapse/rest/admin/rooms.py", line 308, in on_DELETE                                                                                                                                                                                                 │
│synapse_main |     return await self._delete_room(                                                                                                                                                                                                                                        │
│synapse_main |   File "/editable-src/synapse/rest/admin/rooms.py", line 365, in _delete_room                                                                                                                                                                                              │
│synapse_main |     await pagination_handler.purge_room(room_id, force=force_purge)                                                                                                                                                                                                        │
│synapse_main |   File "/editable-src/synapse/handlers/pagination.py", line 417, in purge_room                                                                                                                                                                                             │
│synapse_main |     await self._storage_controllers.purge_events.purge_room(room_id)                                                                                                                                                                                                       │
│synapse_main |   File "/editable-src/synapse/storage/controllers/purge_events.py", line 36, in purge_room                                                                                                                                                                                 │
│synapse_main |     state_groups_to_delete = await self.stores.main.purge_room(room_id)                                                                                                                                                                                                    │
│synapse_main |   File "/editable-src/synapse/storage/databases/main/purge_events.py", line 328, in purge_room                                                                                                                                                                             │
│synapse_main |     state_groups_to_delete = await self.db_pool.runInteraction(                                                                                                                                                                                                            │
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 883, in runInteraction                                                                                                                                                                                            │
│synapse_main |     return await delay_cancellation(_runInteraction())                                                                                                                                                                                                                     │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks                                                                                                                                                                  │
│synapse_main |     result = context.run(                                                                                                                                                                                                                                                  │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator                                                                                                                                                        │
│synapse_main |     return g.throw(self.type, self.value, self.tb)                                                                                                                                                                                                                         │
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 850, in _runInteraction                                                                                                                                                                                           │
│synapse_main |     result = await self.runWithConnection(                                                                                                                                                                                                                                 │
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 978, in runWithConnection                                                                                                                                                                                         │
│synapse_main |     return await make_deferred_yieldable(                                                                                                                                                                                                                                  │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext                                                                                                                                                                       │
│synapse_main |     result = inContext.theWork()  # type: ignore[attr-defined]                                                                                                                                                                                                             │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>                                                                                                                                                                        │
│synapse_main |     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]                                                                                                                                                                                                │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext                                                                                                                                                                    │
│synapse_main |     return self.currentContext().callWithContext(ctx, func, *args, **kw)                                                                                                                                                                                                   │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext                                                                                                                                                                     │
│synapse_main |     return func(*args, **kw)                                                                                                                                                                                                                                               │
│synapse_main |   File "/usr/local/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection                                                                                                                                                              ┤
│synapse_main |     result = func(conn, *args, **kw)                                                                                                                                                                                                                                       ┤
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 971, in inner_func                                                                                                                                                                                                │
│synapse_main |     return func(db_conn, *args, **kwargs)                                                                                                                                                                                                                                  │
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 712, in new_transaction                                                                                                                                                                                           ┤
│synapse_main |     r = func(cursor, *args, **kwargs)                                                                                                                                                                                                                                      ┤
│synapse_main |   File "/editable-src/synapse/storage/databases/main/purge_events.py", line 456, in _purge_room_txn                                                                                                                                                                        ┤
│synapse_main |     txn.execute("DELETE FROM %s WHERE room_id=?" % (table,), (room_id,))                                                                                                                                                                                                   ┤
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 388, in execute                                                                                                                                                                                                   ┤
│synapse_main |     self._do_execute(self.txn.execute, sql, *args)                                                                                                                                                                                                                         ┤
│synapse_main |   File "/editable-src/synapse/storage/database.py", line 436, in _do_execute                                                                                                                                                                                               ┤
│synapse_main |     return func(sql, *args, **kwargs)                                                                                                                                                                                                                                      │
│synapse_main | sqlite3.IntegrityError: FOREIGN KEY constraint failed                                                                                                                                                                                                                      │

Hooray for constraints.

We'll need to delete from the partial state tables.
We should probably accepting new partial state events in that room, to avoid a race.

[DMRobertson]

I think we consider this closed by #15069.