Federated room cannot be joined again when left and deleted before

[waclaw66]

Description

When a federated room (e.g. #synapse:matrix.org) is joined from self hosted federated homeserver, then left, deleted via API from self homeserver, then rejoin fails.

Steps to reproduce

1. Join some room (e.g. #synapse:matrix.org) from self hosted homeserver (same version of Synapse 1.27.0).
2. Leave that room
3. Delete that room using API /_synapse/admin/v1/rooms/[room_id]/delete
4. Try to join that room again
5. It fails with following error...

synapse.http.server: [POST-10040] Failed handle request via 'JoinRoomAliasServlet': <XForwardedForRequest at 0x7f57646fa970 method='POST' uri='/_matrix/client/r0/join/%23synapse%3Amatrix.org' clientproto='HTTP/1.1' site='8008'>
    Traceback (most recent call last):
      File "/usr/lib/python3.9/site-packages/synapse/http/server.py", line 252, in _async_render_wrapper
        callback_return = await self._async_render(request)
      File "/usr/lib/python3.9/site-packages/synapse/http/server.py", line 430, in _async_render
        callback_return = await raw_callback_return
      File "/usr/lib/python3.9/site-packages/synapse/rest/client/v1/room.py", line 301, in on_POST
        await self.room_member_handler.update_membership(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 333, in update_membership
        result = await self.update_membership_locked(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 549, in update_membership_locked
        remote_join_response = await self._remote_join(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/room_member.py", line 1091, in _remote_join
        event_id, stream_id = await self.federation_handler.do_invite_join(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 1400, in do_invite_join
        max_stream_id = await self._persist_auth_tree(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 2050, in _persist_auth_tree
        await self.persist_events_and_notify(
      File "/usr/lib/python3.9/site-packages/synapse/handlers/federation.py", line 2925, in persist_events_and_notify
        events, max_stream_token = await self.storage.persistence.persist_events(
      File "/usr/lib/python3.9/site-packages/synapse/storage/persist_events.py", line 262, in persist_events
        ret_vals = await make_deferred_yieldable(
    twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'psycopg2.errors.UniqueViolation'>: duplicate key value violates unique constraint "event_auth_chains_pkey"
    DETAIL:  Key (event_id)=($e9U026auDHIgaZPAqlblvPupACjl7jcZDblP970dJPs) already exists.

    /usr/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py:208:run
    --- <exception caught here> ---
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:172:handle_queue_loop
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:324:persisting_queue
    /usr/lib/python3.9/site-packages/synapse/storage/persist_events.py:532:_persist_events
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:171:_persist_events_and_state_updates
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:661:runInteraction
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:744:runWithConnection
    /usr/lib64/python3.9/site-packages/twisted/python/threadpool.py:250:inContext
    /usr/lib64/python3.9/site-packages/twisted/python/threadpool.py:266:<lambda>
    /usr/lib64/python3.9/site-packages/twisted/python/context.py:122:callWithContext
    /usr/lib64/python3.9/site-packages/twisted/python/context.py:85:callWithContext
    /usr/lib64/python3.9/site-packages/twisted/enterprise/adbapi.py:306:_runWithConnection
    /usr/lib64/python3.9/site-packages/twisted/python/compat.py:464:reraise
    /usr/lib64/python3.9/site-packages/twisted/enterprise/adbapi.py:297:_runWithConnection
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:739:inner_func
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:539:new_transaction
    /usr/lib/python3.9/site-packages/synapse/logging/utils.py:71:wrapped
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:379:_persist_events_txn
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:472:_persist_event_auth_chain_txn
    /usr/lib/python3.9/site-packages/synapse/storage/databases/main/events.py:630:_add_chain_cover_index
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:896:simple_insert_many_txn
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:274:execute_batch
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:319:_do_execute
    /usr/lib/python3.9/site-packages/synapse/storage/database.py:274:<lambda>
    /usr/lib64/python3.9/site-packages/psycopg2/extras.py:1209:execute_batch
    ]]

The room should be rejoined without any problem.

Version information

• Homeserver: matrix.bolesiny.net

• Version: 1.27.0

• Install method: Fedora rpm package

• Platform: Fedora 33

[waclaw66]

Related #9418

[clokep]

@waclaw66 This seems to be a duplicate of #9418. I'm going to close it unless you feel there's a reason that it is not.

[waclaw66]

Maybe it's the same bug causing the problem, but those are different cases, which could lead to it.
You can close it in favour of #9418, there is a reference, in case the fix wouldn't work for my problem I can reopen it.

@waclaw66 This seems to be a duplicate of #9418. I'm going to close it unless you feel there's a reason that it is not.

[clokep]

Ah, I see that this is due to leaving while the other is due to a failure to join. 👍 Hopefully the fix is the same!

[waclaw66]

Is there any hotfix? Leaving and joining the room again works properly, but when a room is deleted via API and joined again it crashes as shown. So there supposed to be some remain of that deleted room in database which causes that crash after joining it again.

[clokep]

When you deleted the room you also purged it I'm guessing?

I suspect that purge_room_state needs to be updated to remove from event_auth_chains also?

[waclaw66]

Right, I purged the history, then I left and finally deleted it.

When you deleted the room you also purged it I'm guessing?

I suspect that purge_room_state needs to be updated to remove from event_auth_chains also?

[waclaw66]

That's strange
select * from state_groups_state where room_id='!ehXvUhWNASUkSLvAGP:matrix.org'
gives me more than 300k records, but
POST http://localhost:8008/_synapse/admin/v1/purge_history/!ehXvUhWNASUkSLvAGP:matrix.org
with data
{ "delete_local_events": true, "purge_up_to_ts": 1614270664000 }
returns
{"errcode":"M_NOT_FOUND","error":"there is no event to be purged"}

[clokep]

purge_history is different than purging the room (it purges the events before a certain time). I was asking whether you gave {"purge": true} when calling /_synapse/admin/v1/rooms/[room_id]/delete.

[waclaw66]

Understood, I used synapse-admin to delete the room finally.
It seems, it doesn't purge...
https://github.com/Awesome-Technologies/synapse-admin/blob/684c44e47057e42fb217688e08706e9f2e6767a7/src/synapse/dataProvider.js#L69

purge_history is different than purging the room (it purges the events before a certain time). I was asking whether you gave {"purge": true} when calling /_synapse/admin/v1/rooms/[room_id]/delete.