Deleting events causes database corruption

[richvdh]

Deleting events - whether you do it via the purge history admin API, the delete room admin API, or by enabling retention on your server - has potential to corrupt the database.

In short, the delete operations can race against other operations (notably event persistence), leading the database to get into an inconsistent state. The main reason is that the delete operations do not invalidate the in-memory caches, but in general there are a bunch of race conditions.

Examples include:

• Error backfilling in room for 10 minutes after purging history #9864
• completely broken room after purge_room/rejoin #11521
• non-existent event-forward-extremities (Forward extremity event missing from database, leading to No state group for unknown or outlier event #12507 (comment)), or outlier forward extremities ("No state group for unknown or outlier event" error in federation #13026). Also ERROR - POST-758856 - Failed handle request via 'RoomMembershipRestServlet' – when trying to leave room #13524.

[richvdh]

Other related issues:

• POST /read_markers fails after history purge if cache is cleared #11738
• Push notifications stop working when a history purge deletes the event a read receipt is pointing at #12880 (though I don't think that's database corruption, it's just stuff not working)

[dklimpel]

Is it a solution to delete the entire cache before deleting? And would that make sense?

[dklimpel]

Is this related to?

• state_groups (& state_groups_state & state_group_edges) are not fully purged alongside the rooms #12821
• My HS has accumulated thousands of unreferenced state groups #3364

[richvdh]

Is it a solution to delete the entire cache before deleting? And would that make sense?

Sadly that's not sufficient, since the cache might be immediately repopulated before the delete actually happens.

Is this related to?

I wouldn't say so. Those are the history purge deleting less stuff than they should; they do not cause inconsistency between Synapse and the database and cannot cause incorrect data in the database.

[richvdh]

the delete operations can race against other operations (notably event persistence)

Incidentally: I think purging a complete room with no local members should be safe from races, but there are still cache coherency bugs (#11521).

[rettichschnidi]

I am a homeserver admin affected by this - database is ever growing and deleting 90+ days old messages is a requirement (privacy aware NGO). Anything I can do to help getting this issue resolved?