History purge failed with: ValueError: max() arg is an empty sequence

[Kokokokoka]

Description

On history purge, this happened:

2021-08-20 21:25:22,304 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626834204073 => stream_or
dering 6232155)
2021-08-20 23:17:30,415 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5673 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:17:30,443 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050432 => stream_or
dering 6232211)
2021-08-20 23:17:30,453 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841050443 => stream_or
dering 6232211)
2021-08-20 23:19:03,094 - synapse.handlers.pagination - 286 - ERROR - _purge_history-5691 - [purge] failed
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/handlers/pagination.py", line 279, in _purge_history
    await self.storage.purge_events.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/purge_events.py", line 54, in purge_history
    state_groups = await self.stores.main.purge_history(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 49, in purge_history
    return await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/purge_events.py", line 105, in _purge_history_txn
    max_depth = max(row[1] for row in rows)
ValueError: max() arg is an empty sequence
2021-08-20 23:19:15,824 - synapse.handlers.pagination - 203 - WARNING - purge_history_for_rooms_in_range-0 - [purge] purging events not possible: No event found (ts 1626841155788 => stream_or
dering 6232211)

Then there's no connection with element (crash?)

Version information

Synapse: 1.40,
on Debian 11,
using matrix repo.
This is a migration from 1.0 version

[erikjohnston]

I think the fix here is to handle the case of the DB query returning no rows.

[anoadragon453]

What caused the purge out of interest? Do you have the experimental room retention feature enabled, or was the room purged manually?

[Kokokokoka]

I answered in a PR reply, sorry, I'll copy it here:
I would love to find out the cause of this issue. The problem is, that we migrated from 1.0->1.41. Rooms, messages, files got cleared by some mystique scripts before (maybe this is the cause of the bug, i'm trying to find this out in the db).
Now there is message retention policy, it helps a lot. I don't know if there's a policy for empty rooms/rooms with only deleted accounts which weren't accessed for x. I had to manually implement local files|thumbnails cleaner (which checks what elems have last access ts > x, and then purge the elems which aren't in the db.

For instance, how could you have a room graph that doesn't have any leaves?

Can I see this in the db somehow?
(For now I did:
empty rooms:
select count(*) from rooms where not exists(select from room_memberships where room_memberships.room_id = rooms.room_id); to find empty rooms (26)
and:
rooms that become empty if deactivated|ppl with activity > x ago are substracted:

select count(*) from rooms where room_id not in(select room_id from room_memberships where
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1));
count
-------
 1421
(1 row))

rooms with 1 active:

select count(*) from (select room_id, count(user_id) cn from room_memberships where 
user_id not in (select name from users where name in (select user_id from (select  
user_id, max(timestamp) ts from user_daily_visits group by user_id) x where ts < 
1627215050807) or deactivated = 1) group by room_id)x where cn =1;
count
796
(1 row)

So, I don't really know whether this room was purged manually, all I know that there was some script that managed local_media+should have managed messages and we had syn v1.0

[richvdh]

There are a few known reasons why we could end up with a room without any forward-extremities: see #8094 which is another symptom of such a situation. Some examples include:

• There was a background update introduced in Synapse 1.10 (Backgroud update to clean out rooms from current state #6802) which removes the forward-extremities for any room where there are no active members.
• Ill-advised database surgery, including running various scripts that can be found on the internet.

[callahad]

Now that #10690 is merged, can we close this?

[Kokokokoka]

I think so, if there's no objections, i'll gladly do this.