Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

slow cleanup in device_lists_outbound_pokes #16480

Open
richvdh opened this issue Oct 12, 2023 · 1 comment
Open

slow cleanup in device_lists_outbound_pokes #16480

richvdh opened this issue Oct 12, 2023 · 1 comment
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-Performance Performance, both client-facing and admin-facing S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.

Comments

@richvdh
Copy link
Member

richvdh commented Oct 12, 2023

There is a cleanup job called prune_old_outbound_device_pokes (https://github.com/matrix-org/synapse/blob/v1.93.0/synapse/storage/databases/main/devices.py#L1297-L1388) which periodically attempts to clear out the device_lists_outbound_pokes table a bit.

The SELECT query can sometimes be very slow and IO-intensive. Sometimes it even takes longer than the 1 hour request timeout, meaning that it gets stuck without ever making any progress.

@erikjohnston suggests:

I do kinda want to change this, so we have two tables: the current table, and a duplicate which only holds the latest destination/user -> stream ID
@reivilibre reivilibre mentioned this issue Oct 13, 2023
Closed
@reivilibre
Copy link
Contributor

matrix.org currently has this patched out as the issue came back.

@reivilibre reivilibre added A-Performance Performance, both client-facing and admin-facing S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db labels Oct 13, 2023
reivilibre added a commit that referenced this issue Oct 16, 2023
@reivilibre
TEMPORARY Disable prune_old_outbound_device_pokes
96769a2 
See #16480
@matrixbot matrixbot mentioned this issue Dec 22, 2023
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db A-Performance Performance, both client-facing and admin-facing S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@richvdh @reivilibre