Message retention policies at the room and server levels #5815

babolivier · 2019-08-02T10:00:24Z

Implements most of MSC1763, except for the management of min_period in a room's retention policy.

This is not a generic implementation, because it relies on server admins setting boundaries for what users can provide as values in a room's retention policy, and considers the rooms lacking a policy as following the server's. Changing that should be fairly trivial, though, and without consequences.

Note to the reviewer: Sorry this PR is a bit large, every commit should be reviewable independently.

codecov · 2019-08-02T11:02:37Z

Codecov Report

Merging #5815 into dinsic will decrease coverage by 18.46%.
The diff coverage is 20.43%.

@@             Coverage Diff             @@
##           dinsic    #5815       +/-   ##
===========================================
- Coverage    63.6%   45.13%   -18.47%     
===========================================
  Files         332      332               
  Lines       36459    36543       +84     
  Branches     6018     6035       +17     
===========================================
- Hits        23191    16495     -6696     
- Misses      11635    19034     +7399     
+ Partials     1633     1014      -619

codecov · 2019-08-02T11:02:37Z

Codecov Report

❗ No coverage uploaded for pull request base (dinsic@62f5e3b). Click here to learn what that means.
The diff coverage is 61.65%.

@@            Coverage Diff            @@
##             dinsic    #5815   +/-   ##
=========================================
  Coverage          ?   63.74%           
=========================================
  Files             ?      332           
  Lines             ?    36691           
  Branches          ?     6085           
=========================================
  Hits              ?    23388           
  Misses            ?    11648           
  Partials          ?     1655

babolivier · 2019-08-16T17:52:58Z

FTR I haven't got the time to include tests yet, and am planning on writing some on Monday. In the mean time, what's already here should be functional and reviewable.

richvdh

I haven't dug right into the the code, but I'm a bit confused about what the settings are meant to do and how they are meant to interact. I think maybe we need to reconsider how this works?

synapse/config/server.py

synapse/handlers/pagination.py

richvdh · 2019-08-19T15:19:17Z

Also: Please make sure that you test that backfill behaves sensibly once a purge has happened

stale

synapse/config/server.py

synapse/storage/room.py

stale

babolivier · 2019-08-27T16:30:56Z

For the record, after some discussion with @erikjohnston, we figured it would be better and less risky to hide expired events from the clients instead of filtering out at the federation level, and to just let the expired events get deleted by the purge job(s). @giomfo approved this change on behalf of DINSIC.

The current state of this PR should reflect this decision.

richvdh

This is looking much better! A few things, but I think they're all pretty small.

synapse/config/server.py

synapse/federation/federation_server.py

synapse/handlers/federation.py

synapse/handlers/pagination.py

synapse/storage/room.py

synapse/storage/schema/delta/55/room_retention.sql

synapse/visibility.py

babolivier · 2019-08-28T14:31:24Z

In e200257 I added a new event_id column to the retention table because @erikjohnston pointed out that _update_retention_policy_for_room_txn would also be called on events received by backfilling, therefore there was a risk for an old event to overwrite more recent data and to end up with a room_retention table that doesn't represent the current state of rooms.

With this new column, we can do an INNER JOIN with the current_state_events table to figure out which policy is the right one to apply.

stale

synapse/storage/room.py

synapse/storage/schema/delta/55/room_retention.sql

synapse/handlers/pagination.py

stale

richvdh

🚢

Synapse 1.7.0rc1 (2019-12-09) ============================= Features -------- - Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436)) - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858)) - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119)) - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176)) - Configure privacy-preserving settings by default for the room directory. ([\#6354](#6354)) - Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409)) - Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434)) Bugfixes -------- - Transfer non-standard power levels on room upgrade. ([\#6237](#6237)) - Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241)) - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329)) - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332)) - Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333)) - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406)) - Fix an intermittent exception when handling read-receipts. ([\#6408](#6408)) - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420)) - Fix startup error when http proxy is defined. ([\#6421](#6421)) - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449)) - Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451)) - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462)) - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470)) - Improve sanity-checking when receiving events over federation. ([\#6472](#6472)) - Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491)) - Fix small performance regression for sending invites. ([\#6493](#6493)) - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494)) Improved Documentation ---------------------- - Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490)) - Fix link in the user directory documentation. ([\#6388](#6388)) - Add build instructions to the docker readme. ([\#6390](#6390)) - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443)) - Write some docs for the quarantine_media api. ([\#6458](#6458)) - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461)) Deprecations and Removals ------------------------- - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488)) Internal Changes ---------------- - Add benchmarks for structured logging and improve output performance. ([\#6266](#6266)) - Improve the performance of outputting structured logging. ([\#6322](#6322)) - Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480)) - Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362)) - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379)) - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392)) - Clarifications for the email configuration settings. ([\#6423](#6423)) - Add more tests to the blacklist when running in worker mode. ([\#6429](#6429)) - Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487)) - Port synapse.rest.client.v1 to async/await. ([\#6482](#6482)) - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483)) - Port SyncHandler to async/await. ([\#6484](#6484))

Synapse 1.7.0 (2019-12-13) ========================== This release changes the default settings so that only local authenticated users can query the server's room directory. See the [upgrade notes](UPGRADE.rst#upgrading-to-v170) for details. Support for SQLite versions before 3.11 is now deprecated. A future release will refuse to start if used with an SQLite version before 3.11. Administrators are reminded that SQLite should not be used for production instances. Instructions for migrating to Postgres are available [here](docs/postgres.md). A future release of synapse will, by default, disable federation for servers using SQLite. No significant changes since 1.7.0rc2. Synapse 1.7.0rc2 (2019-12-11) ============================= Bugfixes -------- - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](#6497)) - Fix support for SQLite 3.7. ([\#6499](#6499)) - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](#6507), [\#6509](#6509)) Synapse 1.7.0rc1 (2019-12-09) ============================= Features -------- - Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436)) - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858)) - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119)) - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176)) - Configure privacy-preserving settings by default for the room directory. ([\#6355](#6355)) - Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409)) - Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434)) Bugfixes -------- - Transfer non-standard power levels on room upgrade. ([\#6237](#6237)) - Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241)) - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329)) - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332)) - Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333)) - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406)) - Fix an intermittent exception when handling read-receipts. ([\#6408](#6408)) - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420)) - Fix startup error when http proxy is defined. ([\#6421](#6421)) - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449)) - Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451)) - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462)) - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470)) - Improve sanity-checking when receiving events over federation. ([\#6472](#6472)) - Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491)) - Fix small performance regression for sending invites. ([\#6493](#6493)) - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494)) Improved Documentation ---------------------- - Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490)) - Fix link in the user directory documentation. ([\#6388](#6388)) - Add build instructions to the docker readme. ([\#6390](#6390)) - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443)) - Write some docs for the quarantine_media api. ([\#6458](#6458)) - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461)) Deprecations and Removals ------------------------- - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488)) Internal Changes ---------------- - Add benchmarks for structured logging and improve output performance. ([\#6266](#6266)) - Improve the performance of outputting structured logging. ([\#6322](#6322)) - Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480)) - Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362)) - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379)) - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392)) - Clarifications for the email configuration settings. ([\#6423](#6423)) - Add more tests to the blacklist when running in worker mode. ([\#6429](#6429)) - Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487)) - Port synapse.rest.client.v1 to async/await. ([\#6482](#6482)) - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483)) - Port SyncHandler to async/await. ([\#6484](#6484))

babolivier added the dinsic label Aug 2, 2019

babolivier force-pushed the babolivier/dinsic-message-retention branch from 14a83fa to 8ef235c Compare August 2, 2019 12:19

ara4n mentioned this pull request Aug 10, 2019

MSC1763: Proposal for specifying configurable message retention periods matrix-org/matrix-spec-proposals#1763

Open

anoadragon453 mentioned this pull request Aug 12, 2019

We need to actually censor redactions from the DB (SYN-284) #1287

Closed

babolivier marked this pull request as ready for review August 16, 2019 17:53

babolivier requested a review from a team August 16, 2019 17:53

richvdh previously requested changes Aug 19, 2019

View reviewed changes

babolivier requested a review from a team August 21, 2019 10:50

richvdh previously requested changes Aug 21, 2019

View reviewed changes

aaronraimist mentioned this pull request Aug 27, 2019

Add option to prune history after a certain time element-hq/element-web#3104

Closed

babolivier requested a review from a team August 27, 2019 16:31

richvdh previously requested changes Aug 28, 2019

View reviewed changes

babolivier requested a review from a team August 28, 2019 15:21

richvdh previously requested changes Aug 28, 2019

View reviewed changes

synapse/storage/room.py Outdated Show resolved Hide resolved

synapse/storage/room.py Outdated Show resolved Hide resolved

synapse/storage/schema/delta/55/room_retention.sql Show resolved Hide resolved

synapse/handlers/pagination.py Outdated Show resolved Hide resolved

babolivier requested a review from a team August 28, 2019 16:20

richvdh approved these changes Aug 28, 2019

View reviewed changes

babolivier force-pushed the babolivier/dinsic-message-retention branch 2 times, most recently from 077609b to c8a113a Compare August 28, 2019 17:14

babolivier added 2 commits August 28, 2019 18:16

Implement per-room message retention policies

7071459

Merge branch 'dinsic' into babolivier/dinsic-message-retention

e5df12a

babolivier force-pushed the babolivier/dinsic-message-retention branch from c8a113a to e5df12a Compare August 28, 2019 17:17

babolivier merged commit 99eec6d into dinsic Aug 28, 2019

babolivier mentioned this pull request Nov 12, 2019

Implement message retention policies (MSC1763) #6358

Merged

babolivier deleted the babolivier/dinsic-message-retention branch January 9, 2020 15:46

andrevmatos mentioned this pull request Jan 11, 2020

Investigate if matrix retention policy could replace purger script raiden-network/raiden-service-bundle#80

Closed

lrq3000 mentioned this pull request Feb 15, 2020

RTC/Riot: warn about media and centralization on matrix.org? privacytools/privacytools.io#1395

Closed

ara4n mentioned this pull request Mar 1, 2022

We need a way to be able to expire data from a HS. (SPEC-141) matrix-org/matrix-spec#35

Open

anoadragon453 mentioned this pull request May 9, 2022

redactions are not relayed to clients if the original event has been deleted (eg via message retention) #12603

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Message retention policies at the room and server levels #5815

Message retention policies at the room and server levels #5815

babolivier commented Aug 2, 2019 •

edited

Loading

codecov bot commented Aug 2, 2019

codecov bot commented Aug 2, 2019 •

edited

Loading

babolivier commented Aug 16, 2019

richvdh left a comment

richvdh commented Aug 19, 2019

babolivier commented Aug 27, 2019 •

edited

Loading

richvdh left a comment

babolivier commented Aug 28, 2019

richvdh left a comment

Message retention policies at the room and server levels #5815

Message retention policies at the room and server levels #5815

Conversation

babolivier commented Aug 2, 2019 • edited Loading

codecov bot commented Aug 2, 2019

Codecov Report

codecov bot commented Aug 2, 2019 • edited Loading

Codecov Report

babolivier commented Aug 16, 2019

richvdh left a comment

Choose a reason for hiding this comment

richvdh commented Aug 19, 2019

babolivier commented Aug 27, 2019 • edited Loading

richvdh left a comment

Choose a reason for hiding this comment

babolivier commented Aug 28, 2019

richvdh left a comment

Choose a reason for hiding this comment

babolivier commented Aug 2, 2019 •

edited

Loading

codecov bot commented Aug 2, 2019 •

edited

Loading

babolivier commented Aug 27, 2019 •

edited

Loading