-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Store arbitrary relations from events #11391
Conversation
I'd be curious if @richvdh or @erikjohnston have thoughts on how we can improve this query to make it either faster or less expensive! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
generally seems sound to me, modulo a few comments.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
@@ -811,3 +860,65 @@ def _create_user(self, localpart: str) -> Tuple[str, str]: | |||
access_token = self.login(localpart, "abc123") | |||
|
|||
return user_id, access_token | |||
|
|||
def test_background_update(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Putting this in this file seems dirty, but it seems best to keep relations tests together? Maybe it would be better to start a storage test file and we can add to it in the future when fleshing out some unit tests?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's best here.
for event_id, parent_id in missing_thread_relations | ||
], | ||
) | ||
# Iterate the parent IDs and invalidate caches. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if we usually clear caches as part of background updates, but in this case I think it could be a bit broken without doing it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems sensible
@richvdh Thanks for the review! I think I've handled your comments -- I've left a couple of notes of places I was unsure about. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm other than a couple of nits
for event_id, parent_id in missing_thread_relations | ||
], | ||
) | ||
# Iterate the parent IDs and invalidate caches. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems sensible
@@ -811,3 +860,65 @@ def _create_user(self, localpart: str) -> Tuple[str, str]: | |||
access_token = self.login(localpart, "abc123") | |||
|
|||
return user_id, access_token | |||
|
|||
def test_background_update(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's best here.
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Synapse 1.48.0 (2021-11-30) =========================== This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag. This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0. No significant changes since 1.48.0rc1. Synapse 1.48.0rc1 (2021-11-25) ============================== Features -------- - Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11161](matrix-org/synapse#11161)) - Support filtering by relation senders & types per [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11236](matrix-org/synapse#11236)) - Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\#11318](matrix-org/synapse#11318), [\#11371](matrix-org/synapse#11371)) - Support the stable version of [MSC2778](matrix-org/matrix-spec-proposals#2778): the `m.login.application_service` login type. Contributed by @tulir. ([\#11335](matrix-org/synapse#11335)) - Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\#11223](matrix-org/synapse#11223)) - Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\#11228](matrix-org/synapse#11228)) - Add an admin API to un-shadow-ban a user. ([\#11347](matrix-org/synapse#11347)) - Add an admin API to run background database schema updates. ([\#11352](matrix-org/synapse#11352)) - Add an admin API for blocking a room. ([\#11324](matrix-org/synapse#11324)) - Update the JWT login type to support custom a `sub` claim. ([\#11361](matrix-org/synapse#11361)) - Store and allow querying of arbitrary event relations. ([\#11391](matrix-org/synapse#11391)) Bugfixes -------- - Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error when stored in the DB. ([\#11230](matrix-org/synapse#11230)) - Prevent [MSC2716](matrix-org/matrix-spec-proposals#2716) historical state events from being pushed to an application service via `/transactions`. ([\#11265](matrix-org/synapse#11265)) - Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix. ([\#11288](matrix-org/synapse#11288)) - Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module interface were not affected. ([\#11340](matrix-org/synapse#11340)) - Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time. ([\#11355](matrix-org/synapse#11355)) - Fix a bug introduced in 1.45.0 where the `read_templates` method of the module API would error. ([\#11377](matrix-org/synapse#11377)) - Fix an issue introduced in 1.47.0 which prevented servers re-joining rooms they had previously left, if their signing keys were replaced. ([\#11379](matrix-org/synapse#11379)) - Fix a bug introduced in 1.13.0 where creating and publishing a room could cause errors if `room_list_publication_rules` is configured. ([\#11392](matrix-org/synapse#11392)) - Improve performance of various background database updates. ([\#11421](matrix-org/synapse#11421), [\#11422](matrix-org/synapse#11422)) Improved Documentation ---------------------- - Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's. ([\#11281](matrix-org/synapse#11281)) - Fix typos in the documentation for the `username_available` admin API. Contributed by Stanislav Motylkov. ([\#11286](matrix-org/synapse#11286)) - Add Single Sign-On, SAML and CAS pages to the documentation. ([\#11298](matrix-org/synapse#11298)) - Change the word 'Home server' as one word 'homeserver' in documentation. ([\#11320](matrix-org/synapse#11320)) - Fix missing quotes for wildcard domains in `federation_certificate_verification_whitelist`. ([\#11381](matrix-org/synapse#11381)) Deprecations and Removals ------------------------- - Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\#11333](matrix-org/synapse#11333), [\#11395](matrix-org/synapse#11395)) Internal Changes ---------------- - Add type annotations to `synapse.metrics`. ([\#10847](matrix-org/synapse#10847)) - Split out federated PDU retrieval function into a non-cached version. ([\#11242](matrix-org/synapse#11242)) - Clean up code relating to to-device messages and sending ephemeral events to application services. ([\#11247](matrix-org/synapse#11247)) - Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`. ([\#11278](matrix-org/synapse#11278)) - Drop unused database tables `room_stats_historical` and `user_stats_historical`. ([\#11280](matrix-org/synapse#11280)) - Require all files in synapse/ and tests/ to pass mypy unless specifically excluded. ([\#11282](matrix-org/synapse#11282), [\#11285](matrix-org/synapse#11285), [\#11359](matrix-org/synapse#11359)) - Add missing type hints to `synapse.app`. ([\#11287](matrix-org/synapse#11287)) - Remove unused parameters on `FederationEventHandler._check_event_auth`. ([\#11292](matrix-org/synapse#11292)) - Add type hints to `synapse._scripts`. ([\#11297](matrix-org/synapse#11297)) - Fix an issue which prevented the `remove_deleted_devices_from_device_inbox` background database schema update from running when updating from a recent Synapse version. ([\#11303](matrix-org/synapse#11303)) - Add type hints to storage classes. ([\#11307](matrix-org/synapse#11307), [\#11310](matrix-org/synapse#11310), [\#11311](matrix-org/synapse#11311), [\#11312](matrix-org/synapse#11312), [\#11313](matrix-org/synapse#11313), [\#11314](matrix-org/synapse#11314), [\#11316](matrix-org/synapse#11316), [\#11322](matrix-org/synapse#11322), [\#11332](matrix-org/synapse#11332), [\#11339](matrix-org/synapse#11339), [\#11342](matrix-org/synapse#11342)) - Add type hints to `synapse.util`. ([\#11321](matrix-org/synapse#11321), [\#11328](matrix-org/synapse#11328)) - Improve type annotations in Synapse's test suite. ([\#11323](matrix-org/synapse#11323), [\#11330](matrix-org/synapse#11330)) - Test that room alias deletion works as intended. ([\#11327](matrix-org/synapse#11327)) - Add type annotations for some methods and properties in the module API. ([\#11341](matrix-org/synapse#11341)) - Fix running `scripts-dev/complement.sh`, which was broken in v1.47.0rc1. ([\#11368](matrix-org/synapse#11368)) - Rename internal functions for token generation to better reflect what they do. ([\#11369](matrix-org/synapse#11369), [\#11370](matrix-org/synapse#11370)) - Add type hints to configuration classes. ([\#11377](matrix-org/synapse#11377)) - Publish a `develop` image to Docker Hub. ([\#11380](matrix-org/synapse#11380)) - Keep fallback key marked as used if it's re-uploaded. ([\#11382](matrix-org/synapse#11382)) - Use `auto_attribs` on the `attrs` class `RefreshTokenLookupResult`. ([\#11386](matrix-org/synapse#11386)) - Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. ([\#11388](matrix-org/synapse#11388)) - Do not run the broken MSC2716 tests when running `scripts-dev/complement.sh`. ([\#11389](matrix-org/synapse#11389)) - Remove dead code from supporting ACME. ([\#11393](matrix-org/synapse#11393)) - Refactor including the bundled relations when serializing an event. ([\#11408](matrix-org/synapse#11408))
Synapse 1.48.0 (2021-11-30) =========================== This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag. This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0. Features -------- - Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11161](matrix-org/synapse#11161)) - Support filtering by relation senders & types per [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11236](matrix-org/synapse#11236)) - Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\#11318](matrix-org/synapse#11318), [\#11371](matrix-org/synapse#11371)) - Support the stable version of [MSC2778](matrix-org/matrix-spec-proposals#2778): the `m.login.application_service` login type. Contributed by @tulir. ([\#11335](matrix-org/synapse#11335)) - Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\#11223](matrix-org/synapse#11223)) - Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\#11228](matrix-org/synapse#11228)) - Add an admin API to un-shadow-ban a user. ([\#11347](matrix-org/synapse#11347)) - Add an admin API to run background database schema updates. ([\#11352](matrix-org/synapse#11352)) - Add an admin API for blocking a room. ([\#11324](matrix-org/synapse#11324)) - Update the JWT login type to support custom a `sub` claim. ([\#11361](matrix-org/synapse#11361)) - Store and allow querying of arbitrary event relations. ([\#11391](matrix-org/synapse#11391)) Deprecations and Removals ------------------------- - Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\#11333](matrix-org/synapse#11333), [\#11395](matrix-org/synapse#11395))
Synapse 1.48.0 (2021-11-30) =========================== This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag. This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0. No significant changes since 1.48.0rc1. Synapse 1.48.0rc1 (2021-11-25) ============================== Features -------- - Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\matrix-org#11161](matrix-org#11161)) - Support filtering by relation senders & types per [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\matrix-org#11236](matrix-org#11236)) - Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\matrix-org#11318](matrix-org#11318), [\matrix-org#11371](matrix-org#11371)) - Support the stable version of [MSC2778](matrix-org/matrix-spec-proposals#2778): the `m.login.application_service` login type. Contributed by @tulir. ([\matrix-org#11335](matrix-org#11335)) - Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\matrix-org#11223](matrix-org#11223)) - Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\matrix-org#11228](matrix-org#11228)) - Add an admin API to un-shadow-ban a user. ([\matrix-org#11347](matrix-org#11347)) - Add an admin API to run background database schema updates. ([\matrix-org#11352](matrix-org#11352)) - Add an admin API for blocking a room. ([\matrix-org#11324](matrix-org#11324)) - Update the JWT login type to support custom a `sub` claim. ([\matrix-org#11361](matrix-org#11361)) - Store and allow querying of arbitrary event relations. ([\matrix-org#11391](matrix-org#11391)) Bugfixes -------- - Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error when stored in the DB. ([\matrix-org#11230](matrix-org#11230)) - Prevent [MSC2716](matrix-org/matrix-spec-proposals#2716) historical state events from being pushed to an application service via `/transactions`. ([\matrix-org#11265](matrix-org#11265)) - Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix. ([\matrix-org#11288](matrix-org#11288)) - Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module interface were not affected. ([\matrix-org#11340](matrix-org#11340)) - Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time. ([\matrix-org#11355](matrix-org#11355)) - Fix a bug introduced in 1.45.0 where the `read_templates` method of the module API would error. ([\matrix-org#11377](matrix-org#11377)) - Fix an issue introduced in 1.47.0 which prevented servers re-joining rooms they had previously left, if their signing keys were replaced. ([\matrix-org#11379](matrix-org#11379)) - Fix a bug introduced in 1.13.0 where creating and publishing a room could cause errors if `room_list_publication_rules` is configured. ([\matrix-org#11392](matrix-org#11392)) - Improve performance of various background database updates. ([\matrix-org#11421](matrix-org#11421), [\matrix-org#11422](matrix-org#11422)) Improved Documentation ---------------------- - Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's. ([\matrix-org#11281](matrix-org#11281)) - Fix typos in the documentation for the `username_available` admin API. Contributed by Stanislav Motylkov. ([\matrix-org#11286](matrix-org#11286)) - Add Single Sign-On, SAML and CAS pages to the documentation. ([\matrix-org#11298](matrix-org#11298)) - Change the word 'Home server' as one word 'homeserver' in documentation. ([\matrix-org#11320](matrix-org#11320)) - Fix missing quotes for wildcard domains in `federation_certificate_verification_whitelist`. ([\matrix-org#11381](matrix-org#11381)) Deprecations and Removals ------------------------- - Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\matrix-org#11333](matrix-org#11333), [\matrix-org#11395](matrix-org#11395)) Internal Changes ---------------- - Add type annotations to `synapse.metrics`. ([\matrix-org#10847](matrix-org#10847)) - Split out federated PDU retrieval function into a non-cached version. ([\matrix-org#11242](matrix-org#11242)) - Clean up code relating to to-device messages and sending ephemeral events to application services. ([\matrix-org#11247](matrix-org#11247)) - Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`. ([\matrix-org#11278](matrix-org#11278)) - Drop unused database tables `room_stats_historical` and `user_stats_historical`. ([\matrix-org#11280](matrix-org#11280)) - Require all files in synapse/ and tests/ to pass mypy unless specifically excluded. ([\matrix-org#11282](matrix-org#11282), [\matrix-org#11285](matrix-org#11285), [\matrix-org#11359](matrix-org#11359)) - Add missing type hints to `synapse.app`. ([\matrix-org#11287](matrix-org#11287)) - Remove unused parameters on `FederationEventHandler._check_event_auth`. ([\matrix-org#11292](matrix-org#11292)) - Add type hints to `synapse._scripts`. ([\matrix-org#11297](matrix-org#11297)) - Fix an issue which prevented the `remove_deleted_devices_from_device_inbox` background database schema update from running when updating from a recent Synapse version. ([\matrix-org#11303](matrix-org#11303)) - Add type hints to storage classes. ([\matrix-org#11307](matrix-org#11307), [\matrix-org#11310](matrix-org#11310), [\matrix-org#11311](matrix-org#11311), [\matrix-org#11312](matrix-org#11312), [\matrix-org#11313](matrix-org#11313), [\matrix-org#11314](matrix-org#11314), [\matrix-org#11316](matrix-org#11316), [\matrix-org#11322](matrix-org#11322), [\matrix-org#11332](matrix-org#11332), [\matrix-org#11339](matrix-org#11339), [\matrix-org#11342](matrix-org#11342)) - Add type hints to `synapse.util`. ([\matrix-org#11321](matrix-org#11321), [\matrix-org#11328](matrix-org#11328)) - Improve type annotations in Synapse's test suite. ([\matrix-org#11323](matrix-org#11323), [\matrix-org#11330](matrix-org#11330)) - Test that room alias deletion works as intended. ([\matrix-org#11327](matrix-org#11327)) - Add type annotations for some methods and properties in the module API. ([\matrix-org#11341](matrix-org#11341)) - Fix running `scripts-dev/complement.sh`, which was broken in v1.47.0rc1. ([\matrix-org#11368](matrix-org#11368)) - Rename internal functions for token generation to better reflect what they do. ([\matrix-org#11369](matrix-org#11369), [\matrix-org#11370](matrix-org#11370)) - Add type hints to configuration classes. ([\matrix-org#11377](matrix-org#11377)) - Publish a `develop` image to Docker Hub. ([\matrix-org#11380](matrix-org#11380)) - Keep fallback key marked as used if it's re-uploaded. ([\matrix-org#11382](matrix-org#11382)) - Use `auto_attribs` on the `attrs` class `RefreshTokenLookupResult`. ([\matrix-org#11386](matrix-org#11386)) - Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. ([\matrix-org#11388](matrix-org#11388)) - Do not run the broken MSC2716 tests when running `scripts-dev/complement.sh`. ([\matrix-org#11389](matrix-org#11389)) - Remove dead code from supporting ACME. ([\matrix-org#11393](matrix-org#11393)) - Refactor including the bundled relations when serializing an event. ([\matrix-org#11408](matrix-org#11408))
event_thread_relation
and make it a no-op.The hope here is that if we add a new relation type (e.g.
m.foo
) we could search theevent_relations
table form.foo
events and do further processing on those events, instead of having to crawl all events again.Note that this will have some of the same issues mentioned in #11375 as the method used in the background update is pretty much identical.
Fixes #11387