Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Attempt to annotate events_forward_extremities #11314

Merged
merged 5 commits into from
Nov 12, 2021
Merged

Attempt to annotate events_forward_extremities #11314

merged 5 commits into from
Nov 12, 2021

Conversation

DMRobertson
Copy link
Contributor

It was particularly painful to decipher why the DataStore mro was inconsistent.

David Robertson added 2 commits November 11, 2021 17:13
Make DataStore inherit from EventForwardExtremitiesStore before Cache…
2f57fb1 
…InvalidationWorkerStore

the former implicitly inherits from the latter, so they should be
ordered like this when used.
Attempt to anntoate events_forward_extermities
ab01dad
@DMRobertson DMRobertson requested a review from a team as a code owner November 11, 2021 19:22
clokep
clokep approved these changes Nov 12, 2021
View reviewed changes
synapse/storage/databases/main/events_forward_extremities.py
Comment on lines +26 to +29
class EventForwardExtremitiesStore(
EventFederationWorkerStore,
CacheInvalidationWorkerStore,
):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any concerns this could introduce bugs? The only way I could see that happening is if we're shadowing methods, but I really hope we're not doing that unless it is a worker vs. non-worker version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was scared of doing this, but I'm scared of the storage classes as a whole.

This is a very reasonable question that I don't know how to answer easily.

EventFederationWorkerStore has the following methods. Those without notes aren't defined anywhere else.

  • get_auth_chain
  • get_auth_chain_ids
  • _get_auth_chain_ids_using_cover_index_txn
  • _get_auth_chain_ids_txn
  • get_auth_chain_difference
    • is also defined on StateResolutionStore and is defined on TestStateResolutionStore. I think these are wrappers rather than storage mixins. (?)
  • _get_auth_chain_difference_using_cover_index_txn
  • _get_auth_chain_difference_txn
  • get_oldest_event_ids_with_depth_in_room
  • get_insertion_event_backwards_extremities_in_room
  • get_max_depth_of
  • get_min_depth_of
  • get_prev_events_for_room
  • _get_prev_events_for_room_txn
  • _get_prev_events_for_room_txn
  • get_latest_event_ids_in_room
  • get_min_depth
  • _get_min_depth_interaction
  • get_forward_extremities_for_room_at_stream_ordering
  • _get_forward_extremeties_for_room
  • get_backfill_events
  • _get_backfill_events
  • get_missing_events
    • defined on federation client and transport, but not any other store
  • _get_missing_events
  • get_successor_events
  • _delete_old_forward_extrem_cache
  • insert_insertion_extremity
  • insert_received_event_to_staging
  • remove_received_event_from_staging
  • get_next_staged_event_id_for_room
  • get_next_staged_event_for_room
  • prune_staged_events_in_room
  • get_all_rooms_with_staged_incoming_events
  • _get_stats_for_federation_staging

So the only shadowing is by a wrapper rather than a mixin. I think this is safe? But I'm mainly relying on CI not to explode to be honest.

CacheInvalidationWorkerStore to follow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CacheInvalidationWorkerStore has the following methods. Those without notes aren't defined elsewhere.

  • get_all_updated_caches
  • process_replication_rows defined on
    • CacheInvalidationWorkerStore
    • DeviceInboxWorkerStore
    • EventsWorkerStore
    • PrescenceStore
    • ReceiptsWorkerStore
    • each of these ends with a call to the same method on the super()class, so I think this will traverse the whole MRO. I don't know how we can enforce that though.
  • _process_event_stream_row
  • _invalidate_caches_for_event
  • invalidate_cache_and_stream
  • _invalidate_cache_and_stream
  • _invalidate_all_cache_and_stream
  • _invalidate_state_caches_and_stream
  • _send_invalidation_to_replication
  • get_cache_stream_token_for_writer

Given that every process_replication_rows implementation calls the superclass will be safe provided that EventForwardExtremitiesStore's MRO is consistent with that of DataStore and any other top-level Stores. If the latter was not the case, we'd get a big TypeError at import time when trying to define the DataStore type.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough review! 👍 I agree with your assessment!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect we might need to do this again if I keep pulling in superclasses like this. Maybe there's some way to automate this process?

@DMRobertson
Copy link
Contributor Author

The merge defers to the mypy config added in 95547e5.

@DMRobertson DMRobertson enabled auto-merge (squash) November 12, 2021 15:28
@DMRobertson DMRobertson merged commit 6f8f3d4 into develop Nov 12, 2021
@DMRobertson DMRobertson deleted the dmr/typing/storage/events_forward_extermities branch November 12, 2021 15:58
babolivier added a commit to matrix-org/synapse-dinsic that referenced this pull request Dec 6, 2021
@babolivier
Merge tag 'v1.48.0' into babolivier/dinsic_1.48.0
996bed0 
Synapse 1.48.0 (2021-11-30)
===========================

This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag.

This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0.

No significant changes since 1.48.0rc1.

Synapse 1.48.0rc1 (2021-11-25)
==============================

Features
--------

- Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11161](matrix-org/synapse#11161))
- Support filtering by relation senders & types per [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\#11236](matrix-org/synapse#11236))
- Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\#11318](matrix-org/synapse#11318), [\#11371](matrix-org/synapse#11371))
- Support the stable version of [MSC2778](matrix-org/matrix-spec-proposals#2778): the `m.login.application_service` login type. Contributed by @tulir. ([\#11335](matrix-org/synapse#11335))
- Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\#11223](matrix-org/synapse#11223))
- Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\#11228](matrix-org/synapse#11228))
- Add an admin API to un-shadow-ban a user. ([\#11347](matrix-org/synapse#11347))
- Add an admin API to run background database schema updates. ([\#11352](matrix-org/synapse#11352))
- Add an admin API for blocking a room. ([\#11324](matrix-org/synapse#11324))
- Update the JWT login type to support custom a `sub` claim. ([\#11361](matrix-org/synapse#11361))
- Store and allow querying of arbitrary event relations. ([\#11391](matrix-org/synapse#11391))

Bugfixes
--------

- Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error when stored in the DB. ([\#11230](matrix-org/synapse#11230))
- Prevent [MSC2716](matrix-org/matrix-spec-proposals#2716) historical state events from being pushed to an application service via `/transactions`. ([\#11265](matrix-org/synapse#11265))
- Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix. ([\#11288](matrix-org/synapse#11288))
- Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module interface were not affected. ([\#11340](matrix-org/synapse#11340))
- Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time. ([\#11355](matrix-org/synapse#11355))
- Fix a bug introduced in 1.45.0 where the `read_templates` method of the module API would error. ([\#11377](matrix-org/synapse#11377))
- Fix an issue introduced in 1.47.0 which prevented servers re-joining rooms they had previously left, if their signing keys were replaced. ([\#11379](matrix-org/synapse#11379))
- Fix a bug introduced in 1.13.0 where creating and publishing a room could cause errors if `room_list_publication_rules` is configured. ([\#11392](matrix-org/synapse#11392))
- Improve performance of various background database updates. ([\#11421](matrix-org/synapse#11421), [\#11422](matrix-org/synapse#11422))

Improved Documentation
----------------------

- Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's. ([\#11281](matrix-org/synapse#11281))
- Fix typos in the documentation for the `username_available` admin API. Contributed by Stanislav Motylkov. ([\#11286](matrix-org/synapse#11286))
- Add Single Sign-On, SAML and CAS pages to the documentation. ([\#11298](matrix-org/synapse#11298))
- Change the word 'Home server' as one word 'homeserver' in documentation. ([\#11320](matrix-org/synapse#11320))
- Fix missing quotes for wildcard domains in `federation_certificate_verification_whitelist`. ([\#11381](matrix-org/synapse#11381))

Deprecations and Removals
-------------------------

- Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\#11333](matrix-org/synapse#11333), [\#11395](matrix-org/synapse#11395))

Internal Changes
----------------

- Add type annotations to `synapse.metrics`. ([\#10847](matrix-org/synapse#10847))
- Split out federated PDU retrieval function into a non-cached version. ([\#11242](matrix-org/synapse#11242))
- Clean up code relating to to-device messages and sending ephemeral events to application services. ([\#11247](matrix-org/synapse#11247))
- Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`. ([\#11278](matrix-org/synapse#11278))
- Drop unused database tables `room_stats_historical` and `user_stats_historical`. ([\#11280](matrix-org/synapse#11280))
- Require all files in synapse/ and tests/ to pass mypy unless specifically excluded. ([\#11282](matrix-org/synapse#11282), [\#11285](matrix-org/synapse#11285), [\#11359](matrix-org/synapse#11359))
- Add missing type hints to `synapse.app`. ([\#11287](matrix-org/synapse#11287))
- Remove unused parameters on `FederationEventHandler._check_event_auth`. ([\#11292](matrix-org/synapse#11292))
- Add type hints to `synapse._scripts`. ([\#11297](matrix-org/synapse#11297))
- Fix an issue which prevented the `remove_deleted_devices_from_device_inbox` background database schema update from running when updating from a recent Synapse version. ([\#11303](matrix-org/synapse#11303))
- Add type hints to storage classes. ([\#11307](matrix-org/synapse#11307), [\#11310](matrix-org/synapse#11310), [\#11311](matrix-org/synapse#11311), [\#11312](matrix-org/synapse#11312), [\#11313](matrix-org/synapse#11313), [\#11314](matrix-org/synapse#11314), [\#11316](matrix-org/synapse#11316), [\#11322](matrix-org/synapse#11322), [\#11332](matrix-org/synapse#11332), [\#11339](matrix-org/synapse#11339), [\#11342](matrix-org/synapse#11342))
- Add type hints to `synapse.util`. ([\#11321](matrix-org/synapse#11321), [\#11328](matrix-org/synapse#11328))
- Improve type annotations in Synapse's test suite. ([\#11323](matrix-org/synapse#11323), [\#11330](matrix-org/synapse#11330))
- Test that room alias deletion works as intended. ([\#11327](matrix-org/synapse#11327))
- Add type annotations for some methods and properties in the module API. ([\#11341](matrix-org/synapse#11341))
- Fix running `scripts-dev/complement.sh`, which was broken in v1.47.0rc1. ([\#11368](matrix-org/synapse#11368))
- Rename internal functions for token generation to better reflect what they do. ([\#11369](matrix-org/synapse#11369), [\#11370](matrix-org/synapse#11370))
- Add type hints to configuration classes. ([\#11377](matrix-org/synapse#11377))
- Publish a `develop` image to Docker Hub. ([\#11380](matrix-org/synapse#11380))
- Keep fallback key marked as used if it's re-uploaded. ([\#11382](matrix-org/synapse#11382))
- Use `auto_attribs` on the `attrs` class `RefreshTokenLookupResult`. ([\#11386](matrix-org/synapse#11386))
- Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. ([\#11388](matrix-org/synapse#11388))
- Do not run the broken MSC2716 tests when running `scripts-dev/complement.sh`. ([\#11389](matrix-org/synapse#11389))
- Remove dead code from supporting ACME. ([\#11393](matrix-org/synapse#11393))
- Refactor including the bundled relations when serializing an event. ([\#11408](matrix-org/synapse#11408))
Fizzadar added a commit to Fizzadar/synapse that referenced this pull request Jan 5, 2022
@Fizzadar
Merge tag 'v1.48.0' into beeper-master
d0596f1 
Synapse 1.48.0 (2021-11-30)
===========================

This release removes support for the long-deprecated `trust_identity_server_for_password_resets` configuration flag.

This release also fixes some performance issues with some background database updates introduced in Synapse 1.47.0.

No significant changes since 1.48.0rc1.

Synapse 1.48.0rc1 (2021-11-25)
==============================

Features
--------

- Experimental support for the thread relation defined in [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\matrix-org#11161](matrix-org#11161))
- Support filtering by relation senders & types per [MSC3440](matrix-org/matrix-spec-proposals#3440). ([\matrix-org#11236](matrix-org#11236))
- Add support for the `/_matrix/client/v3` and `/_matrix/media/v3` APIs from Matrix v1.1. ([\matrix-org#11318](matrix-org#11318), [\matrix-org#11371](matrix-org#11371))
- Support the stable version of [MSC2778](matrix-org/matrix-spec-proposals#2778): the `m.login.application_service` login type. Contributed by @tulir. ([\matrix-org#11335](matrix-org#11335))
- Add a new version of delete room admin API `DELETE /_synapse/admin/v2/rooms/<room_id>` to run it in the background. Contributed by @dklimpel. ([\matrix-org#11223](matrix-org#11223))
- Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it. ([\matrix-org#11228](matrix-org#11228))
- Add an admin API to un-shadow-ban a user. ([\matrix-org#11347](matrix-org#11347))
- Add an admin API to run background database schema updates. ([\matrix-org#11352](matrix-org#11352))
- Add an admin API for blocking a room. ([\matrix-org#11324](matrix-org#11324))
- Update the JWT login type to support custom a `sub` claim. ([\matrix-org#11361](matrix-org#11361))
- Store and allow querying of arbitrary event relations. ([\matrix-org#11391](matrix-org#11391))

Bugfixes
--------

- Fix a long-standing bug wherein display names or avatar URLs containing null bytes cause an internal server error when stored in the DB. ([\matrix-org#11230](matrix-org#11230))
- Prevent [MSC2716](matrix-org/matrix-spec-proposals#2716) historical state events from being pushed to an application service via `/transactions`. ([\matrix-org#11265](matrix-org#11265))
- Fix a long-standing bug where uploading extremely thin images (e.g. 1000x1) would fail. Contributed by @Neeeflix. ([\matrix-org#11288](matrix-org#11288))
- Fix a bug, introduced in Synapse 1.46.0, which caused the `check_3pid_auth` and `on_logged_out` callbacks in legacy password authentication provider modules to not be registered. Modules using the generic module interface were not affected. ([\matrix-org#11340](matrix-org#11340))
- Fix a bug introduced in 1.41.0 where space hierarchy responses would be incorrectly reused if multiple users were to make the same request at the same time. ([\matrix-org#11355](matrix-org#11355))
- Fix a bug introduced in 1.45.0 where the `read_templates` method of the module API would error. ([\matrix-org#11377](matrix-org#11377))
- Fix an issue introduced in 1.47.0 which prevented servers re-joining rooms they had previously left, if their signing keys were replaced. ([\matrix-org#11379](matrix-org#11379))
- Fix a bug introduced in 1.13.0 where creating and publishing a room could cause errors if `room_list_publication_rules` is configured. ([\matrix-org#11392](matrix-org#11392))
- Improve performance of various background database updates. ([\matrix-org#11421](matrix-org#11421), [\matrix-org#11422](matrix-org#11422))

Improved Documentation
----------------------

- Suggest users of the Debian packages add configuration to `/etc/matrix-synapse/conf.d/` to prevent, upon upgrade, being asked to choose between their configuration and the maintainer's. ([\matrix-org#11281](matrix-org#11281))
- Fix typos in the documentation for the `username_available` admin API. Contributed by Stanislav Motylkov. ([\matrix-org#11286](matrix-org#11286))
- Add Single Sign-On, SAML and CAS pages to the documentation. ([\matrix-org#11298](matrix-org#11298))
- Change the word 'Home server' as one word 'homeserver' in documentation. ([\matrix-org#11320](matrix-org#11320))
- Fix missing quotes for wildcard domains in `federation_certificate_verification_whitelist`. ([\matrix-org#11381](matrix-org#11381))

Deprecations and Removals
-------------------------

- Remove deprecated `trust_identity_server_for_password_resets` configuration flag. ([\matrix-org#11333](matrix-org#11333), [\matrix-org#11395](matrix-org#11395))

Internal Changes
----------------

- Add type annotations to `synapse.metrics`. ([\matrix-org#10847](matrix-org#10847))
- Split out federated PDU retrieval function into a non-cached version. ([\matrix-org#11242](matrix-org#11242))
- Clean up code relating to to-device messages and sending ephemeral events to application services. ([\matrix-org#11247](matrix-org#11247))
- Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`. ([\matrix-org#11278](matrix-org#11278))
- Drop unused database tables `room_stats_historical` and `user_stats_historical`. ([\matrix-org#11280](matrix-org#11280))
- Require all files in synapse/ and tests/ to pass mypy unless specifically excluded. ([\matrix-org#11282](matrix-org#11282), [\matrix-org#11285](matrix-org#11285), [\matrix-org#11359](matrix-org#11359))
- Add missing type hints to `synapse.app`. ([\matrix-org#11287](matrix-org#11287))
- Remove unused parameters on `FederationEventHandler._check_event_auth`. ([\matrix-org#11292](matrix-org#11292))
- Add type hints to `synapse._scripts`. ([\matrix-org#11297](matrix-org#11297))
- Fix an issue which prevented the `remove_deleted_devices_from_device_inbox` background database schema update from running when updating from a recent Synapse version. ([\matrix-org#11303](matrix-org#11303))
- Add type hints to storage classes. ([\matrix-org#11307](matrix-org#11307), [\matrix-org#11310](matrix-org#11310), [\matrix-org#11311](matrix-org#11311), [\matrix-org#11312](matrix-org#11312), [\matrix-org#11313](matrix-org#11313), [\matrix-org#11314](matrix-org#11314), [\matrix-org#11316](matrix-org#11316), [\matrix-org#11322](matrix-org#11322), [\matrix-org#11332](matrix-org#11332), [\matrix-org#11339](matrix-org#11339), [\matrix-org#11342](matrix-org#11342))
- Add type hints to `synapse.util`. ([\matrix-org#11321](matrix-org#11321), [\matrix-org#11328](matrix-org#11328))
- Improve type annotations in Synapse's test suite. ([\matrix-org#11323](matrix-org#11323), [\matrix-org#11330](matrix-org#11330))
- Test that room alias deletion works as intended. ([\matrix-org#11327](matrix-org#11327))
- Add type annotations for some methods and properties in the module API. ([\matrix-org#11341](matrix-org#11341))
- Fix running `scripts-dev/complement.sh`, which was broken in v1.47.0rc1. ([\matrix-org#11368](matrix-org#11368))
- Rename internal functions for token generation to better reflect what they do. ([\matrix-org#11369](matrix-org#11369), [\matrix-org#11370](matrix-org#11370))
- Add type hints to configuration classes. ([\matrix-org#11377](matrix-org#11377))
- Publish a `develop` image to Docker Hub. ([\matrix-org#11380](matrix-org#11380))
- Keep fallback key marked as used if it's re-uploaded. ([\matrix-org#11382](matrix-org#11382))
- Use `auto_attribs` on the `attrs` class `RefreshTokenLookupResult`. ([\matrix-org#11386](matrix-org#11386))
- Rename unstable `access_token_lifetime` configuration option to `refreshable_access_token_lifetime` to make it clear it only concerns refreshable access tokens. ([\matrix-org#11388](matrix-org#11388))
- Do not run the broken MSC2716 tests when running `scripts-dev/complement.sh`. ([\matrix-org#11389](matrix-org#11389))
- Remove dead code from supporting ACME. ([\matrix-org#11393](matrix-org#11393))
- Refactor including the bundled relations when serializing an event. ([\matrix-org#11408](matrix-org#11408))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@clokep clokep clokep approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@DMRobertson @clokep