Add a type hint for `get_device_handler()` and deal with the fallout. #14055

clokep · 2022-10-04T19:23:01Z

I'm not 100% thrilled with the way this came out, but not having a type hint on properties of HomeServer is wildly dangerous IMO.

clokep · 2022-10-04T20:03:07Z

Hm, looks like some real failures. Will need to investigate!

DMRobertson

Drive-by comment: I like the pattern of having an "updater" object which either makes a change or asks an authoritative worker (in this case, the master) to make it for us.

DMRobertson · 2022-10-31T13:14:03Z

This would have caught a slip-up of mine. LMK if you want a hand getting this over the line.

Edit: namely the slip-up fixed in https://github.com/matrix-org/synapse/pull/14336/files#r1009407320

clokep · 2022-10-31T13:27:43Z

This would have caught a slip-up of mine. LMK if you want a hand getting this over the line.

I mostly just haven't had time to look back at it. If you'd like to, feel free to push more commits! 🎉

clokep · 2022-11-18T13:56:06Z

I'm pretty sure this will pass everything now. :)

DMRobertson

I think this is sane---though I'm still generally baffled by what does and doesn't get allocated to workers etc

DMRobertson · 2022-11-21T12:13:29Z

synapse/handlers/device.py

@@ -320,6 +324,8 @@ async def handle_room_un_partial_stated(self, room_id: str) -> None:


 class DeviceHandler(DeviceWorkerHandler):


What's the difference between DeviceHandler and DeviceWorkerHandler now? Has this changed as a result of your patch here?

I moved get_dehydrated_device from DeviceHandler to DeviceWorkerHandler. The only other change is that their device_list_updater are now DeviceListUpdater or DeviceListWorkerUpdater, respectively. The latter has an overridden method (user_device_resync) to call across replication to the main process.

synapse/module_api/__init__.py

synapse/rest/admin/__init__.py

clokep · 2022-11-22T18:29:20Z

@DMRobertson This needed more changes, I'm pretty sure it'll pass this time though.

DMRobertson

Many thanks again for this, I think this really helps clarify the tangled web we've weaved.

Sytest passed this time. Fingers cross complement will too, modulo flakes!

…14055) This was the last untyped handler from the HomeServer object. Since it was being treated as Any (and thus unchecked) it was being used incorrectly in a few places.

@ashfame

Synapse 1.73.0 (2022-12-06) =========================== Please note that legacy Prometheus metric names have been removed in this release; see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.73/docs/upgrade.md#legacy-prometheus-metric-names-have-now-been-removed) for more details. No significant changes since 1.73.0rc2. Synapse 1.73.0rc2 (2022-12-01) ============================== Bugfixes -------- - Fix a regression in Synapse 1.73.0rc1 where Synapse's main process would stop responding to HTTP requests when a user with a large number of devices logs in. ([\matrix-org#14582](matrix-org#14582)) Synapse 1.73.0rc1 (2022-11-29) ============================== Features -------- - Speed-up `/messages` with `filter_events_for_client` optimizations. ([\matrix-org#14527](matrix-org#14527)) - Improve DB performance by reducing amount of data that gets read in `device_lists_changes_in_room`. ([\matrix-org#14534](matrix-org#14534)) - Adds support for handling avatar in SSO OIDC login. Contributed by @ashfame. ([\matrix-org#13917](matrix-org#13917)) - Move MSC3030 `/timestamp_to_event` endpoints to stable `v1` location (`/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`, `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`). ([\matrix-org#14471](matrix-org#14471)) - Reduce database load of [Client-Server endpoints](https://spec.matrix.org/v1.5/client-server-api/#aggregations) which return bundled aggregations. ([\matrix-org#14491](matrix-org#14491), [\matrix-org#14508](matrix-org#14508), [\matrix-org#14510](matrix-org#14510)) - Add unstable support for an Extensible Events room version (`org.matrix.msc1767.10`) via [MSC1767](matrix-org/matrix-spec-proposals#1767), [MSC3931](matrix-org/matrix-spec-proposals#3931), [MSC3932](matrix-org/matrix-spec-proposals#3932), and [MSC3933](matrix-org/matrix-spec-proposals#3933). ([\matrix-org#14520](matrix-org#14520), [\matrix-org#14521](matrix-org#14521), [\matrix-org#14524](matrix-org#14524)) - Prune user's old devices on login if they have too many. ([\matrix-org#14038](matrix-org#14038), [\matrix-org#14580](matrix-org#14580)) Bugfixes -------- - Fix a long-standing bug where paginating from the start of a room did not work. Contributed by @gnunicorn. ([\matrix-org#14149](matrix-org#14149)) - Fix a bug introduced in Synapse 1.58.0 where a user with presence state `org.matrix.msc3026.busy` would mistakenly be set to `online` when calling `/sync` or `/events` on a worker process. ([\matrix-org#14393](matrix-org#14393)) - Fix a bug introduced in Synapse 1.70.0 where a receipt's thread ID was not sent over federation. ([\matrix-org#14466](matrix-org#14466)) - Fix a long-standing bug where the [List media admin API](https://matrix-org.github.io/synapse/latest/admin_api/media_admin_api.html#list-all-media-in-a-room) would fail when processing an image with broken thumbnail information. ([\matrix-org#14537](matrix-org#14537)) - Fix a bug introduced in Synapse 1.67.0 where two logging context warnings would be logged on startup. ([\matrix-org#14574](matrix-org#14574)) - In application service transactions that include the experimental `org.matrix.msc3202.device_one_time_key_counts` key, include a duplicate key of `org.matrix.msc3202.device_one_time_keys_count` to match the name proposed by [MSC3202](matrix-org/matrix-spec-proposals#3202). ([\matrix-org#14565](matrix-org#14565)) - Fix a bug introduced in Synapse 0.9 where Synapse would fail to fetch server keys whose IDs contain a forward slash. ([\matrix-org#14490](matrix-org#14490)) Improved Documentation ---------------------- - Fixed link to 'Synapse administration endpoints'. ([\matrix-org#14499](matrix-org#14499)) Deprecations and Removals ------------------------- - Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. ([\matrix-org#14538](matrix-org#14538)) Internal Changes ---------------- - Improve type hinting throughout Synapse. ([\matrix-org#14055](matrix-org#14055), [\matrix-org#14412](matrix-org#14412), [\matrix-org#14529](matrix-org#14529), [\matrix-org#14452](matrix-org#14452)). - Remove old stream ID tracking code. Contributed by Nick @beeper (@Fizzadar). ([\matrix-org#14376](matrix-org#14376), [\matrix-org#14468](matrix-org#14468)) - Remove the `worker_main_http_uri` configuration setting. This is now handled via internal replication. ([\matrix-org#14400](matrix-org#14400), [\matrix-org#14476](matrix-org#14476)) - Refactor `federation_sender` and `pusher` configuration loading. ([\matrix-org#14496](matrix-org#14496)) ([\matrix-org#14509](matrix-org#14509), [\matrix-org#14573](matrix-org#14573)) - Faster joins: do not wait for full state when creating events to send. ([\matrix-org#14403](matrix-org#14403)) - Faster joins: filter out non local events when a room doesn't have its full state. ([\matrix-org#14404](matrix-org#14404)) - Faster joins: send events to initial list of servers if we don't have the full state yet. ([\matrix-org#14408](matrix-org#14408)) - Faster joins: use servers list approximation received during `send_join` (potentially updated with received membership events) in `assert_host_in_room`. ([\matrix-org#14515](matrix-org#14515)) - Fix type logic in TCP replication code that prevented correctly ignoring blank commands. ([\matrix-org#14449](matrix-org#14449)) - Remove option to skip locking of tables when performing emulated upserts, to avoid a class of bugs in future. ([\matrix-org#14469](matrix-org#14469)) - `scripts-dev/federation_client`: Fix routing on servers with `.well-known` files. ([\matrix-org#14479](matrix-org#14479)) - Reduce default third party invite rate limit to 216 invites per day. ([\matrix-org#14487](matrix-org#14487)) - Refactor conversion of device list changes in room to outbound pokes to track unconverted rows using a `(stream ID, room ID)` position instead of updating the `converted_to_destinations` flag on every row. ([\matrix-org#14516](matrix-org#14516)) - Add more prompts to the bug report form. ([\matrix-org#14522](matrix-org#14522)) - Extend editorconfig rules on indent and line length to `.pyi` files. ([\matrix-org#14526](matrix-org#14526)) - Run Rust CI when `Cargo.lock` changes. This is particularly useful for dependabot updates. ([\matrix-org#14571](matrix-org#14571)) - Fix a possible variable shadow in `create_new_client_event`. ([\matrix-org#14575](matrix-org#14575)) - Bump various dependencies in the `poetry.lock` file and in CI scripts. ([\matrix-org#14557](matrix-org#14557), [\matrix-org#14559](matrix-org#14559), [\matrix-org#14560](matrix-org#14560), [\matrix-org#14500](matrix-org#14500), [\matrix-org#14501](matrix-org#14501), [\matrix-org#14502](matrix-org#14502), [\matrix-org#14503](matrix-org#14503), [\matrix-org#14504](matrix-org#14504), [\matrix-org#14505](matrix-org#14505)). # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmOPLnYACgkQLS76LzL7 # 4Edwpg/+KXpg2ZdiJ0Yaly9VHVeiqdHRi5D7WPS6n8YBsdRx9EQHzOBkD5HAW8hE # oz0c+zDS01ORlEWD825NYXjgaE1ijtZFvGxsftYTVuTYlVRR2m+r9jhDv9pVHT53 # TKtQVKpG0IUsuyukRBrweDcEeO0MA0nGpvaaQUhmftzWgy4yD3AjZyIgx0Ckg8pg # OwgrzGqA7FQs4MEeOxmk1H39fZg4dlo4nmI4whvAodgaGeS9sU8t+3Qj4PVod8v/ # AkVesJcruaTHuVMb+Xp8JKezb09SsIR94gmHalC5sL+41+6XAy9BtQ/cRDfCReG3 # U1I1x1h1+EQjTP6XzMmjQHLbfI2gUJBC4I2p3e2gZ4cMm9rVz94R1dBiRk8ZgRIC # cJFD9BvaAtb2PSTvyFBoHsrrn/u12i8fYFWu4Z4rO6dOGI83dZHeZzVw4UsVeqIK # 5+njQwcwQsrwL3AKLjbbdqmbmhXcF6LchIK2L+NuuvdiOfvXvkO0bdjBryVEbMqB # IOtAAWzwYaoUwVucMbBtXt/EqQS7biGkbDxsL8CDvaBwM/JSsUWXBafsV1FmxF2A # q6KAeKpfelefoegosTYD0Md+l39xdF8Z19XaKV3GeHZEY+HE3RJXJm+Pa8SJ+IF8 # Y1od9cB/H+fYSsWCWj1OJNqTIAozh6f1Pe2nFuFDxdBwABXc/pg= # =IBEL # -----END PGP SIGNATURE----- # gpg: Signature made Tue Dec 6 11:58:46 2022 GMT # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # poetry.lock # synapse/push/bulk_push_rule_evaluator.py # synapse/storage/databases/main/account_data.py # synapse/storage/databases/main/receipts.py

@ashfame

Synapse 1.73.0 (2022-12-06) =========================== Please note that legacy Prometheus metric names have been removed in this release; see [the upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.73/docs/upgrade.md#legacy-prometheus-metric-names-have-now-been-removed) for more details. No significant changes since 1.73.0rc2. Synapse 1.73.0rc2 (2022-12-01) ============================== Bugfixes -------- - Fix a regression in Synapse 1.73.0rc1 where Synapse's main process would stop responding to HTTP requests when a user with a large number of devices logs in. ([\#14582](matrix-org/synapse#14582)) Synapse 1.73.0rc1 (2022-11-29) ============================== Features -------- - Speed-up `/messages` with `filter_events_for_client` optimizations. ([\#14527](matrix-org/synapse#14527)) - Improve DB performance by reducing amount of data that gets read in `device_lists_changes_in_room`. ([\#14534](matrix-org/synapse#14534)) - Adds support for handling avatar in SSO OIDC login. Contributed by @ashfame. ([\#13917](matrix-org/synapse#13917)) - Move MSC3030 `/timestamp_to_event` endpoints to stable `v1` location (`/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`, `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`). ([\#14471](matrix-org/synapse#14471)) - Reduce database load of [Client-Server endpoints](https://spec.matrix.org/v1.5/client-server-api/#aggregations) which return bundled aggregations. ([\#14491](matrix-org/synapse#14491), [\#14508](matrix-org/synapse#14508), [\#14510](matrix-org/synapse#14510)) - Add unstable support for an Extensible Events room version (`org.matrix.msc1767.10`) via [MSC1767](matrix-org/matrix-spec-proposals#1767), [MSC3931](matrix-org/matrix-spec-proposals#3931), [MSC3932](matrix-org/matrix-spec-proposals#3932), and [MSC3933](matrix-org/matrix-spec-proposals#3933). ([\#14520](matrix-org/synapse#14520), [\#14521](matrix-org/synapse#14521), [\#14524](matrix-org/synapse#14524)) - Prune user's old devices on login if they have too many. ([\#14038](matrix-org/synapse#14038), [\#14580](matrix-org/synapse#14580)) Bugfixes -------- - Fix a long-standing bug where paginating from the start of a room did not work. Contributed by @gnunicorn. ([\#14149](matrix-org/synapse#14149)) - Fix a bug introduced in Synapse 1.58.0 where a user with presence state `org.matrix.msc3026.busy` would mistakenly be set to `online` when calling `/sync` or `/events` on a worker process. ([\#14393](matrix-org/synapse#14393)) - Fix a bug introduced in Synapse 1.70.0 where a receipt's thread ID was not sent over federation. ([\#14466](matrix-org/synapse#14466)) - Fix a long-standing bug where the [List media admin API](https://matrix-org.github.io/synapse/latest/admin_api/media_admin_api.html#list-all-media-in-a-room) would fail when processing an image with broken thumbnail information. ([\#14537](matrix-org/synapse#14537)) - Fix a bug introduced in Synapse 1.67.0 where two logging context warnings would be logged on startup. ([\#14574](matrix-org/synapse#14574)) - In application service transactions that include the experimental `org.matrix.msc3202.device_one_time_key_counts` key, include a duplicate key of `org.matrix.msc3202.device_one_time_keys_count` to match the name proposed by [MSC3202](matrix-org/matrix-spec-proposals#3202). ([\#14565](matrix-org/synapse#14565)) - Fix a bug introduced in Synapse 0.9 where Synapse would fail to fetch server keys whose IDs contain a forward slash. ([\#14490](matrix-org/synapse#14490)) Improved Documentation ---------------------- - Fixed link to 'Synapse administration endpoints'. ([\#14499](matrix-org/synapse#14499)) Deprecations and Removals ------------------------- - Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. ([\#14538](matrix-org/synapse#14538)) Internal Changes ---------------- - Improve type hinting throughout Synapse. ([\#14055](matrix-org/synapse#14055), [\#14412](matrix-org/synapse#14412), [\#14529](matrix-org/synapse#14529), [\#14452](matrix-org/synapse#14452)). - Remove old stream ID tracking code. Contributed by Nick @beeper (@Fizzadar). ([\#14376](matrix-org/synapse#14376), [\#14468](matrix-org/synapse#14468)) - Remove the `worker_main_http_uri` configuration setting. This is now handled via internal replication. ([\#14400](matrix-org/synapse#14400), [\#14476](matrix-org/synapse#14476)) - Refactor `federation_sender` and `pusher` configuration loading. ([\#14496](matrix-org/synapse#14496)) ([\#14509](matrix-org/synapse#14509), [\#14573](matrix-org/synapse#14573)) - Faster joins: do not wait for full state when creating events to send. ([\#14403](matrix-org/synapse#14403)) - Faster joins: filter out non local events when a room doesn't have its full state. ([\#14404](matrix-org/synapse#14404)) - Faster joins: send events to initial list of servers if we don't have the full state yet. ([\#14408](matrix-org/synapse#14408)) - Faster joins: use servers list approximation received during `send_join` (potentially updated with received membership events) in `assert_host_in_room`. ([\#14515](matrix-org/synapse#14515)) - Fix type logic in TCP replication code that prevented correctly ignoring blank commands. ([\#14449](matrix-org/synapse#14449)) - Remove option to skip locking of tables when performing emulated upserts, to avoid a class of bugs in future. ([\#14469](matrix-org/synapse#14469)) - `scripts-dev/federation_client`: Fix routing on servers with `.well-known` files. ([\#14479](matrix-org/synapse#14479)) - Reduce default third party invite rate limit to 216 invites per day. ([\#14487](matrix-org/synapse#14487)) - Refactor conversion of device list changes in room to outbound pokes to track unconverted rows using a `(stream ID, room ID)` position instead of updating the `converted_to_destinations` flag on every row. ([\#14516](matrix-org/synapse#14516)) - Add more prompts to the bug report form. ([\#14522](matrix-org/synapse#14522)) - Extend editorconfig rules on indent and line length to `.pyi` files. ([\#14526](matrix-org/synapse#14526)) - Run Rust CI when `Cargo.lock` changes. This is particularly useful for dependabot updates. ([\#14571](matrix-org/synapse#14571)) - Fix a possible variable shadow in `create_new_client_event`. ([\#14575](matrix-org/synapse#14575)) - Bump various dependencies in the `poetry.lock` file and in CI scripts. ([\#14557](matrix-org/synapse#14557), [\#14559](matrix-org/synapse#14559), [\#14560](matrix-org/synapse#14560), [\#14500](matrix-org/synapse#14500), [\#14501](matrix-org/synapse#14501), [\#14502](matrix-org/synapse#14502), [\#14503](matrix-org/synapse#14503), [\#14504](matrix-org/synapse#14504), [\#14505](matrix-org/synapse#14505)).

clokep added 8 commits October 4, 2022 13:53

Add return type to get_device_handler.

7433621

For servlets on the main process, assert DeviceHandler.

1d9689f

Some calls to the handler can only be done on the main process.

6cd2bfb

Create a separate DeviceListUpdaterWorker.

f517c48

Fix-up E2EKeysHandler.

7a2cd11

Fix-up the module API.

97cf182

Lint

6af6d09

Newsfragment

e550205

DMRobertson reviewed Oct 5, 2022

View reviewed changes

DMRobertson mentioned this pull request Oct 31, 2022

Fix dehydrated device REST checks #14336

Merged

clokep and others added 6 commits October 31, 2022 09:27

Merge branch 'develop' into clokep/device-handler

6c8e914

Merge remote-tracking branch 'origin/develop' into clokep/device-handler

a0e3d05

Fix new typing errors.

f0635e1

Fix tests.

844d010

Fix tests again.

29d66fa

Merge branch 'develop' into clokep/device-handler

1ad1c3f

clokep marked this pull request as ready for review November 18, 2022 13:55

clokep requested a review from a team as a code owner November 18, 2022 13:55

clokep requested review from DMRobertson and removed request for a team November 18, 2022 13:55

DMRobertson approved these changes Nov 21, 2022

View reviewed changes

clokep added 2 commits November 22, 2022 12:16

Merge remote-tracking branch 'origin/develop' into clokep/device-handler

541ab6c

Fix tests for real?

2d7bd2a

clokep requested a review from DMRobertson November 22, 2022 18:29

DMRobertson approved these changes Nov 22, 2022

View reviewed changes

clokep merged commit 6d47b7e into develop Nov 22, 2022

clokep deleted the clokep/device-handler branch November 22, 2022 19:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a type hint for `get_device_handler()` and deal with the fallout. #14055

Add a type hint for `get_device_handler()` and deal with the fallout. #14055

clokep commented Oct 4, 2022

clokep commented Oct 4, 2022

DMRobertson left a comment

DMRobertson commented Oct 31, 2022 •

edited

Loading

clokep commented Oct 31, 2022

clokep commented Nov 18, 2022

DMRobertson left a comment

DMRobertson Nov 21, 2022

clokep Nov 22, 2022

clokep commented Nov 22, 2022

DMRobertson left a comment

		@@ -320,6 +324,8 @@ async def handle_room_un_partial_stated(self, room_id: str) -> None:


		class DeviceHandler(DeviceWorkerHandler):

Add a type hint for get_device_handler() and deal with the fallout. #14055

Add a type hint for get_device_handler() and deal with the fallout. #14055

Conversation

clokep commented Oct 4, 2022

clokep commented Oct 4, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

DMRobertson commented Oct 31, 2022 • edited Loading

clokep commented Oct 31, 2022

clokep commented Nov 18, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

DMRobertson Nov 21, 2022

Choose a reason for hiding this comment

clokep Nov 22, 2022

Choose a reason for hiding this comment

clokep commented Nov 22, 2022

DMRobertson left a comment

Choose a reason for hiding this comment

Add a type hint for `get_device_handler()` and deal with the fallout. #14055

Add a type hint for `get_device_handler()` and deal with the fallout. #14055

DMRobertson commented Oct 31, 2022 •

edited

Loading