Policy server part 1: Actually call the policy server #18387

turt2live · 2025-05-02T19:14:25Z

Roughly reviewable commit-by-commit.

This is the first part of adding policy server support to Synapse. Other parts (unordered), which may or may not be bundled into fewer PRs, include:

Implementation of a bulk API
Supporting a moderation server config (the fallback_* options of https://github.com/element-hq/policyserv_spam_checker )
Adding an "early event hook" for appservices to receive federation transactions before events are processed formally
Performance and stability improvements

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct
(run the linters)

turt2live · 2025-05-15T05:03:10Z

I'm not overly familiar with Synapse's codebase yet, and struggling to find good places to copy/paste some code to set up some tests. I think ideally we have tests on RoomPolicyHandler, but I'm just not sure how to make a test that can spin up a second homeserver to "join" the room to appease the is_in_room check within RoomPolicyHandler. I'm confident we can intercept the outbound /check call for testing, but getting the room to have 2 servers in it looks difficult at best.

Thoughts (or code - please just push to the branch 😇) are appreciated on this.

(equally, I'm personally okay to just land this without tests, but I suspect team policy might override that opinion)

kegsay

I'm just not sure how to make a test that can spin up a second homeserver to "join" the room to appease the is_in_room check within RoomPolicyHandler

https://github.com/element-hq/synapse/blob/develop/tests/handlers/test_federation_event.py#L48 is probably your best bet to copypasta I think? You can see it injecting a remote member https://github.com/element-hq/synapse/blob/develop/tests/handlers/test_federation_event.py#L114

kegsay · 2025-05-15T06:46:48Z

synapse/federation/federation_client.py

+            return recommendation
+        except Exception as e:
+            logger.warning(
+                "get_pdu_policy_recommendation: server %s responded with error, assuming OK recommendation: %s",


Nit: I'd rather use f-strings personally, they are waaaay more ergonomic. E.g

logger.warning(f"get_pdu_policy_recommendation: server {destination} responded with error, assuming OK recommendation: {e}")

We don't use f-strings in loggers, while they are more ergonomic it means that you are passing in the formatted string, rather than allowing the loggers to do so. The main consequence is that if the logger is disabled you don't necessarily want to pay the cost of string interpolation.

Otherwise, yeah f-strings are awesome.

with my limited python exposure, +1 to fstrings, but definitely don't want to make things harder for the logger

kegsay · 2025-05-15T06:50:34Z

synapse/handlers/room_policy.py

+            return True  # no policy server == default allow
+
+        if policy_server == self._hs.hostname:
+            return True  # Synapse itself can't be a policy server (currently)


And it can't be run on the same domain with a .well-known lookup or something? I'd rather we didn't hard code this given if/when Synapse can be a PS we need to remember to remove this specific conditional instead of just intuitively pointing the via at the HS.

I'm not entirely sure that Synapse will be happy with trying to contact itself over federation because that's a pretty weird thing to do in Matrix. Any Synapse support for being a policy server (either natively or via modules) would involve a performant hook anyway, and would be inserted here to short-circuit the federation call.

Removing the check would also obviously slow down message sending in rooms we know are misconfigured (not pointing to a real policy server). With the current highly experimental architecture we also do not expect that pre-existing servers will be policy servers, and operators will need to pick a dedicated domain name without human users on that server. This will probably change down the line.

I'll add some text to the docstring to reduce the chances of forgetting in the meantime. Hopefully when someone writes a test for "can Synapse be a policy server", they'll worst case find the if statement by stepping through the call path.

I also ended up adding a test for this, so the failure should be even more clear if we do add such functionality

synapse/handlers/room_policy.py

…00-check

This reverts commit 70987fa.

synapse/federation/transport/client.py

synapse/federation/federation_base.py

synapse/federation/federation_client.py

synapse/handlers/room_policy.py

tests/handlers/test_room_policy.py

Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>

turt2live · 2025-05-21T20:49:06Z

I don't know what I did to make the sytests angry aside from merging in develop

devonh · 2025-05-21T21:48:42Z

I don't know what I did to make the sytests angry aside from merging in develop

Seems to have just been flakey tests sadly

devonh

I think this is good to merge now. It follows the same patterns as the spam checker code where applicable so has pretty strong precedence. The new tests are also very appreciated.

@kfiven

Builds on NetBSD 10 amd64, and builds/tests-ok on NetBSD 9 amd64 using dependencies from 2025Q2. NB: A security update to synapse is scheduled for July 22. Consult https://matrix.org/blog/2025/07/security-predisclosure/ for further details. Those running synapse in production may wish to update to 1.134.0 to reduce the magnitude of change when updating to the July 22 version (although that will be a big update regardless). Note that the usual pkgsrc pre-commit test is upgrading from the current pkgsrc version and briefly checking operation. Therefore, not upgrading has a theoretical risk of encountering a 1.127.1 to 1.135.0 update bug when 1.127.1 to 134.0 and 1.134.0 to 1.135.0 are ok. # Synapse 1.134.0 (2025-07-15) - Support for [MSC4235](matrix-org/matrix-spec-proposals#4235): `via` query param for hierarchy endpoint. Contributed by Krishan (@kfiven). ([\#18070](element-hq/synapse#18070)) - Add `forget_forced_upon_leave` capability as per [MSC4267](matrix-org/matrix-spec-proposals#4267). ([\#18196](element-hq/synapse#18196)) - Add `federated_user_may_invite` spam checker callback which receives the entire invite event. Contributed by @tulir @ Beeper. ([\#18241](element-hq/synapse#18241)) # Synapse 1.133.0 (2025-07-01) - Add support for the [MSC4260 user report API](matrix-org/matrix-spec-proposals#4260). ([\#18120](element-hq/synapse#18120)) # Synapse 1.132.0 (2025-06-17) - Add support for [MSC4155](matrix-org/matrix-spec-proposals#4155) Invite Filtering. ([\#18288](element-hq/synapse#18288)) - Add experimental `user_may_send_state_event` module API callback. ([\#18455](element-hq/synapse#18455)) - Add experimental `get_media_config_for_user` and `is_user_allowed_to_upload_media_of_size` module API callbacks that allow overriding of media repository maximum upload size. ([\#18457](element-hq/synapse#18457)) - Add experimental `get_ratelimit_override_for_user` module API callback that allows overriding of per-user ratelimits. ([\#18458](element-hq/synapse#18458)) - Pass `room_config` argument to `user_may_create_room` spam checker module callback. ([\#18486](element-hq/synapse#18486)) - Support configuration of default and extra user types. ([\#18456](element-hq/synapse#18456)) - Successful requests to `/_matrix/app/v1/ping` will now force Synapse to reattempt delivering transactions to appservices. ([\#18521](element-hq/synapse#18521)) - Support the import of the `RatelimitOverride` type from `synapse.module_api` in modules and rename `messages_per_second` to `per_second`. ([\#18513](element-hq/synapse#18513)) # Synapse 1.131.0 (2025-06-03) - Add `msc4263_limit_key_queries_to_users_who_share_rooms` config option as per [MSC4263](matrix-org/matrix-spec-proposals#4263). ([\#18180](element-hq/synapse#18180)) - Add option to allow registrations that begin with `_`. Contributed by `_` (@hex5f). ([\#18262](element-hq/synapse#18262)) - Include room ID in response to the [Room Deletion Status Admin API](https://element-hq.github.io/synapse/latest/admin_api/rooms.html#status-of-deleting-rooms). ([\#18318](element-hq/synapse#18318)) - Add support for calling Policy Servers ([MSC4284](matrix-org/matrix-spec-proposals#4284)) to mark events as spam. ([\#18387](element-hq/synapse#18387)) # Synapse 1.130.0 (2025-05-20) - Add an Admin API endpoint `GET /_synapse/admin/v1/scheduled_tasks` to fetch scheduled tasks. ([\#18214](element-hq/synapse#18214)) - Add config option `user_directory.exclude_remote_users` which, when enabled, excludes remote users from user directory search results. ([\#18300](element-hq/synapse#18300)) - Add support for handling `GET /devices/` on workers. ([\#18355](element-hq/synapse#18355)) # Synapse 1.129.0 (2025-05-06) - Add `passthrough_authorization_parameters` in OIDC configuration to allow passing parameters to the authorization grant URL. ([\#18232](element-hq/synapse#18232)) - Add `total_event_count`, `total_message_count`, and `total_e2ee_event_count` fields to the homeserver usage statistics. ([\#18260](element-hq/synapse#18260)) # Synapse 1.128.0 (2025-04-08) - Add an access token introspection cache to make Matrix Authentication Service integration ([MSC3861](matrix-org/matrix-spec-proposals#3861)) more efficient. ([\#18231](element-hq/synapse#18231)) - Add background job to clear unreferenced state groups. ([\#18254](element-hq/synapse#18254)) - Hashes of media files are now tracked by Synapse. Media quarantines will now apply to all files with the same hash. ([\#18277](element-hq/synapse#18277), [\#18302](element-hq/synapse#18302), [\#18296](element-hq/synapse#18296))

@kfiven

chat/matrix-synapse: Update package in anticipation of security fix Revisions pulled up: - chat/matrix-synapse/Makefile 1.112 - chat/matrix-synapse/PLIST 1.59 - chat/matrix-synapse/cargo-depends.mk 1.27 - chat/matrix-synapse/distinfo 1.80 --- Module Name: pkgsrc Committed By: gdt Date: Thu Jul 17 11:24:44 UTC 2025 Modified Files: pkgsrc/chat/matrix-synapse: Makefile PLIST cargo-depends.mk distinfo Log Message: chat/matrix-synapse: Update to 1.134.0 Builds on NetBSD 10 amd64, and builds/tests-ok on NetBSD 9 amd64 using dependencies from 2025Q2. NB: A security update to synapse is scheduled for July 22. Consult https://matrix.org/blog/2025/07/security-predisclosure/ for further details. Those running synapse in production may wish to update to 1.134.0 to reduce the magnitude of change when updating to the July 22 version (although that will be a big update regardless). Note that the usual pkgsrc pre-commit test is upgrading from the current pkgsrc version and briefly checking operation. Therefore, not upgrading has a theoretical risk of encountering a 1.127.1 to 1.135.0 update bug when 1.127.1 to 134.0 and 1.134.0 to 1.135.0 are ok. # Synapse 1.134.0 (2025-07-15) - Support for [MSC4235](matrix-org/matrix-spec-proposals#4235): `via` query param for hierarchy endpoint. Contributed by Krishan (@kfiven). ([\#18070](element-hq/synapse#18070)) - Add `forget_forced_upon_leave` capability as per [MSC4267](matrix-org/matrix-spec-proposals#4267). ([\#18196](element-hq/synapse#18196)) - Add `federated_user_may_invite` spam checker callback which receives the entire invite event. Contributed by @tulir @ Beeper. ([\#18241](element-hq/synapse#18241)) # Synapse 1.133.0 (2025-07-01) - Add support for the [MSC4260 user report API](matrix-org/matrix-spec-proposals#4260). ([\#18120](element-hq/synapse#18120)) # Synapse 1.132.0 (2025-06-17) - Add support for [MSC4155](matrix-org/matrix-spec-proposals#4155) Invite Filtering. ([\#18288](element-hq/synapse#18288)) - Add experimental `user_may_send_state_event` module API callback. ([\#18455](element-hq/synapse#18455)) - Add experimental `get_media_config_for_user` and `is_user_allowed_to_upload_media_of_size` module API callbacks that allow overriding of media repository maximum upload size. ([\#18457](element-hq/synapse#18457)) - Add experimental `get_ratelimit_override_for_user` module API callback that allows overriding of per-user ratelimits. ([\#18458](element-hq/synapse#18458)) - Pass `room_config` argument to `user_may_create_room` spam checker module callback. ([\#18486](element-hq/synapse#18486)) - Support configuration of default and extra user types. ([\#18456](element-hq/synapse#18456)) - Successful requests to `/_matrix/app/v1/ping` will now force Synapse to reattempt delivering transactions to appservices. ([\#18521](element-hq/synapse#18521)) - Support the import of the `RatelimitOverride` type from `synapse.module_api` in modules and rename `messages_per_second` to `per_second`. ([\#18513](element-hq/synapse#18513)) # Synapse 1.131.0 (2025-06-03) - Add `msc4263_limit_key_queries_to_users_who_share_rooms` config option as per [MSC4263](matrix-org/matrix-spec-proposals#4263). ([\#18180](element-hq/synapse#18180)) - Add option to allow registrations that begin with `_`. Contributed by `_` (@hex5f). ([\#18262](element-hq/synapse#18262)) - Include room ID in response to the [Room Deletion Status Admin API](https://element-hq.github.io/synapse/latest/admin_api/rooms.html#status-of-deleting-rooms). ([\#18318](element-hq/synapse#18318)) - Add support for calling Policy Servers ([MSC4284](matrix-org/matrix-spec-proposals#4284)) to mark events as spam. ([\#18387](element-hq/synapse#18387)) # Synapse 1.130.0 (2025-05-20) - Add an Admin API endpoint `GET /_synapse/admin/v1/scheduled_tasks` to fetch scheduled tasks. ([\#18214](element-hq/synapse#18214)) - Add config option `user_directory.exclude_remote_users` which, when enabled, excludes remote users from user directory search results. ([\#18300](element-hq/synapse#18300)) - Add support for handling `GET /devices/` on workers. ([\#18355](element-hq/synapse#18355)) # Synapse 1.129.0 (2025-05-06) - Add `passthrough_authorization_parameters` in OIDC configuration to allow passing parameters to the authorization grant URL. ([\#18232](element-hq/synapse#18232)) - Add `total_event_count`, `total_message_count`, and `total_e2ee_event_count` fields to the homeserver usage statistics. ([\#18260](element-hq/synapse#18260)) # Synapse 1.128.0 (2025-04-08) - Add an access token introspection cache to make Matrix Authentication Service integration ([MSC3861](matrix-org/matrix-spec-proposals#3861)) more efficient. ([\#18231](element-hq/synapse#18231)) - Add background job to clear unreferenced state groups. ([\#18254](element-hq/synapse#18254)) - Hashes of media files are now tracked by Synapse. Media quarantines will now apply to all files with the same hash. ([\#18277](element-hq/synapse#18277), [\#18302](element-hq/synapse#18302), [\#18296](element-hq/synapse#18296))

turt2live and others added 11 commits May 2, 2025 13:01

Copy over core of module to Synapse

1cca9f2

Check policy server at same spot as spam checker

a34c7af

Merge branch 'develop' into travis/policyserv/00-check

3b4e7f9

First-pass linter appeasement

753ce5c

Merge branch 'develop' into travis/policyserv/00-check

d8141d8

Attempt to fix linting

0cd6562

Empty commit to kick CI

f71c846

add changelog

bc09886

Add some rationale commentary

fd78613

Improve docs

5984463

Hopefully avoid cyclic dependency

a27a539

turt2live and others added 2 commits May 15, 2025 05:05

Attempt to fix linting

08341e8

Empty commit to kick CI

dfaf62a

turt2live marked this pull request as ready for review May 15, 2025 05:06

turt2live requested a review from a team as a code owner May 15, 2025 05:06

Optional?

5af6638

kegsay reviewed May 15, 2025

View reviewed changes

turt2live and others added 10 commits May 15, 2025 12:26

Improve docs

1a30ff4

Merge remote-tracking branch 'origin/develop' into travis/policyserv/…

aad740a

…00-check

Use parse_and_validate_server_name

7f2d834

Add logging for live debugging

70987fa

it helps to await

f35584e

Revert "Add logging for live debugging"

959feb9

This reverts commit 70987fa.

Add untested tests

7118d4a

Attempt to fix linting

eedefc7

Empty commit to kick CI

75826a6

The CI is more intelligent than the IDE

402fc80

turt2live requested a review from kegsay May 15, 2025 20:08

turt2live mentioned this pull request May 20, 2025

[WIP] MSC4284: Policy Servers matrix-org/matrix-spec-proposals#4284

Draft

devonh requested changes May 20, 2025

View reviewed changes

turt2live and others added 3 commits May 21, 2025 13:27

Apply suggestions from code review

ef119d6

Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>

Fix docs

71bec51

Merge branch 'develop' into travis/policyserv/00-check

b676fb2

turt2live requested a review from devonh May 21, 2025 19:42

devonh approved these changes May 21, 2025

View reviewed changes

devonh merged commit b7d4841 into develop May 21, 2025
39 checks passed

devonh deleted the travis/policyserv/00-check branch May 21, 2025 22:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Policy server part 1: Actually call the policy server #18387

Policy server part 1: Actually call the policy server #18387

Uh oh!

turt2live commented May 2, 2025 •

edited

Loading

Uh oh!

turt2live commented May 15, 2025

Uh oh!

kegsay left a comment

Uh oh!

kegsay May 15, 2025

Uh oh!

erikjohnston May 15, 2025

Uh oh!

turt2live May 15, 2025

Uh oh!

kegsay May 15, 2025

Uh oh!

turt2live May 15, 2025

Uh oh!

turt2live May 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turt2live commented May 21, 2025

Uh oh!

devonh commented May 21, 2025

Uh oh!

devonh left a comment

Uh oh!

Uh oh!

Uh oh!

Policy server part 1: Actually call the policy server #18387

Policy server part 1: Actually call the policy server #18387

Uh oh!

Conversation

turt2live commented May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Checklist

Uh oh!

turt2live commented May 15, 2025

Uh oh!

kegsay left a comment

Choose a reason for hiding this comment

Uh oh!

kegsay May 15, 2025

Choose a reason for hiding this comment

Uh oh!

erikjohnston May 15, 2025

Choose a reason for hiding this comment

Uh oh!

turt2live May 15, 2025

Choose a reason for hiding this comment

Uh oh!

kegsay May 15, 2025

Choose a reason for hiding this comment

Uh oh!

turt2live May 15, 2025

Choose a reason for hiding this comment

Uh oh!

turt2live May 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

turt2live commented May 21, 2025

Uh oh!

devonh commented May 21, 2025

Uh oh!

devonh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

turt2live commented May 2, 2025 •

edited

Loading