Merge tag 'v1.42.0rc1' into develop

Synapse 1.42.0rc1 (2021-09-01)
==============================

Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).

Features
--------

- Add support for [MSC3231](matrix-org/matrix-spec-proposals#3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](#10142))
- Add support for [MSC3283](matrix-org/matrix-spec-proposals#3283): Expose `enable_set_displayname` in capabilities. ([\#10452](#10452))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](#10524))
- Add pagination to the spaces summary based on updates to [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\#10613](#10613), [\#10725](#10725))

Bugfixes
--------

- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](#10232))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](#10561))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](#10581), [\#10734](#10734))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](#10593))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](#10644))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](#10654))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](#10677))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](#10679))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](#10684))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](#10703))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](#10713))

Improved Documentation
----------------------

- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](#10192))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](#10595))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](#10639))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](#10708))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](#10711))

Deprecations and Removals
-------------------------

- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](#8830))

Internal Changes
----------------

- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](#10608))
- Clean up some of the federation event authentication code for clarity. ([\#10614](#10614), [\#10615](#10615), [\#10624](#10624), [\#10640](#10640))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](#10621))
- Remove not needed database updates in modify user admin API. ([\#10627](#10627))
- Convert room member storage tuples to `attrs` classes. ([\#10629](#10629), [\#10642](#10642))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](#10630))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](#10645))
- Improve the performance of the `/hierarchy` API (from [MSC2946](matrix-org/matrix-spec-proposals#2946)) by caching responses received over federation. ([\#10647](#10647))
- Run a nightly CI build against Twisted trunk. ([\#10651](#10651), [\#10672](#10672))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](#10662))
- Simplify tests for device admin rest API. ([\#10664](#10664))
- Add missing type hints to REST servlets. ([\#10665](#10665), [\#10666](#10666), [\#10674](#10674))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](#10667))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](#10686))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](#10692))
- Remove unused `compare_digest` function. ([\#10706](#10706))

Author: reivilibre

CHANGES.md

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.42.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.42.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 01 Sep 2021 11:37:48 +0100
+
 matrix-synapse-py3 (1.41.1) stable; urgency=high
 
   * New synapse release 1.41.1.

changelog.d/10142.feature

@@ -47,7 +47,7 @@
 except ImportError:
     pass
 
-__version__ = "1.41.1"
+__version__ = "1.42.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

changelog.d/10192.doc

@@ -430,10 +430,11 @@ def _delete_pushers(txn) -> int:
             """
 
             txn.execute(sql, (last_pusher, batch_size))
+            rows = txn.fetchall()
 
             last = None
             num_deleted = 0
-            for row in txn:
+            for row in rows:
                 last = row[0]
                 num_deleted += 1
                 self.db_pool.simple_delete_txn(

changelog.d/10232.bugfix

@@ -379,6 +379,50 @@ def test_no_email_sent_after_removed(self):
         pushers = list(pushers)
         self.assertEqual(len(pushers), 0)
 
+    def test_remove_unlinked_pushers_background_job(self):
+        """Checks that all existing pushers associated with unlinked email addresses are removed
+        upon running the remove_deleted_email_pushers background update.
+        """
+        # disassociate the user's email address manually (without deleting the pusher).
+        # This resembles the old behaviour, which the background update below is intended
+        # to clean up.
+        self.get_success(
+            self.hs.get_datastore().user_delete_threepid(
+                self.user_id, "email", "a@example.com"
+            )
+        )
+
+        # Run the "remove_deleted_email_pushers" background job
+        self.get_success(
+            self.hs.get_datastore().db_pool.simple_insert(
+                table="background_updates",
+                values={
+                    "update_name": "remove_deleted_email_pushers",
+                    "progress_json": "{}",
+                    "depends_on": None,
+                },
+            )
+        )
+
+        # ... and tell the DataStore that it hasn't finished all updates yet
+        self.hs.get_datastore().db_pool.updates._all_done = False
+
+        # Now let's actually drive the updates to completion
+        while not self.get_success(
+            self.hs.get_datastore().db_pool.updates.has_completed_background_updates()
+        ):
+            self.get_success(
+                self.hs.get_datastore().db_pool.updates.do_next_background_update(100),
+                by=0.1,
+            )
+
+        # Check that all pushers with unlinked addresses were deleted
+        pushers = self.get_success(
+            self.hs.get_datastore().get_pushers_by({"user_name": self.user_id})
+        )
+        pushers = list(pushers)
+        self.assertEqual(len(pushers), 0)
+
     def _check_for_mail(self) -> Tuple[Sequence, Dict]:
         """
         Assert that synapse sent off exactly one email notification.