After updating the Synapse server to version 1.109.0~rc1, new messages do not leave / arrive.

[ELForcer]

Description

I updated the package using apt, after which after a while I noticed high CPU consumption for Synapse processes, and then that messages were not actually sent to the recipients. Restarting the service didn't help.
At the moment, rolling back to package version 1.108.0 helped.

Steps to reproduce

sudo apt update
sudo apt upgrade

Homeserver

homeserver

Synapse Version

1.109.0rc1

Installation Method

Debian packages from packages.matrix.org

Database

PostgreSQL 16

Workers

I don't know

Platform

System:
  Host: userver Kernel: 6.1.0-21-amd64 arch: x86_64 bits: 64 Console: pty pts/8 Distro: Debian
    GNU/Linux 12 (bookworm)
CPU:
  Info: quad core model: Intel Xeon E5620 bits: 64 type: MT MCP cache: L2: 1024 KiB
  Speed (MHz): avg: 1600 min/max: 1600/2400 cores: 1: 1600 2: 1600 3: 1600 4: 1600 5: 1600
    6: 1600 7: 1600 8: 1600
RAID:
  Hardware-1: Hewlett-Packard Smart Array G6 controllers driver: hpsa
Drives:
  Local Storage: total: 476.89 GiB used: 287.85 GiB (60.4%)
  ID-1: /dev/sda model: LOGICAL VOLUME size: 476.89 GiB
Partition:
  ID-1: / size: 72.78 GiB used: 24.99 GiB (34.3%) fs: ext4 dev: /dev/sda1
  ID-2: /home size: 365.61 GiB used: 258.44 GiB (70.7%) fs: ext4 dev: /dev/sda5
  ID-3: /var/log size: 29.21 GiB used: 4.42 GiB (15.1%) fs: ext4 dev: /dev/sda6
Swap:
  Alert: No swap data was found.
Info:
  Processes: 406 Uptime: 27d 3h 41m Memory: 39.16 GiB used: 6.16 GiB (15.7%) Init: systemd
  target: graphical (5) Shell: Bash inxi: 3.3.26

Configuration

No response

Relevant log output

2024-06-06 15:38:43,354 - root - 355 - WARNING - main - ***** STARTING SERVER *****
2024-06-06 15:38:43,354 - root - 356 - WARNING - main - Server /opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/app/homeserver.py version 1.109.0rc1
2024-06-06 15:38:43,354 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc
2024-06-06 15:38:43,354 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
2024-06-06 15:38:43,873 - synapse.metrics.background_process_metrics - 253 - ERROR - _handle_new_device_update_async-0 - Background process '_handle_new_device_update_async' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/metrics/background_process_metrics.py", line 251, in run
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/handlers/device.py", line 873, in _handle_new_device_update_async
    assert max_stream_id == stream_id
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError
2024-06-06 15:38:54,016 - synapse.handlers.sync - 523 - WARNING - GET-2 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:54,375 - synapse.handlers.sync - 523 - WARNING - GET-7 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:54,378 - synapse.handlers.sync - 523 - WARNING - GET-8 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:54,628 - synapse.handlers.sync - 523 - WARNING - GET-11 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:54,773 - synapse.handlers.sync - 523 - WARNING - GET-15 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:54,806 - synapse.handlers.sync - 523 - WARNING - GET-17 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,049 - synapse.handlers.sync - 523 - WARNING - GET-20 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,341 - synapse.handlers.sync - 523 - WARNING - GET-26 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,348 - synapse.handlers.sync - 523 - WARNING - GET-27 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,621 - synapse.handlers.sync - 523 - WARNING - GET-30 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,755 - synapse.handlers.sync - 523 - WARNING - GET-35 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,781 - synapse.handlers.sync - 523 - WARNING - GET-36 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:55,893 - synapse.handlers.sync - 523 - WARNING - GET-38 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:56,025 - synapse.handlers.sync - 523 - WARNING - GET-41 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:57,191 - synapse.handlers.sync - 523 - WARNING - GET-44 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:57,535 - synapse.handlers.sync - 523 - WARNING - GET-57 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:57,637 - synapse.handlers.sync - 523 - WARNING - GET-59 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:58,015 - synapse.handlers.sync - 523 - WARNING - GET-62 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:58,651 - synapse.handlers.sync - 523 - WARNING - GET-67 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:58,820 - synapse.handlers.sync - 523 - WARNING - GET-70 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:59,159 - synapse.handlers.sync - 523 - WARNING - GET-73 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:59,321 - synapse.handlers.sync - 523 - WARNING - GET-76 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:59,816 - synapse.handlers.sync - 523 - WARNING - GET-81 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:59,885 - synapse.handlers.sync - 523 - WARNING - GET-83 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:38:59,953 - synapse.handlers.sync - 523 - WARNING - GET-85 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:00,113 - synapse.handlers.sync - 523 - WARNING - GET-88 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:00,190 - synapse.handlers.sync - 523 - WARNING - GET-92 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:00,314 - synapse.handlers.sync - 523 - WARNING - GET-94 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:00,570 - synapse.handlers.sync - 523 - WARNING - GET-97 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:01,126 - synapse.handlers.sync - 523 - WARNING - GET-100 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:01,308 - synapse.handlers.sync - 523 - WARNING - GET-103 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:02,060 - synapse.handlers.sync - 523 - WARNING - GET-111 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:02,786 - synapse.handlers.sync - 523 - WARNING - GET-118 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:02,971 - synapse.handlers.sync - 523 - WARNING - GET-121 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,157 - synapse.handlers.sync - 523 - WARNING - GET-124 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,496 - synapse.handlers.sync - 523 - WARNING - GET-129 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,509 - synapse.handlers.sync - 523 - WARNING - GET-131 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,836 - synapse.handlers.sync - 523 - WARNING - GET-138 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,844 - synapse.handlers.sync - 523 - WARNING - GET-139 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:04,855 - synapse.handlers.sync - 523 - WARNING - GET-143 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,172 - synapse.handlers.sync - 523 - WARNING - GET-150 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,233 - synapse.handlers.sync - 523 - WARNING - GET-134 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,498 - synapse.handlers.sync - 523 - WARNING - GET-156 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,499 - synapse.handlers.sync - 523 - WARNING - GET-158 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,815 - synapse.handlers.sync - 523 - WARNING - GET-163 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:05,851 - synapse.handlers.sync - 523 - WARNING - GET-166 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:06,045 - synapse.handlers.sync - 523 - WARNING - GET-173 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:06,061 - synapse.handlers.sync - 523 - WARNING - GET-174 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:06,156 - synapse.handlers.sync - 523 - WARNING - GET-177 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:07,239 - synapse.handlers.sync - 523 - WARNING - GET-180 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:07,603 - synapse.handlers.sync - 523 - WARNING - GET-184 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:07,892 - synapse.handlers.sync - 523 - WARNING - GET-188 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:08,179 - synapse.handlers.sync - 523 - WARNING - GET-191 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:08,695 - synapse.handlers.sync - 523 - WARNING - GET-194 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:08,860 - synapse.handlers.sync - 523 - WARNING - GET-198 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:09,287 - synapse.handlers.sync - 523 - WARNING - GET-203 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:09,462 - synapse.handlers.sync - 523 - WARNING - GET-207 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:09,931 - synapse.handlers.sync - 523 - WARNING - GET-210 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,025 - synapse.handlers.sync - 523 - WARNING - GET-212 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,127 - synapse.handlers.sync - 523 - WARNING - GET-214 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,233 - synapse.handlers.sync - 523 - WARNING - GET-218 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,256 - synapse.handlers.sync - 523 - WARNING - GET-221 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,517 - synapse.handlers.sync - 523 - WARNING - GET-224 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:10,705 - synapse.handlers.sync - 523 - WARNING - GET-227 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:11,311 - synapse.handlers.sync - 523 - WARNING - GET-233 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:11,413 - synapse.handlers.sync - 523 - WARNING - GET-236 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:12,225 - synapse.handlers.sync - 523 - WARNING - GET-238 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:12,846 - synapse.handlers.sync - 523 - WARNING - GET-240 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:13,089 - synapse.handlers.sync - 523 - WARNING - GET-242 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:39:15,318 - synapse.handlers.sync - 523 - WARNING - GET-399 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:14,381 - synapse.handlers.sync - 523 - WARNING - GET-78123 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:14,613 - synapse.handlers.sync - 523 - WARNING - GET-78271 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:14,616 - synapse.handlers.sync - 523 - WARNING - GET-78272 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:14,927 - synapse.handlers.sync - 523 - WARNING - GET-78455 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:14,981 - synapse.handlers.sync - 523 - WARNING - GET-78493 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,052 - synapse.handlers.sync - 523 - WARNING - GET-78526 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,310 - synapse.handlers.sync - 523 - WARNING - GET-78681 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,652 - synapse.handlers.sync - 523 - WARNING - GET-78883 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,691 - synapse.handlers.sync - 523 - WARNING - GET-78912 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,928 - synapse.handlers.sync - 523 - WARNING - GET-79049 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:15,966 - synapse.handlers.sync - 523 - WARNING - GET-79072 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:16,157 - synapse.handlers.sync - 523 - WARNING - GET-79185 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:16,238 - synapse.handlers.sync - 523 - WARNING - GET-79225 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:16,302 - synapse.handlers.sync - 523 - WARNING - GET-79262 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:17,284 - synapse.handlers.sync - 523 - WARNING - GET-79868 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:17,643 - synapse.handlers.sync - 523 - WARNING - GET-80090 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:18,043 - synapse.handlers.sync - 523 - WARNING - GET-80326 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:18,225 - synapse.handlers.sync - 523 - WARNING - GET-80440 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:18,752 - synapse.handlers.sync - 523 - WARNING - GET-80750 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:18,936 - synapse.handlers.sync - 523 - WARNING - GET-80855 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:19,484 - synapse.handlers.sync - 523 - WARNING - GET-81182 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:19,583 - synapse.handlers.sync - 523 - WARNING - GET-81242 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:19,989 - synapse.handlers.sync - 523 - WARNING - GET-81497 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,193 - synapse.handlers.sync - 523 - WARNING - GET-81603 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,193 - synapse.handlers.sync - 523 - WARNING - GET-81617 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,274 - synapse.handlers.sync - 523 - WARNING - GET-81666 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,385 - synapse.handlers.sync - 523 - WARNING - GET-81705 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,714 - synapse.handlers.sync - 523 - WARNING - GET-81843 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:20,837 - synapse.handlers.sync - 523 - WARNING - GET-81885 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:21,393 - synapse.handlers.sync - 523 - WARNING - GET-82096 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:21,518 - synapse.handlers.sync - 523 - WARNING - GET-82131 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:22,338 - synapse.handlers.sync - 523 - WARNING - GET-82375 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:22,979 - synapse.handlers.sync - 523 - WARNING - GET-82557 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:23,224 - synapse.handlers.sync - 523 - WARNING - GET-82584 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:41:25,447 - synapse.handlers.sync - 523 - WARNING - GET-82894 - Timed out waiting for worker to catch up. Returning empty response
2024-06-06 15:43:24,558 - synapse.handlers.sync - 523 - WARNING - GET-153624 - Timed out waiting for worker to catch up. Returning empty response
...

Anything else that would be useful to know?

No response

[anoadragon453]

Hi @ELForcer, thanks for your report.

Do you have any workers configured? If you don't know, it's likely you don't.

For developers, here's the relevant bit of the code:

synapse/synapse/handlers/device.py

Lines 863 to 874 in b71d277

	# Advance `(stream_id, room_id)`.
	# `max_stream_id` comes from *before* the query for unconverted
	# rows, which means that any unconverted rows must have a larger
	# stream ID.
	if max_stream_id > stream_id:
	stream_id, room_id = max_stream_id, ""
	await self.store.set_device_change_last_converted_pos(
	stream_id, room_id
	)
	else:
	assert max_stream_id == stream_id
	# Avoid moving `room_id` backwards.

#17211 may be a possible contender for the regression, as it's only been introduced in v1.109.0rc1 as touches nearby code (outbound device pokes).

[anoadragon453]

@wrjlewis mentioned that a possible workaround is to run:

SELECT setval('device_lists_sequence', (
    SELECT GREATEST(MAX(stream_id), 0) FROM device_lists_stream
));

And then start the server on v1.109.0rc1. However the code should still be fixed.

Edit: See the below comment for warnings on running this query.

[erikjohnston]

@anoadragon453 That SQL should only be run if Synapse prompts you to do so, and while Synapse is offline. Moving streams can be quite dangerous if you get it wrong.

I think this an edge case with #17229, where we didn't include device_lists_changes_converted_stream_position table in the new stream ID generator. I think the easiest fix is to just skip the assert.

[erikjohnston]

Actually, I've changed tack, but PR up at #17292

[sandhose]

@ELForcer This should have been fixed in v1.109.0-rc2. Could you try upgrading and report back?

[ELForcer]

Hello. It's gotten worse at the moment. The matrix-synapse service no longer starts.

user@userver:/var/log/matrix-synapse$ systemctl status matrix-synapse
● matrix-synapse.service - Synapse Matrix homeserver
     Loaded: loaded (/lib/systemd/system/matrix-synapse.service; enabled; preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2024-06-11 21:44:12 +04; 2s ago
    Process: 428001 ExecStartPre=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ --generate-keys (code=exited, status=0/SUCCESS)
    Process: 428006 ExecStart=/opt/venvs/matrix-synapse/bin/python -m synapse.app.homeserver --config-path=/etc/matrix-synapse/homeserver.yaml --config-path=/etc/matrix-synapse/conf.d/ (code=exited, status=1/FAILURE)
   Main PID: 428006 (code=exited, status=1/FAILURE)
        CPU: 4.496s

Log:

2024-06-11 21:43:09,382 - root - 355 - WARNING - main - ***** STARTING SERVER *****
2024-06-11 21:43:09,382 - root - 356 - WARNING - main - Server /opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/app/homeserver.py version 1.109.0rc2
2024-06-11 21:43:09,383 - root - 361 - WARNING - main - Copyright (c) 2023 New Vector, Inc
2024-06-11 21:43:09,383 - root - 362 - WARNING - main - Licensed under the AGPL 3.0 license. Website: https://github.com/element-hq/synapse
2024-06-11 21:43:09,638 - synapse.storage.util.sequence - 180 - WARNING - main - Postgres sequence push_rules_stream_sequence is behind table push_rules_stream: 73 < 77
2024-06-11 21:43:09,639 - synapse.app._base - 220 - ERROR - main - Exception during startup
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/app/homeserver.py", line 359, in setup
    hs.setup()
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/server.py", line 348, in setup
    self.datastores = Databases(self.DATASTORE_CLASS, self)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/__init__.py", line 100, in __init__
    main = main_store_class(database, db_conn, hs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/__init__.py", line 170, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/events_bg_updates.py", line 107, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/experimental_features.py", line 40, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/devices.py", line 1747, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/devices.py", line 95, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/roommember.py", line 1549, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/room.py", line 2207, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/room.py", line 1750, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/room.py", line 150, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/registration.py", line 2327, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/profile.py", line 44, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/presence.py", line 83, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/presence.py", line 64, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/transactions.py", line 81, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/state.py", line 751, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/state.py", line 109, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/state.py", line 549, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_federation.py", line 2064, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/media_repository.py", line 199, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/media_repository.py", line 112, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/filtering.py", line 49, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1896, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/metrics.py", line 75, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/event_push_actions.py", line 263, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/end_to_end_keys.py", line 1449, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/e2e_room_keys.py", line 60, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/search.py", line 435, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/search.py", line 144, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/account_data.py", line 68, in __init__
    super().__init__(database, db_conn, hs)
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/databases/main/push_rule.py", line 143, in __init__
    self._push_rules_stream_id_gen = MultiWriterIdGenerator(
                                     ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/util/id_generators.py", line 298, in __init__
    self._sequence_gen.check_consistency(
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/storage/util/sequence.py", line 187, in check_consistency
    raise IncorrectDatabaseSetup(
synapse.storage.engines._base.IncorrectDatabaseSetup: 
Postgres sequence 'push_rules_stream_sequence' is inconsistent with associated
table 'push_rules_stream'. This can happen if Synapse has been downgraded and
then upgraded again, or due to a bad migration.

To fix this error, shut down Synapse (including any and all workers)
and run the following SQL:

    SELECT setval('push_rules_stream_sequence', (
        SELECT GREATEST(MAX(stream_id), 0) FROM push_rules_stream
    ));

See docs/postgres.md for more information.

My understanding is that it cannot automatically perform the operation:

 SELECT setval('push_rules_stream_sequence', (
 SELECT GREATEST(MAX(stream_id), 0) FROM push_rules_stream
 ));

and this SQL query needs to be executed independently?

[ELForcer]

At the moment I have rolled back to package version 1.108.0.

[frebib]

+1, I am also hitting this Postgres sequence 'push_rules_stream_sequence' is inconsistent with associated table 'push_rules_stream'. This can happen if Synapse has been downgraded and then upgraded again, or due to a bad migration. error now after rolling between 1.108.0, 1.109.0rc1 and 1.109.0rc2 when I noticed that sync'ing stopped working after upgrading to 1.109.0rc2. Also have rolled back to 1.108.0 for now without running the query until we hear anything about potential next steps

[sandhose]

I was able to reproduce locally. Steps to reproduce:

• deploy 1.108.0
• change a push rule (like disable notifications in a room)
• deploy 1.109.0rc1
• change a push rule
• roll back to 1.108.0
• change a push rule
• deploy 1.109.0rc2, it refuses to start

Database looks like this:

synapse=# select stream_id, event_stream_ordering, instance_name from push_rules_stream;
 stream_id | event_stream_ordering | instance_name
-----------+-----------------------+---------------
         2 |                    10 |
         3 |                    12 |
         4 |                    13 | master
         5 |                    14 |

synapse=# select instance_name, stream_id from stream_positions where stream_name = 'push_rules_stream';
 instance_name | stream_id
---------------+-----------
 master        |         4

---

Even simpler to trigger:

• deploy v1.109.0rc2
• downgrade to v1.108.0
• change a push rule
• upgrade back to v1.109.0rc2, it should refuse to start

[sandhose]

I believe we fixed this, debs for v1.109.0rc3 with the fix should be available in a few hours. If it indeed fixes your issue, we should be able to release v1.109.0 tomorrow

[frebib]

rc3 works fine for me. Sync seems to work fine, too. Not sure what the weirdness I saw with rc1/rc2 was about.
Thanks :)