Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

synapse.api.errors.StoreError: 404: No row found during background process rotate_notifs #14454

Closed
lsrmg opened this issue Nov 15, 2022 · 25 comments
Labels
A-Push Issues related to push/notifications O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@lsrmg
Copy link

lsrmg commented Nov 15, 2022

Description

For some versions of matrix-synapse we're getting the following error in our homeserver-logs. We didn't noticed any issues while using the service at all, everythin was running fine except for the presence-status since the introduction of incremental /sync requests some versions ago.

With the new synapse version 1.71.0 the /sync requests from all clients keep failing with the message "No row found".
It seems to me that there are some null-values in the database where they not ment to be.

Steps to reproduce

  • Updating to matrix-synapse-py3 1.71.0

Homeserver

on-premise/local network

Synapse Version

1.71.0

Installation Method

Debian packages from packages.matrix.org

Platform

Debian 11

Relevant log output

2022-11-11 20:15:50,596 - synapse.metrics.background_process_metrics - 242 - ERROR - rotate_notifs-18 - Background process 'rotate_notifs' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1052, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1097, in _handle_new_receipts_for_notifs_txn
    old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

Anything else that would be useful to know?

There are two issues like this in #14219 and #14300.

@clokep
Copy link
Member

clokep commented Nov 15, 2022

Do you have purging enabled on your homeserver?

@lsrmg
Copy link
Author

lsrmg commented Nov 15, 2022

No, I don't have retention/purging enabled.

@erikjohnston
Copy link
Member

@lsrmg that stack trace doesn't seem to match v1.71.0, in that the line numbers don't line up. Is it possible the running synapse was different from the one that was installed?

Secondly, that stack trace is for a background process, which while not ideal shouldn't cause problems with /sync. Could you post some logs (or send them to me at @erikj:jki.re) for the time period that /sync was 404ing for please?

@erikjohnston erikjohnston added X-Needs-Info This issue is blocked awaiting information from the reporter T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. labels Nov 16, 2022
@lsrmg
Copy link
Author

lsrmg commented Nov 16, 2022

Thank you for you quick answer! I send you the full logs to @erikj:jki.re.

You're right, I restored the working state and must have copied the wron snipped.

Today I installed 1.71.0 and secured the right log:

2022-11-16 20:54:59,691 - synapse.metrics.background_process_metrics - 244 - ERROR - rotate_notifs-3 - Background process 'rotate_notifs' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run
    return await func(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1296, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1341, in _handle_new_receipts_for_notifs_txn
    old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

And the error on the client-side (Element 1.11.14):
request: /_matrix/client/r0/sync?filter=0&timeout=0&since=s7426708_211201868_0_11098700_23391459_11895_341941_195689_0
answer: {"errcode":"M_UNKNOWN","error":"No row found"}

@lsrmg
Copy link
Author

lsrmg commented Nov 27, 2022

Same issue present in synapse 1.72 for me

@DMRobertson DMRobertson changed the title synapse.api.errors.StoreError: 404: No row found synapse.api.errors.StoreError: 404: No row found during background process rotate_notifs Nov 28, 2022
@DMRobertson DMRobertson changed the title synapse.api.errors.StoreError: 404: No row found during background process rotate_notifs synapse.api.errors.StoreError: 404: No row found during background process rotate_notifs Nov 28, 2022
@H-Shay
Copy link
Contributor

H-Shay commented Nov 29, 2022

@lsrmg do you have logs?

@lsrmg
Copy link
Author

lsrmg commented Nov 29, 2022

@lsrmg do you have logs?

@H-Shay sorry, but I only have the logs provided above.

@H-Shay
Copy link
Contributor

H-Shay commented Dec 1, 2022

I took a look at the logs you sent my teammate and while I do see the error from the background process, I am not seeing anything related to /sync. Can you provide logs for the time that you send a few requests to /sync and see them fail? If possible, if you could provide logs with the logging level set to debug that would help immensely. Feel free to send them to @shayshay:matrix.org.

@H-Shay
Copy link
Contributor

H-Shay commented Dec 2, 2022

Looking at the debug logs provided, I do see /sync requests failing:

2022-12-02 00:32:39,162 - synapse.access.http.8008 - 460 - INFO - GET-564 - **anon-IP** - 8008 - {@xxxx:xxxxx-xxxxx} Processed request: 0.003sec/0.001sec (0.002sec, 0.000sec) (0.000sec/0.000sec/0) 46B 404 "GET /_matrix/client/r0/sync?filter=2&timeout=0&since=s7583202_215985817_237228_11477561_23942297_12075_355636_197795_0 HTTP/1.0" "browser" [0 dbevts]

Looking closer, it looks like /sync calls unread_notifs_for_room_id which calls get_unread_event_push_actions_by_room_for_user, and one of the queries contained within is failing.

It appears that the query that is failing is

2022-12-02 00:32:38,434 - synapse.storage.SQL - 416 - DEBUG - GET-534 - [SQL] {get_unread_event_push_actions_by_room-286} SELECT stream_ordering FROM event_push_summary_stream_ordering

with the message

2022-12-02 00:32:38,450 - synapse.storage.txn - 786 - DEBUG - GET-534 - [TXN FAIL] {get_unread_event_push_actions_by_room-286} 404: No row found

which afaict originates from this chunk of code here:

# The (inclusive) event stream ordering that was previously summarised.
        rotated_upto_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
            txn,
            table="event_push_summary_stream_ordering",
            keyvalues={},
            retcol="stream_ordering",
        )

which seems to have been added in #13877. @clokep, I think this is part of the threading work? You are more familiar with it than I, do you have any thoughts on what might be the issue?

@H-Shay
Copy link
Contributor

H-Shay commented Dec 2, 2022

Also just noting that the debug logs I looked at were from Synapse v1.72.0, not 1.71.0.

@clokep
Copy link
Member

clokep commented Dec 5, 2022

I think this is part of the threading work?

It was not part of threading -- this process existed earlier. (Threading just made it a bit more complicated!)

do you have any thoughts on what might be the issue?

I've only seen this be an issue when message retention is enabled. I'm not sure beyond that.

@reivilibre
Copy link
Contributor

I've stared at this a bit and can't see why the row in event_push_summary_stream_ordering would go missing — we don't appear to delete it anywhere, for example.

@lsrmg which database engine are you using (Postgres or SQLite)? Have you done anything 'unusual' to the database at some point in time (e.g. restored from a backup; ported from SQLite to Postgres using the portdb script; etc).

Please would you be able to run

SELECT * FROM applied_schema_deltas ORDER BY file;

for me?

.

@lsrmg
Copy link
Author

lsrmg commented Dec 8, 2022

@reivilibre I'm using Postgres version 13.8

At some point in time I migrated the DB between two servers (same synapse version) via db_dump and rebuild the indexes (from this).

Output from SELECT * FROM applied_schema_deltas ORDER BY file;:

17 drop_indexes.sql
17 server_keys.sql
17 user_threepids.sql
18 server_keys_bigger_ints.sql
19 event_index.sql
20 __pycache__
20 dummy.sql
20 pushers.py
21 end_to_end_keys.sql
21 receipts.sql
22 receipts_index.sql
22 user_threepids_unique.sql
23 drop_state_index.sql
24 stats_reporting.sql
25 00background_updates.sql
25 __pycache__
25 fts.py
25 guest_access.sql
25 history_visibility.sql
25 tags.sql

@reivilibre
Copy link
Contributor

Is that output being cut off for some reason (e.g. it's too tall)? I would expect many more entries, particularly recent ones.

I'm also surprised that none of the names contain a / in them. Odd.

One option is you can use PAGER=cat psql to connect to Postgres whilst disabling the pager, so you can get the full output.

@lsrmg
Copy link
Author

lsrmg commented Dec 8, 2022

You're right, it was cut off. Thanks for the tip!
I'm sorry for the inconvenience I didn't noticed.

 version |                               file                                
---------+-------------------------------------------------------------------
      17 | 17/drop_indexes.sql
      17 | 17/server_keys.sql
      17 | 17/user_threepids.sql
      18 | 18/server_keys_bigger_ints.sql
      19 | 19/event_index.sql
      20 | 20/__pycache__
      20 | 20/dummy.sql
      20 | 20/pushers.py
      21 | 21/end_to_end_keys.sql
      21 | 21/receipts.sql
      22 | 22/receipts_index.sql
      22 | 22/user_threepids_unique.sql
      23 | 23/drop_state_index.sql
      24 | 24/stats_reporting.sql
      25 | 25/00background_updates.sql
      25 | 25/__pycache__
      25 | 25/fts.py
      25 | 25/guest_access.sql
      25 | 25/history_visibility.sql
      25 | 25/tags.sql
      26 | 26/account_data.sql
      27 | 27/__pycache__
      27 | 27/account_data.sql
      27 | 27/forgotten_memberships.sql
      27 | 27/ts.py
      28 | 28/event_push_actions.sql
      28 | 28/events_room_stream.sql
      28 | 28/public_roms_index.sql
      28 | 28/receipts_user_id_index.sql
      28 | 28/upgrade_times.sql
      28 | 28/users_is_guest.sql
      29 | 29/push_actions.sql
      30 | 30/__pycache__
      30 | 30/alias_creator.sql
      30 | 30/as_users.py
      30 | 30/deleted_pushers.sql
      30 | 30/presence_stream.sql
      30 | 30/public_rooms.sql
      30 | 30/push_rule_stream.sql
      30 | 30/state_stream.sql
      30 | 30/threepid_guest_access_tokens.sql
      31 | 31/__pycache__
      31 | 31/invites.sql
      31 | 31/local_media_repository_url_cache.sql
      31 | 31/pushers.py
      31 | 31/pushers_index.sql
      31 | 31/search_update.py
      32 | 32/events.sql
      32 | 32/openid.sql
      32 | 32/pusher_throttle.sql
      32 | 32/remove_indices.sql
      32 | 32/reports.sql
      33 | 33/__pycache__
      33 | 33/access_tokens_device_index.sql
      33 | 33/devices.sql
      33 | 33/devices_for_e2e_keys.sql
      33 | 33/devices_for_e2e_keys_clear_unknown_device.sql
      33 | 33/event_fields.py
      33 | 33/remote_media_ts.py
      33 | 33/user_ips_index.sql
      34 | 34/__pycache__
      34 | 34/appservice_stream.sql
      34 | 34/cache_stream.py
      34 | 34/device_inbox.sql
      34 | 34/push_display_name_rename.sql
      34 | 34/received_txn_purge.py
      35 | 35/add_state_index.sql
      35 | 35/contains_url.sql
      35 | 35/device_outbox.sql
      35 | 35/device_stream_id.sql
      35 | 35/event_push_actions_index.sql
      35 | 35/public_room_list_change_stream.sql
      35 | 35/state.sql
      35 | 35/state_dedupe.sql
      35 | 35/stream_order_to_extrem.sql
      36 | 36/readd_public_rooms.sql
      37 | 37/__pycache__
      37 | 37/remove_auth_idx.py
      37 | 37/user_threepids.sql
      38 | 38/postgres_fts_gist.sql
      39 | 39/appservice_room_list.sql
      39 | 39/device_federation_stream_idx.sql
      39 | 39/event_push_index.sql
      39 | 39/federation_out_position.sql
      39 | 39/membership_profile.sql
      40 | 40/current_state_idx.sql
      40 | 40/device_inbox.sql
      40 | 40/device_list_streams.sql
      40 | 40/event_push_summary.sql
      40 | 40/pushers.sql
      41 | 41/device_list_stream_idx.sql
      41 | 41/device_outbound_index.sql
      41 | 41/event_search_event_id_idx.sql
      41 | 41/ratelimit.sql
      42 | 42/__pycache__
      42 | 42/current_state_delta.sql
      42 | 42/device_list_last_id.sql
      42 | 42/event_auth_state_only.sql
      42 | 42/user_dir.py
      43 | 43/blocked_rooms.sql
      43 | 43/quarantine_media.sql
      43 | 43/url_cache.sql
      43 | 43/user_share.sql
      44 | 44/expire_url_cache.sql
      45 | 45/group_server.sql
      45 | 45/profile_cache.sql
      46 | 46/drop_refresh_tokens.sql
      46 | 46/drop_unique_deleted_pushers.sql
      46 | 46/group_server.sql
      46 | 46/local_media_repository_url_idx.sql
      46 | 46/user_dir_null_room_ids.sql
      46 | 46/user_dir_typos.sql
      47 | 47/__pycache__
      47 | 47/last_access_media.sql
      47 | 47/postgres_fts_gin.sql
      47 | 47/push_actions_staging.sql
      47 | 47/state_group_seq.py
      48 | 48/__pycache__
      48 | 48/add_user_consent.sql
      48 | 48/add_user_ips_last_seen_index.sql
      48 | 48/deactivated_users.sql
      48 | 48/group_unique_indexes.py
      48 | 48/groups_joinable.sql
      49 | 49/add_user_consent_server_notice_sent.sql
      49 | 49/add_user_daily_visits.sql
      49 | 49/add_user_ips_last_seen_only_index.sql
      50 | 50/__pycache__
      50 | 50/add_creation_ts_users_index.sql
      50 | 50/erasure_store.sql
      50 | 50/make_event_content_nullable.py
      51 | 51/e2e_room_keys.sql
      51 | 51/monthly_active_users.sql
      52 | 52/add_event_to_state_group_index.sql
      52 | 52/device_list_streams_unique_idx.sql
      52 | 52/e2e_room_keys.sql
      53 | 53/add_user_type_to_users.sql
      53 | 53/drop_sent_transactions.sql
      53 | 53/event_format_version.sql
      53 | 53/user_dir_populate.sql
      53 | 53/user_ips_index.sql
      53 | 53/user_share.sql
      53 | 53/user_threepid_id.sql
      53 | 53/users_in_public_rooms.sql
      54 | 54/account_validity_with_renewal.sql
      54 | 54/add_validity_to_server_keys.sql
      54 | 54/delete_forward_extremities.sql
      54 | 54/drop_legacy_tables.sql
      54 | 54/drop_presence_list.sql
      54 | 54/relations.sql
      54 | 54/stats.sql
      54 | 54/stats2.sql
      55 | 55/access_token_expiry.sql
      55 | 55/track_threepid_validations.sql
      55 | 55/users_alter_deactivated.sql
      56 | 56/add_spans_to_device_lists.sql
      56 | 56/current_state_events_membership.sql
      56 | 56/current_state_events_membership_mk2.sql
      56 | 56/delete_keys_from_deleted_backups.sql
      56 | 56/destinations_failure_ts.sql
      56 | 56/destinations_retry_interval_type.sql.postgres
      56 | 56/device_stream_id_insert.sql
      56 | 56/devices_last_seen.sql
      56 | 56/drop_unused_event_tables.sql
      56 | 56/event_expiry.sql
      56 | 56/event_labels.sql
      56 | 56/event_labels_background_update.sql
      56 | 56/fix_room_keys_index.sql
      56 | 56/hidden_devices.sql
      56 | 56/nuke_empty_communities_from_db.sql
      56 | 56/public_room_list_idx.sql
      56 | 56/redaction_censor.sql
      56 | 56/redaction_censor2.sql
      56 | 56/redaction_censor3_fix_update.sql.postgres
      56 | 56/redaction_censor4.sql
      56 | 56/remove_tombstoned_rooms_from_directory.sql
      56 | 56/room_key_etag.sql
      56 | 56/room_membership_idx.sql
      56 | 56/room_retention.sql
      56 | 56/signing_keys.sql
      56 | 56/signing_keys_nonunique_signatures.sql
      56 | 56/state_group_room_idx.sql
      56 | 56/stats_separated.sql
      56 | 56/unique_user_filter_index.py
      56 | 56/user_external_ids.sql
      56 | 56/users_in_public_rooms_idx.sql
      57 | 57/delete_old_current_state_events.sql
      57 | 57/device_list_remote_cache_stale.sql
      57 | 57/local_current_membership.py
      57 | 57/remove_sent_outbound_pokes.sql
      57 | 57/rooms_version_column.sql
      57 | 57/rooms_version_column_2.sql.postgres
      57 | 57/rooms_version_column_3.sql.postgres
      58 | 58/00background_update_ordering.sql
      58 | 58/02remove_dup_outbound_pokes.sql
      58 | 58/03persist_ui_auth.sql
      58 | 58/05cache_instance.sql.postgres
      58 | 58/06dlols_unique_idx.py
      58 | 58/07add_method_to_thumbnail_constraint.sql.postgres
      58 | 58/07persist_ui_auth_ips.sql
      58 | 58/08_media_safe_from_quarantine.sql.postgres
      58 | 58/09shadow_ban.sql
      58 | 58/10_pushrules_enabled_delete_obsolete.sql
      58 | 58/10drop_local_rejections_stream.sql
      58 | 58/10federation_pos_instance_name.sql
      58 | 58/11dehydration.sql
      58 | 58/11fallback.sql
      58 | 58/11user_id_seq.py
      58 | 58/12room_stats.sql
      58 | 58/13remove_presence_allow_inbound.sql
      58 | 58/14events_instance_name.sql
      58 | 58/14events_instance_name.sql.postgres
      58 | 58/15_catchup_destination_rooms.sql
      58 | 58/15unread_count.sql
      58 | 58/16populate_stats_process_rooms_fix.sql
      58 | 58/17_catchup_last_successful.sql
      58 | 58/18stream_positions.sql
      58 | 58/19instance_map.sql.postgres
      58 | 58/19txn_id.sql
      58 | 58/20instance_name_event_tables.sql
      58 | 58/20user_daily_visits.sql
      58 | 58/21as_device_stream.sql
      58 | 58/21drop_device_max_stream_id.sql
      58 | 58/22puppet_token.sql
      58 | 58/22users_have_local_media.sql
      58 | 58/23e2e_cross_signing_keys_idx.sql
      58 | 58/24drop_event_json_index.sql
      58 | 58/25user_external_ids_user_id_idx.sql
      58 | 58/26access_token_last_validated.sql
      58 | 58/27local_invites.sql
      58 | 58/28drop_last_used_column.sql.postgres
      59 | 59/01ignored_user.py
      59 | 59/02shard_send_to_device.sql
      59 | 59/03shard_send_to_device_sequence.sql.postgres
      59 | 59/04_event_auth_chains.sql
      59 | 59/04_event_auth_chains.sql.postgres
      59 | 59/04drop_account_data.sql
      59 | 59/05cache_invalidation.sql
      59 | 59/06chain_cover_index.sql
      59 | 59/06shard_account_data.sql
      59 | 59/06shard_account_data.sql.postgres
      59 | 59/07shard_account_data_fix.sql
      59 | 59/08delete_pushers_for_deactivated_accounts.sql
      59 | 59/08delete_stale_pushers.sql
      59 | 59/09rejected_events_metadata.sql
      59 | 59/10delete_purged_chain_cover.sql
      59 | 59/11add_knock_members_to_stats.sql
      59 | 59/11drop_thumbnail_constraint.sql.postgres
      59 | 59/12account_validity_token_used_ts_ms.sql
      59 | 59/12presence_stream_instance.sql
      59 | 59/12presence_stream_instance_seq.sql.postgres
      59 | 59/13users_to_send_full_presence_to.sql
      59 | 59/14refresh_tokens.sql
      59 | 59/15locks.sql
      59 | 59/16federation_inbound_staging.sql
      60 | 60/01recreate_stream_ordering.sql.postgres
      60 | 60/02change_stream_ordering_columns.sql.postgres
      61 | 61/01change_appservices_txns.sql.postgres
      61 | 61/01insertion_event_lookups.sql
      61 | 61/02drop_redundant_room_depth_index.sql
      61 | 61/02state_groups_state_n_distinct.sql.postgres
      61 | 61/03recreate_min_depth.py
      62 | 62/01insertion_event_extremities.sql
      63 | 63/01create_registration_tokens.sql
      63 | 63/02delete_unlinked_email_pushers.sql
      63 | 63/02populate-rooms-creator.sql
      63 | 63/03session_store.sql
      63 | 63/04add_presence_stream_not_offline_index.sql
      64 | 64/01msc2716_chunk_to_batch_rename.sql.postgres
      65 | 65/01msc2716_insertion_event_edges.sql
      65 | 65/02_thread_relations.sql
      65 | 65/03remove_hidden_devices_from_device_inbox.sql
      65 | 65/04_local_group_updates.sql
      65 | 65/05_remove_room_stats_historical_and_user_stats_historical.sql
      65 | 65/06remove_deleted_devices_from_device_inbox.sql
      65 | 65/07_arbitrary_relations.sql
      65 | 65/08_device_inbox_background_updates.sql
      65 | 65/10_expirable_refresh_tokens.sql
      65 | 65/11_devices_auth_provider_session.sql
      67 | 67/01drop_public_room_list_stream.sql
      68 | 68/01event_columns.sql
      68 | 68/02_msc2409_add_device_id_appservice_stream_type.sql
      68 | 68/03_delete_account_data_for_deactivated_accounts.sql
      68 | 68/04_refresh_tokens_index_next_token_id.sql
      68 | 68/04partial_state_rooms.sql
      68 | 68/05partial_state_rooms_triggers.py
      68 | 68/06_msc3202_add_device_list_appservice_stream_type.sql
      69 | 69/01as_txn_seq.py
      69 | 69/01device_list_oubound_by_room.sql
      69 | 69/02cache_invalidation_index.sql
      70 | 70/01clean_table_purged_rooms.sql
      70 | 70/08_state_group_edges_unique.sql
      71 | 71/01rebuild_event_edges.sql.postgres
      71 | 71/01remove_noop_background_updates.sql
      71 | 71/02event_push_summary_unique.sql
      72 | 72/01add_room_type_to_state_stats.sql
      72 | 72/01event_push_summary_receipt.sql
      72 | 72/02event_push_actions_index.sql
      72 | 72/03bg_populate_events_columns.py
      72 | 72/03drop_event_reference_hashes.sql
      72 | 72/03remove_groups.sql
      72 | 72/04drop_column_application_services_state_last_txn.sql.postgres
      72 | 72/05receipts_event_stream_ordering.sql
      72 | 72/05remove_unstable_private_read_receipts.sql
      72 | 72/06add_consent_ts_to_users.sql
      72 | 72/06thread_notifications.sql
      72 | 72/07force_update_current_state_events_membership.py
      72 | 72/07thread_receipts.sql.postgres
      72 | 72/08begin_cache_invalidation_seq_at_2.sql.postgres
      72 | 72/08thread_receipts.sql
      73 | 73/01event_failed_pull_attempts.sql
      73 | 73/02add_pusher_enabled.sql
      73 | 73/02room_id_indexes_for_purging.sql
      73 | 73/03pusher_device_id.sql
      73 | 73/03users_approved_column.sql
      73 | 73/04partial_join_details.sql
      73 | 73/04pending_device_list_updates.sql
      73 | 73/05old_push_actions.sql.postgres
      73 | 73/06thread_notifications_thread_id_idx.sql
      73 | 73/08thread_receipts_non_null.sql.postgres
      73 | 73/09partial_joined_via_destination.sql
      73 | 73/09threads_table.sql
      73 | 73/10_update_sqlite_fts4_tokenizer.py
      73 | 73/10login_tokens.sql

@reivilibre reivilibre removed the X-Needs-Info This issue is blocked awaiting information from the reporter label Dec 8, 2022
@squahtx squahtx self-assigned this Dec 15, 2022
@squahtx
Copy link
Contributor

squahtx commented Dec 15, 2022

Can you try running the following sql and report back on whether that fixes things?

BEGIN TRANSACTION;
SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;

SELECT MAX(stream_ordering) FROM event_push_summary;

INSERT INTO event_push_summary_stream_ordering(lock, stream_ordering) VALUES ('X', (SELECT MAX(stream_ordering) FROM event_push_summary));

COMMIT;

@squahtx squahtx added A-Push Issues related to push/notifications S-Major Major functionality / product severely impaired, no satisfactory workaround. O-Uncommon Most users are unlikely to come across this or unexpected workflow X-Needs-Info This issue is blocked awaiting information from the reporter labels Dec 15, 2022
@lsrmg
Copy link
Author

lsrmg commented Dec 16, 2022

Thanks for getting into it. I ran the sql and the answer was:

BEGIN
SET
 max 
-----
    
(1 row)

ERROR:  null value in column "stream_ordering" of relation "event_push_summary_stream_ordering" violates not-null constraint
DETAIL:  Failing row contains (X, null).
ROLLBACK

@squahtx
Copy link
Contributor

squahtx commented Dec 16, 2022

In that case, please try:

INSERT INTO event_push_summary_stream_ordering(lock, stream_ordering) VALUES('X', 0);

How long has your Synpase deployment been around, out of interest?

@lsrmg
Copy link
Author

lsrmg commented Dec 16, 2022

That sql worked. We are using this synapse since early 2020.

The error is gone. I rolled back the server to a state before the sql statement and the error imminently appeared again.

I also tested updating to 1.73.0 (after sql), but the "404: No row found" it still present for /sync-Requests.

Shutting the services down I noticed exceptions for every worker that I didn't noticed before:

2022-12-16 01:34:16,871 - background_worker - synapse.metrics.background_process_metrics - 242 - ERROR - send-cmd-8 - Background process 'send-cmd' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/replication/tcp/redis.py", line 226, in _async_send_command
    await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 857, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/txredisapi.py", line 533, in handle_reply
    raise r
txredisapi.ConnectionError: Lost connection

Also there was an other error I've never seen before:

2022-12-16 01:49:34,196 - sync_worker1 - twisted - 274 - ERROR - sentinel - ----------------------------------------
2022-12-16 01:49:34,197 - sync_worker1 - twisted - 274 - ERROR - sentinel - Exception occurred during processing of request from ('127.0.0.1', 33426)
2022-12-16 01:49:34,219 - sync_worker1 - twisted - 274 - ERROR - sentinel - Traceback (most recent call last):
2022-12-16 01:49:34,219 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/usr/lib/python3.9/socketserver.py", line 650, in process_request_thread
2022-12-16 01:49:34,220 - sync_worker1 - twisted - 274 - ERROR - sentinel -     self.finish_request(request, client_address)
2022-12-16 01:49:34,220 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/usr/lib/python3.9/socketserver.py", line 360, in finish_request
2022-12-16 01:49:34,220 - sync_worker1 - twisted - 274 - ERROR - sentinel -     self.RequestHandlerClass(request, client_address, self)
2022-12-16 01:49:34,220 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/usr/lib/python3.9/socketserver.py", line 720, in __init__
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -     self.handle()
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/usr/lib/python3.9/http/server.py", line 427, in handle
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -     self.handle_one_request()
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/usr/lib/python3.9/http/server.py", line 415, in handle_one_request
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -     method()
2022-12-16 01:49:34,221 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/_legacy_exposition.py", line 218, in do_GET
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -     output = generate_latest(registry, emit_help=emit_help)
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/_legacy_exposition.py", line 110, in generate_latest
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -     for metric in registry.collect():
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/__init__.py", line 70, in collect
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -     for metric in REGISTRY.collect():
2022-12-16 01:49:34,222 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/prometheus_client/registry.py", line 97, in collect
2022-12-16 01:49:34,223 - sync_worker1 - twisted - 274 - ERROR - sentinel -     yield from collector.collect()
2022-12-16 01:49:34,223 - sync_worker1 - twisted - 274 - ERROR - sentinel -   File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/util/metrics.py", line 230, in collect
2022-12-16 01:49:34,223 - sync_worker1 - twisted - 274 - ERROR - sentinel -     for pre_update_hook in self._pre_update_hooks.values():
2022-12-16 01:49:34,223 - sync_worker1 - twisted - 274 - ERROR - sentinel - RuntimeError: dictionary changed size during iteration
2022-12-16 01:49:34,223 - sync_worker1 - twisted - 274 - ERROR - sentinel - ----------------------------------------

I don't know if its related to this problem, but I also got this error (before and after sending your sql statement):

2022-12-16 01:43:08,088 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-0 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stream.py", line 1085, in get_federation_out_pos
    await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/stream.py", line 1153, in _reset_federation_positions_txn
    assert set(min_positions) == {"federation", "events"}
AssertionError

Thank you for your help!
I hope this helps. I'm sorry for the strange problems... I can't get the hang of it...

@squahtx
Copy link
Contributor

squahtx commented Dec 19, 2022

I also tested updating to 1.73.0 (after sql), but the "404: No row found" it still present for /sync-Requests.

If possible, could you post the new, full stack trace for that error?

txredisapi.ConnectionError: Lost connection

This one is expected while shutting down a deployment with workers.

RuntimeError: dictionary changed size during iteration

This one's a known issue. It should only affect Prometheus metrics.

assert set(min_positions) == {"federation", "events"}

AssertionError

Is your federation_stream_position table also somehow empty? SELECT * FROM federation_stream_position

@lsrmg
Copy link
Author

lsrmg commented Dec 19, 2022

If possible, could you post the new, full stack trace for that error?

I'm sorry but due to vacation this is not possible until mid-February... I'll get back to you.

Is your federation_stream_position table also somehow empty? SELECT * FROM federation_stream_position

Yes it is. (I don‘t use federation, the server is running in a company network only)

@lsrmg
Copy link
Author

lsrmg commented Jan 10, 2023

Today I had the chance to update to v1.75.0rc1 and collect the logs. I noticed that an issue #14669 resolved a problem that looks like mine, but for me the error still is present after updating. On client-side the /sync request isn't working and preventing me to update since 1.71.0. Before the error is also in the logs but don't prevent clients from syncing.

2023-01-10 21:06:47,829 - synapse.http.server - 108 - INFO - GET-1894 - <XForwardedForRequest at 0x7f7a8816fd30 method='GET' uri='/_matrix/client/r0/sync?filter=1&timeout=0&since=s7876316_225806662_0_12183312_24963652_12350_380413_201974_0' clientproto='HTTP/1.0' site='8008'> SynapseError: 404 - No row found

[...]

2023-01-10 21:03:14,417 - synapse.storage.txn - 691 - DEBUG - _get_stats_for_federation_staging-0 - [TXN START] {_get_stats_for_federation_staging-3c9}
2023-01-10 21:03:14,417 - synapse.storage.SQL - 416 - DEBUG - _get_stats_for_federation_staging-0 - [SQL] {_get_stats_for_federation_staging-3c9} SELECT count(*) FROM federation_inbound_events_staging
2023-01-10 21:03:14,419 - synapse.storage.SQL - 442 - DEBUG - _get_stats_for_federation_staging-0 - [SQL time] {_get_stats_for_federation_staging-3c9} 0.001631 sec
2023-01-10 21:03:14,419 - synapse.storage.SQL - 416 - DEBUG - _get_stats_for_federation_staging-0 - [SQL] {_get_stats_for_federation_staging-3c9} SELECT min(received_ts) FROM federation_inbound_events_staging
2023-01-10 21:03:14,420 - synapse.storage.SQL - 442 - DEBUG - _get_stats_for_federation_staging-0 - [SQL time] {_get_stats_for_federation_staging-3c9} 0.000731 sec
2023-01-10 21:03:14,421 - synapse.storage.txn - 795 - DEBUG - _get_stats_for_federation_staging-0 - [TXN END] {_get_stats_for_federation_staging-3c9} 0.003409 sec
2023-01-10 21:03:14,432 - synapse.storage.txn - 691 - DEBUG - prune_old_user_ips-5 - [TXN START] {has_completed_background_update-3ca}
2023-01-10 21:03:14,432 - synapse.storage.SQL - 416 - DEBUG - prune_old_user_ips-5 - [SQL] {has_completed_background_update-3ca} SELECT 1 FROM background_updates WHERE update_name = ?
2023-01-10 21:03:14,432 - synapse.storage.SQL - 421 - DEBUG - prune_old_user_ips-5 - [SQL values] {has_completed_background_update-3ca} ['devices_last_seen']
2023-01-10 21:03:14,433 - synapse.storage.SQL - 442 - DEBUG - prune_old_user_ips-5 - [SQL time] {has_completed_background_update-3ca} 0.000573 sec
2023-01-10 21:03:14,433 - synapse.storage.txn - 795 - DEBUG - prune_old_user_ips-5 - [TXN END] {has_completed_background_update-3ca} 0.001077 sec
2023-01-10 21:03:14,434 - synapse.storage.txn - 691 - DEBUG - prune_old_user_ips-5 - [TXN START] {_prune_old_user_ips-3cb}
2023-01-10 21:03:14,434 - synapse.storage.SQL - 416 - DEBUG - prune_old_user_ips-5 - [SQL] {_prune_old_user_ips-3cb} DELETE FROM user_ips WHERE last_seen <= ( SELECT COALESCE(MAX(last_seen), -1) FROM ( SELECT last_seen FROM user_ips WHERE last_seen <= ? ORDER BY last_seen ASC LIMIT 5000 ) AS u )
2023-01-10 21:03:14,434 - synapse.storage.SQL - 421 - DEBUG - prune_old_user_ips-5 - [SQL values] {_prune_old_user_ips-3cb} (1673294594434,)
2023-01-10 21:03:14,437 - synapse.storage.SQL - 442 - DEBUG - prune_old_user_ips-5 - [SQL time] {_prune_old_user_ips-3cb} 0.002226 sec
2023-01-10 21:03:14,437 - synapse.storage.txn - 795 - DEBUG - prune_old_user_ips-5 - [TXN END] {_prune_old_user_ips-3cb} 0.002880 sec
2023-01-10 21:03:15,031 - synapse.storage.databases.main.event_push_actions - 1434 - DEBUG - rotate_notifs-0 - Handling new receipts
2023-01-10 21:03:15,032 - synapse.storage.txn - 691 - DEBUG - rotate_notifs-0 - [TXN START] {_handle_new_receipts_for_notifs_txn-3cc}
2023-01-10 21:03:15,033 - synapse.storage.SQL - 416 - DEBUG - rotate_notifs-0 - [SQL] {_handle_new_receipts_for_notifs_txn-3cc} SELECT stream_id FROM event_push_summary_last_receipt_stream_id
2023-01-10 21:03:15,034 - synapse.storage.SQL - 442 - DEBUG - rotate_notifs-0 - [SQL time] {_handle_new_receipts_for_notifs_txn-3cc} 0.001492 sec
2023-01-10 21:03:15,034 - synapse.storage.SQL - 416 - DEBUG - rotate_notifs-0 - [SQL] {_handle_new_receipts_for_notifs_txn-3cc} SELECT stream_ordering FROM event_push_summary_stream_ordering
2023-01-10 21:03:15,035 - synapse.storage.SQL - 442 - DEBUG - rotate_notifs-0 - [SQL time] {_handle_new_receipts_for_notifs_txn-3cc} 0.000652 sec
2023-01-10 21:03:15,035 - synapse.storage.txn - 787 - DEBUG - rotate_notifs-0 - [TXN FAIL] {_handle_new_receipts_for_notifs_txn-3cc} 404: No row found
2023-01-10 21:03:15,035 - synapse.storage.txn - 795 - DEBUG - rotate_notifs-0 - [TXN END] {_handle_new_receipts_for_notifs_txn-3cc} 0.003040 sec
2023-01-10 21:03:15,037 - synapse.metrics.background_process_metrics - 244 - ERROR - rotate_notifs-0 - Background process 'rotate_notifs' threw an exception
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run
    return await func(*args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1436, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 882, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 849, in _runInteraction
    result = await self.runWithConnection(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 977, in runWithConnection
    return await make_deferred_yieldable(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 970, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 711, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1481, in _handle_new_receipts_for_notifs_txn
    old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 1699, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

@DMRobertson
Copy link
Contributor

DMRobertson commented Jan 18, 2023

File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/databases/main/event_push_actions.py", line 1481, in _handle_new_receipts_for_notifs_txn
    old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
  File "/opt/venvs/matrix-synapse/lib/python3.9/site-packages/synapse/storage/database.py", line 1699, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

This corresponds to

# The (inclusive) event stream ordering that was previously summarised.
old_rotate_stream_ordering = self.db_pool.simple_select_one_onecol_txn(
txn,
table="event_push_summary_stream_ordering",
keyvalues={},
retcol="stream_ordering",
)

Which suggests your event_push_summary_stream_ordering table is empty. This table was added back in 2017 with a row:

-- The stream ordering up to which we have aggregated the event_push_actions
-- table into event_push_summary
CREATE TABLE event_push_summary_stream_ordering (
Lock CHAR(1) NOT NULL DEFAULT 'X' UNIQUE, -- Makes sure this table only has one row.
stream_ordering BIGINT NOT NULL,
CHECK (Lock='X')
);
INSERT INTO event_push_summary_stream_ordering (stream_ordering) VALUES (0);

And this row is never deleted; it is only ever updated here:

txn.execute(
"UPDATE event_push_summary_stream_ordering SET stream_ordering = ?",
(rotate_to_stream_ordering,),
)

But what's even more confusing is that there should be a row here, because you ran some SQL to make one according to this comment here. To confirm this, can you run

\d event_push_summary_stream_ordering
SELECT count(*) FROM event_push_summary_stream_ordering;

@lsrmg
Copy link
Author

lsrmg commented Jan 20, 2023

But what's even more confusing is that there should be a row here, because you ran some SQL to make one according to this comment here.

I rolled back the server after trying to insert the SQL, because /sync wasn‘t working even with the row inserted.
I didn‘t know that is has to stay like this… is it? Indeed the table is empty (count=0).

I‘m not sure what’s going on and why since synapse v1.71.0 the /sync endpoint refused to work for me.
Its not this error alone, because the insertion didn‘t fixed /sync not working.

Information: In the meantime I had to enable message-retention on the server for privacy reasons.

@lsrmg
Copy link
Author

lsrmg commented Jan 25, 2023

Good news: The problem has solved, but I don't fully understand what happend.

I have inserted the mentioned line into the "event_push_summary_stream_ordering"-table and noticed a higher CPU-load on my background-worker. I waited 10mins for the load to normalize and installed the synapse 1.75 update.

The last time i tested this approach unsuccessful, I updated synapse and than inserted the line into the table afterwards and instantly tested the client where /sync was failing with "no row found" and I rolled back the server. In the end I don't know if I just had to wait some time.

Thank you all for helping out! I guess @squahtx and @DMRobertson where right with pointing to event_push_summary_stream_ordering. Or 1.75.0 fixed something else failing for me.

@mi-gri If you got the same Problem try inserting the row mentioned here

@clokep clokep closed this as completed Jan 25, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Push Issues related to push/notifications O-Uncommon Most users are unlikely to come across this or unexpected workflow S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Development

No branches or pull requests

7 participants