Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Enabling Metrics while having a federation_sender sharded causes an error #14112

Closed
realtyem opened this issue Oct 9, 2022 · 6 comments · Fixed by #14165
Closed

Enabling Metrics while having a federation_sender sharded causes an error #14112

realtyem opened this issue Oct 9, 2022 · 6 comments · Fixed by #14165
Labels
A-Metrics metrics, measures, stuff we put in Prometheus X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@realtyem
Copy link
Contributor

realtyem commented Oct 9, 2022

Description

My homeserver uses workers extensively, and I like to see the graphs so I enabled metrics. Somewhat recently, I started noticing this log output. This crash does not seem to happen with only a single federation_sender. Synapse doesn't appear to stop functioning, just doesn't provide the actual metric data.

Steps to reproduce

  • Use at least two declarations of "federation_sender", however your distribution method allows.
  • Make sure the homeserver.yaml file has "enable_metrics" set to true.

Homeserver

matrix.littlevortex.net

Synapse Version

1.69.0rc2 from the develop branch

Installation Method

Other (please mention below)

Platform

Custom Docker image assembled from your Dockerfile and Dockerfile-workers. View it here. Python 3.10 and I run with TEST_ONLY_IGNORE_POETRY_LOCKFILE and SYNAPSE_USE_EXPERIMENTAL_FORKING_LAUNCHER enabled so always get latest dependencies and faster start up.
I run an Unraid server 48 core dual Xeon setup with 128Gb of RAM and my normal worker set has 30(including master) members.

Relevant log output

federation_sender1 | 2022-10-06 15:44:22,299 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-5 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found

Anything else that would be useful to know?

When two federation_senders are used, this error does occur in both logs.
@DMRobertson
Copy link
Contributor

Somewhat recently, I started noticing this log output.

Any idea when? How far back do your logs go---can you pin it down to a Synapse version?

This crash does not seem to happen with only a single federation_sender.

Is there a Synapse worker which actually crashes? Synapse doesn't appear to stop functioning,

just doesn't provide the actual metric data.

Additionallu:

I run with TEST_ONLY_IGNORE_POETRY_LOCKFILE and SYNAPSE_USE_EXPERIMENTAL_FORKING_LAUNCHER

These aren't supported and aren't likely to cause the problem... but have you tried without these options?

@DMRobertson DMRobertson added X-Needs-Info This issue is blocked awaiting information from the reporter A-Metrics metrics, measures, stuff we put in Prometheus labels Oct 10, 2022
@realtyem
Copy link
Contributor Author

realtyem commented Oct 10, 2022 •
edited
Loading

Any idea when? How far back do your logs go---can you pin it down to a Synapse version?

I'm sorry to say I don't have an exact date, I only noticed it the first time around Oct 5, 2022. I did take a 2 minute debug log sample of all the workers and master on Oct 6, but I haven't combed through them yet for privacy and since it's a lot, I wasn't sure where to put them here where they wouldn't be looked through by random public.

Is there a Synapse worker which actually crashes? Synapse doesn't appear to stop functioning,

Poor choice of words on my part. An exception occurs, but doesn't seem to actually crash.

These aren't supported and aren't likely to cause the problem... but have you tried without these options?

I'm aware they aren't supported, but were convenient and appear to work great for what they do. I will try again without them and edit this post afterwards.

-Edit: The exception still occurs, but I do have additional info I missed before. More below.

@realtyem
Copy link
Contributor Author

realtyem commented Oct 10, 2022 •
edited
Loading

Context: Logs from before and after have 3 federation_senders defined.

Looks like it's just manifesting as a metrics issue, but this time I saw a _save_and_send_ack error first. I did go back and look and this was in the original log sample I grabbed.

federation_sender1 | 2022-10-06 20:35:47,520 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-0 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)

Afterwards, the metrics exception from the original issue(specifically, the process_event_queue_for_federation) does repeat. This error would show in 2 of the federation_sender logs. In the sample from Oct 6, it was in sender log 1 and 3 but none of these errors were in sender log 2. Instead, sender log 2 had:

federation_sender2 | 2022-10-06 20:35:47,480 - synapse.storage.txn - 717 - WARNING - _save_and_send_ack-0 - [TXN OPERROR] {_reset_federation_positions_txn-31} could not serialize access due to concurrent delete
 0/5

Which did repeat once more. I'll dump the edited down to(what I think) is relevant below.

federation_sender1 log 10-06-2022 
federation_sender1 | 2022-10-06 20:35:47,468 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-0 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,473 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {_reset_federation_positions_txn-30}
federation_sender1 | 2022-10-06 20:35:47,492 - synapse.util.async_helpers - 434 - DEBUG - _save_and_send_ack-1 - Waiting to acquire linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,495 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {_reset_federation_positions_txn-30} 0.021306 sec
federation_sender1 | 2022-10-06 20:35:47,509 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {update_federation_out_pos-32}
federation_sender1 | 2022-10-06 20:35:47,514 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-0 - [TXN FAIL] {update_federation_out_pos-32} 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:47,516 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {update_federation_out_pos-32} 0.006992 sec
federation_sender1 | 2022-10-06 20:35:47,520 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-0 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,520 - synapse.util.async_helpers - 460 - DEBUG - _save_and_send_ack-1 - Acquired linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,520 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-0 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:47,530 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-1 - [TXN START] {update_federation_out_pos-35}
federation_sender1 | 2022-10-06 20:35:47,531 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-1 - [TXN FAIL] {update_federation_out_pos-35} 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:47,532 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-1 - [TXN END] {update_federation_out_pos-35} 0.001596 sec
federation_sender1 | 2022-10-06 20:35:47,535 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-1 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,537 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-1 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:47,658 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-2 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,661 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-2 - [TXN START] {update_federation_out_pos-3d}
federation_sender1 | 2022-10-06 20:35:47,667 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-2 - [TXN FAIL] {update_federation_out_pos-3d} 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:47,668 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-2 - [TXN END] {update_federation_out_pos-3d} 0.006907 sec
federation_sender1 | 2022-10-06 20:35:47,673 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-2 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:47,676 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-2 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:48,343 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-3 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:48,344 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-3 - [TXN START] {update_federation_out_pos-3f}
federation_sender1 | 2022-10-06 20:35:48,346 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-3 - [TXN FAIL] {update_federation_out_pos-3f} 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:35:48,347 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-3 - [TXN END] {update_federation_out_pos-3f} 0.003056 sec
federation_sender1 | 2022-10-06 20:35:48,349 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-3 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender1 | 2022-10-06 20:35:48,350 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-3 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender1 | 2022-10-06 20:36:18,480 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-0 - [TXN START] {get_federation_out_pos-62}
federation_sender1 | 2022-10-06 20:36:18,483 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-0 - [TXN FAIL] {get_federation_out_pos-62} 404: No row found
federation_sender1 | 2022-10-06 20:36:18,483 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-0 - [TXN END] {get_federation_out_pos-62} 0.003489 sec
federation_sender1 | 2022-10-06 20:36:18,485 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-0 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:36:18,985 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-1 - [TXN START] {get_federation_out_pos-6c}
federation_sender1 | 2022-10-06 20:36:18,992 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-1 - [TXN FAIL] {get_federation_out_pos-6c} 404: No row found
federation_sender1 | 2022-10-06 20:36:18,992 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-1 - [TXN END] {get_federation_out_pos-6c} 0.007392 sec
federation_sender1 | 2022-10-06 20:36:18,998 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-1 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:36:19,104 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-2 - [TXN START] {get_federation_out_pos-6d}
federation_sender1 | 2022-10-06 20:36:19,105 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-2 - [TXN FAIL] {get_federation_out_pos-6d} 404: No row found
federation_sender1 | 2022-10-06 20:36:19,106 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-2 - [TXN END] {get_federation_out_pos-6d} 0.001340 sec
federation_sender1 | 2022-10-06 20:36:19,108 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-2 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:36:19,278 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-3 - [TXN START] {get_federation_out_pos-6e}
federation_sender1 | 2022-10-06 20:36:19,280 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-3 - [TXN FAIL] {get_federation_out_pos-6e} 404: No row found
federation_sender1 | 2022-10-06 20:36:19,280 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-3 - [TXN END] {get_federation_out_pos-6e} 0.001660 sec
federation_sender1 | 2022-10-06 20:36:19,282 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-3 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:36:19,403 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-4 - [TXN START] {get_federation_out_pos-6f}
federation_sender1 | 2022-10-06 20:36:19,404 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-4 - [TXN FAIL] {get_federation_out_pos-6f} 404: No row found
federation_sender1 | 2022-10-06 20:36:19,404 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-4 - [TXN END] {get_federation_out_pos-6f} 0.001421 sec
federation_sender1 | 2022-10-06 20:36:19,406 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-4 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:36:59,952 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-5 - [TXN START] {get_federation_out_pos-b1}
federation_sender1 | 2022-10-06 20:36:59,956 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-5 - [TXN FAIL] {get_federation_out_pos-b1} 404: No row found
federation_sender1 | 2022-10-06 20:36:59,956 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-5 - [TXN END] {get_federation_out_pos-b1} 0.003437 sec
federation_sender1 | 2022-10-06 20:36:59,958 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-5 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:37:05,674 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-6 - [TXN START] {get_federation_out_pos-b3}
federation_sender1 | 2022-10-06 20:37:05,676 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-6 - [TXN FAIL] {get_federation_out_pos-b3} 404: No row found
federation_sender1 | 2022-10-06 20:37:05,676 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-6 - [TXN END] {get_federation_out_pos-b3} 0.001456 sec
federation_sender1 | 2022-10-06 20:37:05,678 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-6 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender1 | 2022-10-06 20:37:05,806 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-7 - [TXN START] {get_federation_out_pos-b4}
federation_sender1 | 2022-10-06 20:37:05,807 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-7 - [TXN FAIL] {get_federation_out_pos-b4} 404: No row found
federation_sender1 | 2022-10-06 20:37:05,807 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-7 - [TXN END] {get_federation_out_pos-b4} 0.001628 sec
federation_sender1 | 2022-10-06 20:37:05,809 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-7 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender2 log 10-06-2022 
federation_sender2 | 2022-10-06 20:35:47,464 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-0 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,466 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {_reset_federation_positions_txn-31}
federation_sender2 | 2022-10-06 20:35:47,480 - synapse.storage.txn - 717 - WARNING - _save_and_send_ack-0 - [TXN OPERROR] {_reset_federation_positions_txn-31} could not serialize access due to concurrent delete
 0/5
federation_sender2 | 2022-10-06 20:35:47,487 - synapse.util.async_helpers - 434 - DEBUG - _save_and_send_ack-1 - Waiting to acquire linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,495 - synapse.storage.txn - 717 - WARNING - _save_and_send_ack-0 - [TXN OPERROR] {_reset_federation_positions_txn-31} could not serialize access due to concurrent delete
 1/5
federation_sender2 | 2022-10-06 20:35:47,514 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {_reset_federation_positions_txn-31} 0.047359 sec
federation_sender2 | 2022-10-06 20:35:47,515 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {update_federation_out_pos-34}
federation_sender2 | 2022-10-06 20:35:47,519 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {update_federation_out_pos-34} 0.003754 sec
federation_sender2 | 2022-10-06 20:35:47,525 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-0 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,525 - synapse.util.async_helpers - 460 - DEBUG - _save_and_send_ack-1 - Acquired linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,527 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-1 - [TXN START] {update_federation_out_pos-36}
federation_sender2 | 2022-10-06 20:35:47,528 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-1 - [TXN END] {update_federation_out_pos-36} 0.001314 sec
federation_sender2 | 2022-10-06 20:35:47,532 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-1 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,658 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-2 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:47,662 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-2 - [TXN START] {update_federation_out_pos-3b}
federation_sender2 | 2022-10-06 20:35:47,667 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-2 - [TXN END] {update_federation_out_pos-3b} 0.005100 sec
federation_sender2 | 2022-10-06 20:35:47,669 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-2 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:48,339 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-3 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender2 | 2022-10-06 20:35:48,341 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-3 - [TXN START] {update_federation_out_pos-3d}
federation_sender2 | 2022-10-06 20:35:48,346 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-3 - [TXN END] {update_federation_out_pos-3d} 0.004949 sec
federation_sender2 | 2022-10-06 20:35:48,347 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-3 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender3 log 10-06-2022 
federation_sender3 | 2022-10-06 20:35:47,463 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-0 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,464 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {_reset_federation_positions_txn-2e}
federation_sender3 | 2022-10-06 20:35:47,477 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {_reset_federation_positions_txn-2e} 0.012779 sec
federation_sender3 | 2022-10-06 20:35:47,489 - synapse.util.async_helpers - 434 - DEBUG - _save_and_send_ack-1 - Waiting to acquire linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,502 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-0 - [TXN START] {update_federation_out_pos-2f}
federation_sender3 | 2022-10-06 20:35:47,504 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-0 - [TXN FAIL] {update_federation_out_pos-2f} 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:47,505 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-0 - [TXN END] {update_federation_out_pos-2f} 0.003367 sec
federation_sender3 | 2022-10-06 20:35:47,510 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-0 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,510 - synapse.util.async_helpers - 460 - DEBUG - _save_and_send_ack-1 - Acquired linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,511 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-0 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:47,520 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-1 - [TXN START] {update_federation_out_pos-31}
federation_sender3 | 2022-10-06 20:35:47,522 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-1 - [TXN FAIL] {update_federation_out_pos-31} 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:47,522 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-1 - [TXN END] {update_federation_out_pos-31} 0.001914 sec
federation_sender3 | 2022-10-06 20:35:47,526 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-1 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,529 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-1 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:47,659 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-2 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,664 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-2 - [TXN START] {update_federation_out_pos-3b}
federation_sender3 | 2022-10-06 20:35:47,669 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-2 - [TXN FAIL] {update_federation_out_pos-3b} 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:47,670 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-2 - [TXN END] {update_federation_out_pos-3b} 0.005538 sec
federation_sender3 | 2022-10-06 20:35:47,675 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-2 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:47,675 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-2 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:48,340 - synapse.util.async_helpers - 424 - DEBUG - _save_and_send_ack-3 - Acquired uncontended linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:48,347 - synapse.storage.txn - 690 - DEBUG - _save_and_send_ack-3 - [TXN START] {update_federation_out_pos-3d}
federation_sender3 | 2022-10-06 20:35:48,348 - synapse.storage.txn - 786 - DEBUG - _save_and_send_ack-3 - [TXN FAIL] {update_federation_out_pos-3d} 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:35:48,348 - synapse.storage.txn - 794 - DEBUG - _save_and_send_ack-3 - [TXN END] {update_federation_out_pos-3d} 0.001188 sec
federation_sender3 | 2022-10-06 20:35:48,351 - synapse.util.async_helpers - 483 - DEBUG - _save_and_send_ack-3 - Releasing linearizer lock '_fed_position_linearizer' for key None
federation_sender3 | 2022-10-06 20:35:48,351 - synapse.replication.tcp.client - 487 - ERROR - _save_and_send_ack-3 - Error updating federation stream position
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/replication/tcp/client.py", line 477, in _save_and_send_ack
    await self.store.update_federation_out_pos(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1104, in update_federation_out_pos
    await self.db_pool.simple_update_one(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2042, in simple_update_one
    await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 2062, in simple_update_one_txn
    raise StoreError(404, "No row found (%s)" % (table,))
synapse.api.errors.StoreError: 404: No row found (federation_stream_position)
federation_sender3 | 2022-10-06 20:36:18,480 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-0 - [TXN START] {get_federation_out_pos-60}
federation_sender3 | 2022-10-06 20:36:18,481 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-0 - [TXN FAIL] {get_federation_out_pos-60} 404: No row found
federation_sender3 | 2022-10-06 20:36:18,481 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-0 - [TXN END] {get_federation_out_pos-60} 0.001578 sec
federation_sender3 | 2022-10-06 20:36:18,483 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-0 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:36:18,985 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-1 - [TXN START] {get_federation_out_pos-69}
federation_sender3 | 2022-10-06 20:36:18,986 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-1 - [TXN FAIL] {get_federation_out_pos-69} 404: No row found
federation_sender3 | 2022-10-06 20:36:18,989 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-1 - [TXN END] {get_federation_out_pos-69} 0.004455 sec
federation_sender3 | 2022-10-06 20:36:18,995 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-1 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:36:19,104 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-2 - [TXN START] {get_federation_out_pos-6b}
federation_sender3 | 2022-10-06 20:36:19,105 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-2 - [TXN FAIL] {get_federation_out_pos-6b} 404: No row found
federation_sender3 | 2022-10-06 20:36:19,106 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-2 - [TXN END] {get_federation_out_pos-6b} 0.001578 sec
federation_sender3 | 2022-10-06 20:36:19,107 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-2 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:36:19,278 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-3 - [TXN START] {get_federation_out_pos-6c}
federation_sender3 | 2022-10-06 20:36:19,279 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-3 - [TXN FAIL] {get_federation_out_pos-6c} 404: No row found
federation_sender3 | 2022-10-06 20:36:19,279 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-3 - [TXN END] {get_federation_out_pos-6c} 0.001121 sec
federation_sender3 | 2022-10-06 20:36:19,283 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-3 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:36:19,402 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-4 - [TXN START] {get_federation_out_pos-6d}
federation_sender3 | 2022-10-06 20:36:19,404 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-4 - [TXN FAIL] {get_federation_out_pos-6d} 404: No row found
federation_sender3 | 2022-10-06 20:36:19,404 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-4 - [TXN END] {get_federation_out_pos-6d} 0.001701 sec
federation_sender3 | 2022-10-06 20:36:19,407 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-4 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:36:59,953 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-5 - [TXN START] {get_federation_out_pos-b0}
federation_sender3 | 2022-10-06 20:36:59,956 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-5 - [TXN FAIL] {get_federation_out_pos-b0} 404: No row found
federation_sender3 | 2022-10-06 20:36:59,956 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-5 - [TXN END] {get_federation_out_pos-b0} 0.003465 sec
federation_sender3 | 2022-10-06 20:36:59,958 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-5 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:37:05,675 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-6 - [TXN START] {get_federation_out_pos-b2}
federation_sender3 | 2022-10-06 20:37:05,676 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-6 - [TXN FAIL] {get_federation_out_pos-b2} 404: No row found
federation_sender3 | 2022-10-06 20:37:05,676 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-6 - [TXN END] {get_federation_out_pos-b2} 0.001641 sec
federation_sender3 | 2022-10-06 20:37:05,679 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-6 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
federation_sender3 | 2022-10-06 20:37:05,806 - synapse.storage.txn - 690 - DEBUG - process_event_queue_for_federation-7 - [TXN START] {get_federation_out_pos-b3}
federation_sender3 | 2022-10-06 20:37:05,807 - synapse.storage.txn - 786 - DEBUG - process_event_queue_for_federation-7 - [TXN FAIL] {get_federation_out_pos-b3} 404: No row found
federation_sender3 | 2022-10-06 20:37:05,808 - synapse.storage.txn - 794 - DEBUG - process_event_queue_for_federation-7 - [TXN END] {get_federation_out_pos-b3} 0.001389 sec
federation_sender3 | 2022-10-06 20:37:05,809 - synapse.metrics.background_process_metrics - 242 - ERROR - process_event_queue_for_federation-7 - Background process 'process_event_queue_for_federation' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/federation/sender/__init__.py", line 353, in _process_event_queue_loop
    last_token = await self.store.get_federation_out_pos("events")
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/databases/main/stream.py", line 1090, in get_federation_out_pos
    return await self.db_pool.simple_select_one_onecol(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1665, in simple_select_one_onecol
    return await self.runInteraction(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 881, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1692, in _inlineCallbacks
    result = context.run(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 848, in _runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 976, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 969, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 710, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/synapse/storage/database.py", line 1718, in simple_select_one_onecol_txn
    raise StoreError(404, "No row found")
synapse.api.errors.StoreError: 404: No row found
Should I break this into a separate issue? It feels like it goes beyond the scope of the original issue now. I can add the logs from today in the same manner, but there wasn't anything else interesting in them. Just more of the same.

@realtyem
Copy link
Contributor Author

realtyem commented Oct 11, 2022 •
edited
Loading

I think I may have sorted this out, but it still raises some questions that may need to be addressed. The root of the problem appears to be configure_workers_and_start.py which is building an instance_map and the sharding configuration stuff, then not actually writing anything down. Adding them to my shared config manually made the exceptions disappear.

This might just be my deployment, I'll check more on it.

@realtyem
Copy link
Contributor Author

It was my deployment, but not completely. There is a method in configure_workers_and_start.py that is calling strip() and split(), but it's doing it out of order. If I have spaces in the string of worker types(like: "worker1, worker2, worker3" instead of "worker1,worker2,worker3") then it doesn't parse completely right when converting from string to object list to json(I think, python isn't my native language 😅).

Anyways, change

worker_types = worker_types_env.split(",")

to

worker_types = [x.strip() for x in worker_types_env.split(",")]

and it's all good. Just some sanitizing.

@realtyem
Copy link
Contributor Author

It's not the exact spot I thought it was. I've further tracked this down to line 448.

@realtyem realtyem mentioned this issue Oct 13, 2022
Merged
4 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Metrics metrics, measures, stuff we put in Prometheus X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Strip whitespace from worker types in Dockerfile-workers realtyem/synapse-old
2 participants
@realtyem @DMRobertson