Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

ERROR - Failed to handle edu 'm.receipt': psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update #15936

Open
melroy89 opened this issue Jul 13, 2023 · 10 comments
Open

ERROR - Failed to handle edu 'm.receipt': psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update #15936

melroy89 opened this issue Jul 13, 2023 · 10 comments
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience S-Tolerable Minor significance, cosmetic issues, low or no impact to users. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.

Comments

@melroy89
Copy link

Description

Despite that issue: #3234 is closed. Is is not resolved for me.

I still suffer from errors: "ERROR - PUT-58315 - Failed to handle edu 'm.receipt'". See log output below, for more info.

Steps to reproduce

  • I'm just running the latest Synapse server (docker)
  • The error might get triggered when I join a new room in Elements?

Homeserver

matrix.melroy.org

Synapse Version

{"server_version":"1.87.0","python_version":"3.11.4"}

Installation Method

Docker (matrixdotorg/synapse)

Database

Postgresql V14

Workers

Single process

Platform

Ubuntu 22.04 (running Docker on top of that)

Configuration

Standard Matrix config

Relevant log output

2023-07-13 03:24:39,970 - synapse.federation.federation_server - 1470 - ERROR - PUT-58315 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

Anything else that would be useful to know?

No response
@reivilibre
Copy link
Contributor

do you get any more output showing what concurrent update is causing it? From memory I think there are usually a few more lines after that error message.

(Your issue is different to #3234.)

@reivilibre reivilibre changed the title ERROR - PUT-58315 - Failed to handle edu 'm.receipt' ERROR - PUT-58315 - Failed to handle edu 'm.receipt': psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update Jul 18, 2023
@reivilibre reivilibre changed the title ERROR - PUT-58315 - Failed to handle edu 'm.receipt': psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update ERROR - Failed to handle edu 'm.receipt': psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update Jul 18, 2023
@realtyem
Copy link
Contributor

From the Postgres side, it logs like this:

2023-07-18 15:55:17.396 CDT [6670] ERROR:  could not serialize access due to concurrent update
2023-07-18 15:55:17.396 CDT [6670] STATEMENT:  INSERT INTO receipts_graph (room_id, receipt_type, user_id, thread_id, event_ids, data) VALUES ('!<room_id_hash_thingy>:matrix.org', 'm.read', '<random remote user_id', 'main', '["$N5W<...event_id>?"]', '{"thread_id":"main","ts":1689712423103}') ON CONFLICT (room_id, receipt_type, user_id, thread_id)  DO UPDATE SET event_ids=EXCLUDED.event_ids, data=EXCLUDED.data

@melroy89
Copy link
Author

do you get any more output showing what concurrent update is causing it? From memory I think there are usually a few more lines after that error message.

(Your issue is different to #3234.)

No there are no other messages behind this error message..

2023-07-16 15:02:50,037 - synapse.federation.federation_server - 1470 - ERROR - PUT-181101 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-16 17:47:00,166 - synapse.federation.federation_server - 1470 - ERROR - PUT-190459 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-16 17:47:00,170 - synapse.federation.federation_server - 1470 - ERROR - PUT-190459 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-16 20:52:13,713 - synapse.federation.federation_server - 1470 - ERROR - PUT-206009 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-18 14:52:54,153 - synapse.federation.federation_server - 1470 - ERROR - PUT-331290 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-18 14:54:23,066 - synapse.federation.federation_server - 1470 - ERROR - PUT-331376 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-18 14:56:46,937 - synapse.federation.federation_server - 1470 - ERROR - PUT-331507 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-18 14:56:46,968 - synapse.federation.federation_server - 1470 - ERROR - PUT-331507 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

@melroy89
Copy link
Author

I enabled INFO log on the error log. Maybe this will give you more context?

2023-07-22 17:48:34,385 - synapse.storage.databases.main.event_push_actions - 1596 - INFO - rotate_notifs-284 - Rotating notifications, handling 0 rows
2023-07-22 17:48:34,387 - synapse.storage.databases.main.event_push_actions - 1684 - INFO - rotate_notifs-284 - Rotating notifications, deleted 0 push actions
2023-07-22 17:48:34,453 - synapse.util.caches.lrucache - 210 - INFO - LruCache._expire_old_entries-284 - Dropped 1 items from caches
2023-07-22 17:48:34,460 - synapse.http.matrixfederationclient - 690 - INFO - _maybe_retry_device_resync-284 - {GET-O-16064} [matrix.bootlin.com] Got response headers: 403 Forbidden
2023-07-22 17:48:34,460 - synapse.http.matrixfederationclient - 780 - WARNING - _maybe_retry_device_resync-284 - {GET-O-16064} [matrix.bootlin.com] Request failed: GET matrix-federation://matrix.bootlin.com/_matrix/federation/v1/user/devices/%406frzr%3Amatrix.bootlin.com: HttpResponseException('403: Forbidden')
2023-07-22 17:48:34,460 - synapse.handlers.device - 1279 - WARNING - _maybe_retry_device_resync-284 - Failed to handle device list update for @6frzr:matrix.bootlin.com: 403: Forbidden
2023-07-22 17:48:36,264 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9453 - Received txn 1689898786362 from midnightthoughts.space. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:36,269 - synapse.access.http.8008 - 461 - INFO - PUT-9453 - 65.21.116.72 - 8008 - {midnightthoughts.space} Processed request: 0.005sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689898786362 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:37,558 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9454 - Received txn 1683237755090 from chapo.chat. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:37,571 - synapse.access.http.8008 - 461 - INFO - GET-9450 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 12.653sec/0.001sec (0.000sec, 0.000sec) (0.001sec/0.001sec/1) 418B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898754_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:37,572 - synapse.access.http.8008 - 461 - INFO - PUT-9454 - 94.23.36.217 - 8008 - {chapo.chat} Processed request: 0.015sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.008sec/6) 11B 200 "PUT /_matrix/federation/v1/send/1683237755090 HTTP/1.1" "Synapse/1.79.0" [0 dbevts]
2023-07-22 17:48:37,749 - synapse.metrics._gc - 120 - INFO - sentinel - Collecting gc 1
2023-07-22 17:48:38,117 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9457 - Received txn 1689380252884 from cyber.bingo. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:38,127 - synapse.access.http.8008 - 461 - INFO - GET-9456 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.532sec/0.000sec (0.000sec, 0.000sec) (0.001sec/0.001sec/1) 416B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898755_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:38,128 - synapse.access.http.8008 - 461 - INFO - PUT-9457 - 188.34.180.209 - 8008 - {cyber.bingo} Processed request: 0.012sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.007sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689380252884 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:43,085 - synapse.crypto.keyring - 687 - INFO - PerspectivesKeyFetcher-359 - Requesting keys [_FetchKeyRequest(server_name='forkk.me', minimum_valid_until_ts=1690048123084, key_ids=['ed25519:a_pYEz'])] from notary server matrix.org
2023-07-22 17:48:43,235 - synapse.http.matrixfederationclient - 346 - INFO - PerspectivesKeyFetcher-359 - {POST-O-16069} [matrix.org] Completed request: 200 OK in 0.15 secs, got 436 bytes - POST matrix-federation://matrix.org/_matrix/key/v2/query
2023-07-22 17:48:43,240 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9460 - Received txn 1689998892540 from forkk.me. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:43,247 - synapse.access.http.8008 - 461 - INFO - GET-9459 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 5.094sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 414B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898756_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:43,248 - synapse.access.http.8008 - 461 - INFO - PUT-9460 - 45.56.76.228 - 8008 - {forkk.me} Processed request: 0.163sec/0.000sec (0.001sec, 0.001sec) (0.001sec/0.004sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689998892540 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:44,000 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9463 - Received txn 1689812551086 from hacklab.fi. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:44,008 - synapse.access.http.8008 - 461 - INFO - GET-9462 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.739sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 396B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898757_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:44,010 - synapse.access.http.8008 - 461 - INFO - PUT-9463 - 94.237.12.85 - 8008 - {hacklab.fi} Processed request: 0.011sec/0.000sec (0.000sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1689812551086 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:45,967 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9466 - Received txn 1689869472115 from matrix.org. (PDUs: 0, EDUs: 17)
2023-07-22 17:48:45,974 - synapse.handlers.typing - 398 - INFO - PUT-9466 - Got typing update from @mikeymongol:matrix.org: {'room_id': '!BZVTUuEiNmRcbFeLeI:matrix.org', 'typing': True, 'user_id': '@mikeymongol:matrix.org'}
2023-07-22 17:48:45,979 - synapse.access.http.8008 - 461 - INFO - GET-9465 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 1.943sec/0.003sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 659B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_335_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:45,987 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:45,991 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:45,992 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:45,992 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:45,996 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5bf} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:45,997 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5bf} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:46,002 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5c9} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:46,003 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
 3/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 3/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5ac} could not serialize access due to concurrent update
 4/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5c9} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,008 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
 3/5
2023-07-22 17:48:46,009 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,011 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:46,012 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 4/5
2023-07-22 17:48:46,012 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b8} could not serialize access due to concurrent update
 4/5
2023-07-22 17:48:46,015 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5cf} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:46,015 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,016 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
 0/5
2023-07-22 17:48:46,016 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5b7} could not serialize access due to concurrent update
 5/5
2023-07-22 17:48:46,019 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
 1/5
2023-07-22 17:48:46,020 - synapse.federation.federation_server - 1470 - ERROR - PUT-9466 - Failed to handle edu 'm.receipt'
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/federation/federation_server.py", line 1466, in on_edu
    await handler(origin, content)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 112, in _received_remote_receipt
    await self._handle_new_receipts(receipts)
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/receipts.py", line 120, in _handle_new_receipts
    res = await self.store.insert_receipt(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 798, in insert_receipt
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 930, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 896, in _runInteraction
    result = await self.runWithConnection(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1025, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1018, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 758, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/databases/main/receipts.py", line 843, in _insert_graph_receipt_txn
    self.db_pool.simple_upsert_txn(
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1291, in simple_upsert_txn
    return self.simple_upsert_txn_native_upsert(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 1431, in simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 417, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/database.py", line 469, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.SerializationFailure: could not serialize access due to concurrent update

2023-07-22 17:48:46,022 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5dc} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:46,022 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
 2/5
2023-07-22 17:48:46,029 - synapse.access.http.8008 - 461 - INFO - GET-9468 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 0.020sec/0.001sec (0.001sec, 0.000sec) (0.001sec/0.010sec/3) 735B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_336_1760506_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:46,031 - synapse.storage.txn - 765 - WARNING - PUT-9466 - [TXN OPERROR] {insert_graph_receipt-b5d6} could not serialize access due to concurrent update
 3/5
2023-07-22 17:48:46,036 - synapse.access.http.8008 - 461 - INFO - PUT-9466 - 176.126.240.158 - 8008 - {matrix.org} Processed request: 0.070sec/0.000sec (0.027sec, 0.011sec) (0.062sec/0.379sec/76) 11B 200 "PUT /_matrix/federation/v1/send/1689869472115 HTTP/1.1" "Synapse/1.88.0 (b=matrix-org-hotfixes,b89e8d695d)" [0 dbevts]
2023-07-22 17:48:46,746 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9471 - Received txn 1689697704599 from gnxlxnxx.space. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:46,751 - synapse.access.http.8008 - 461 - INFO - PUT-9471 - 185.232.68.89 - 8008 - {gnxlxnxx.space} Processed request: 0.005sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689697704599 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:48:51,324 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9473 - Received txn 1690023590890 from envs.net. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:51,329 - synapse.access.http.8008 - 461 - INFO - PUT-9473 - 188.40.204.226 - 8008 - {envs.net} Processed request: 0.005sec/0.000sec (0.002sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1690023590890 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:51,835 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9474 - Received txn 1689836865658 from kittenface.studio. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:51,840 - synapse.access.http.8008 - 461 - INFO - PUT-9474 - 62.63.215.235 - 8008 - {kittenface.studio} Processed request: 0.005sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.003sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689836865658 HTTP/1.1" "Synapse/1.88.0" [0 dbevts]
2023-07-22 17:48:52,436 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9475 - Received txn 1686843884444 from neat.chat. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:52,445 - synapse.access.http.8008 - 461 - INFO - GET-9470 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 6.390sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 415B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898758_336_1760522_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:48:52,446 - synapse.access.http.8008 - 461 - INFO - PUT-9475 - 5.161.59.30 - 8008 - {neat.chat} Processed request: 0.011sec/0.000sec (0.002sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1686843884444 HTTP/1.1" "Synapse/1.85.2" [0 dbevts]
2023-07-22 17:48:58,510 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9478 - Received txn 1689570710451 from discuss.online. (PDUs: 0, EDUs: 1)
2023-07-22 17:48:58,513 - synapse.access.http.8008 - 461 - INFO - PUT-9478 - 138.197.80.206 - 8008 - {discuss.online} Processed request: 0.003sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.001sec/2) 11B 200 "PUT /_matrix/federation/v1/send/1689570710451 HTTP/1.1" "Synapse/1.87.0" [0 dbevts]
2023-07-22 17:49:04,264 - synapse.federation.transport.server.federation - 105 - INFO - PUT-9479 - Received txn 1685925939747 from finallycoffee.eu. (PDUs: 0, EDUs: 1)
2023-07-22 17:49:04,273 - synapse.access.http.8008 - 461 - INFO - GET-9477 - 192.168.1.1 - 8008 - {@melroy:melroy.org} Processed request: 11.803sec/0.001sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 408B 200 "GET /_matrix/client/r0/sync?filter=4&timeout=30000&since=s8272075_8898759_336_1760522_155100_145_497503_911898_0_39 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.35 Chrome/114.0.5735.106 Electron/25.1.0 Safari/537.36" [0 dbevts]
2023-07-22 17:49:04,275 - synapse.access.http.8008 - 461 - INFO - PUT-9479 - 78.94.116.221 - 8008 - {finallycoffee.eu} Processed request: 0.011sec/0.000sec (0.001sec, 0.000sec) (0.001sec/0.006sec/5) 11B 200 "PUT /_matrix/federation/v1/send/1685925939747 HTTP/1.1" "Synapse/1.84.1" [0 dbevts]
2023-07-22 17:49:04,382 - synapse.storage.databases.main.event_push_actions - 1306 - INFO - rotate_notifs-285 - Rotating notifications
2023-07-22 17:49:04,383 - synapse.storage.databases.main.event_push_actions - 1510 - INFO - rotate_notifs-285 - Rotating notifications up to: 8272075
2023-07-22 17:49:04,384 - synapse.storage.databases.main.event_push_actions - 1596 - INFO - rotate_notifs-285 - Rotating notifications, handling 0 rows
2023-07-22 17:49:04,386 - synapse.storage.databases.main.event_push_actions - 1684 - INFO - rotate_notifs-285 - Rotating notifications, deleted 0 push actions
2023-07-22 17:49:04,446 - synapse.handlers.presence - 795 - INFO - persist_presence_changes-141 - Persisting 10 unpersisted presence updates
2023-07-22 17:49:04,448 - synapse.federation.sender - 979 - INFO - wake_destinations_needing_catchup-142 - Destination gaw.social has outstanding catch-up, waking up.
2023-07-22 17:49:04,449 - synapse.federation.sender.per_destination_queue - 486 - INFO - federation_transaction_transmission_loop-11754 - Catching up destination gaw.social with 1 PDUs
2023-07-22 17:49:04,450 - synapse.federation.sender.per_destination_queue - 568 - INFO - federation_transaction_transmission_loop-11754 - Catching up rooms to gaw.social: '!gWiVYCURkxIapWZaGy:ipfs.io'
2023-07-22 17:49:04,450 - synapse.federation.sender.transaction_manager - 121 - INFO - federation_transaction_transmission_loop-11754 - TX [gaw.social] {1690039575215} Sending transaction [1690039575215], (PDUs: 1, EDUs: 0)
2023-07-22 17:49:04,452 - synapse.util.caches.lrucache - 210 - INFO - LruCache._expire_old_entries-285 - Dropped 1 items from caches

@MatMaul MatMaul added S-Tolerable Minor significance, cosmetic issues, low or no impact to users. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience labels Jul 27, 2023
@Lowxorx
Copy link

Lowxorx commented Aug 28, 2023

Same error here, with the same setup. Let me know if I can help with logs or anything.

@savyajha
Copy link
Contributor

savyajha commented Sep 1, 2023

I've got a similar issue on my end. If it helps, I've also got the sliding sync proxy running.

@Kimiblock
Copy link

Same issue.

Disabling siding sync proxy doesn’t help.

In fact, when I first knew about this error, I didn’t have any siding sync proxy running.

@DMRobertson
Copy link
Contributor

https://sentry.tools.element.io/organizations/element/issues/69241 looks like an instance of this.

Looks like the error comes from

synapse/synapse/federation/federation_server.py

Line 1481 in 8c3bcea

logger.exception("Failed to handle edu %r", edu_type)

or

synapse/synapse/federation/federation_server.py

Line 1500 in 8c3bcea

logger.exception("Failed to handle edu %r", edu_type)

Would need some investigation to work out what the database is unhappy about.

@lordievader
Copy link

Got the same issue. More or less the same setup. Synapse 1.89.0 running in docker with a PostgreSQL 15 database on another virtual machine. If you need logs or tests run let me know.

@realtyem
Copy link
Contributor

Please check again if this still happens after 1.93.0 has been released. A fix is included that may stop this instance occurring.

@matrixbot matrixbot mentioned this issue Dec 22, 2023
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Database DB stuff like queries, migrations, new/remove columns, indexes, unexpected entries in the db O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience S-Tolerable Minor significance, cosmetic issues, low or no impact to users. T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@melroy89 @realtyem @MatMaul @lordievader @savyajha @DMRobertson @reivilibre @Lowxorx @Kimiblock