Skip to content
This repository was archived by the owner on Apr 26, 2024. It is now read-only.
This repository was archived by the owner on Apr 26, 2024. It is now read-only.

SerializationFailure when rotating push actions #13117

@erikjohnston

Description

@erikjohnston

We recently changed things so we rotate notifications almost immediately, rather than waiting a day. This interacts badly with people sending read receipts, as that transaction attempts to modify the push summary tables.

We need to fix this or revert before we make another release.

Broke by #13005

Stack trace:

SerializationFailure: could not serialize access due to concurrent update

  File "synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "synapse/storage/databases/main/event_push_actions.py", line 806, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
  File "synapse/storage/database.py", line 835, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/storage/database.py", line 807, in _runInteraction
    result = await self.runWithConnection(
  File "synapse/storage/database.py", line 930, in runWithConnection
    return await make_deferred_yieldable(
  File "twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "synapse/storage/database.py", line 923, in inner_func
    return func(db_conn, *args, **kwargs)
  File "synapse/storage/database.py", line 671, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "synapse/storage/databases/main/event_push_actions.py", line 850, in _rotate_notifs_txn
    self._rotate_notifs_before_txn(txn, rotate_to_stream_ordering)
  File "synapse/storage/databases/main/event_push_actions.py", line 950, in _rotate_notifs_before_txn
    txn.execute_batch(
  File "synapse/storage/database.py", line 329, in execute_batch
    self._do_execute(
  File "synapse/storage/database.py", line 395, in _do_execute
    return func(sql, *args, **kwargs)
  File "synapse/storage/database.py", line 330, in <lambda>
    lambda the_sql: execute_batch(self.txn, the_sql, args), sql
  File "psycopg2/extras.py", line 1187, in execute_batch
    cur.execute(b";".join(sqls))

Sentry: https://sentry.tools.element.io/organizations/element/issues/924/

Metadata

Metadata

Assignees

Labels

S-MajorMajor functionality / product severely impaired, no satisfactory workaround.T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.X-RegressionSomething broke which worked on a previous releaseX-Release-BlockerMust be resolved before making a release

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions