Skip to content

Restarting nats stops stream processing forever #3177

Closed
@matrixbot

Description

This issue was originally created by @recht at matrix-org/dendrite#3177.

Background information

  • Dendrite version or git SHA: 0.13.1
  • SQLite3 or Postgres?: Postgres
  • Running in Docker?: Yes
  • go version: 1.19.7
  • Client used (if applicable):

Description

  • What is the problem: Dendrite stops processing events if nats is restarted.
  • Who is affected: Everybody
  • How is this bug manifesting: Restart the entire nats cluster in a non-rolling fashion. This will trigger a nats: Server Shutdown error which exits the JetStreamConsumers, but leaves the rest of the system running. Only remediation seems to be to restart the server.
  • When did this first appear:

Steps to reproduce

  • Start Dendrite
  • Kill all nats servers simultaneously
  • Messages are no longer processed

Ideally Dendrite would reconnect once nats is available again. Alternatively the service should exit, or at the very list trigger a signal somewhere that streams are no longer being processed.

Metadata

Assignees

No one assigned

    Labels

    T-DefectBugs, crashes, hangs, security vulnerabilities, or other reported issues.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions