Skip to content

A failed log append in a worker leaves records and writes pending forever #53

Description

@SimonHeybrock

If appending an event to the log raises in a worker (for example a full disk), the executor swallows the exception and the backend's state stays as before the append. A record whose Finished failed to append stays pending; with persist (#52), a write whose Written failed to append stays pending too. Waiters then hang: client.output of another client, client.wait, and close(), which waits for pending writes.

The log is the backend's only storage, so a failed append means the backend no longer knows its own state. Options:

  • the backend stops: every waiter and every later call raises with the append's error;
  • the worker retries the append, then stops as above.

Found in the review of #52.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions