Skip to content

Improve logging after restate is re-started. #52

@igalshilman

Description

@igalshilman

When hard killing restate and then recovering, it tries to recover previously undelivered invocations (I think), and it causes our SDK to log a lot ERRORs and WARNs like the snippet below.

The error message is not very informative, and most likely to appear every time restate is re-started.
Since that is a common situation, can we improve that log message?

[restate] [2023-05-20T15:23:34.073Z] WARN: Error in input stream: Error: State machine is closed. Canceling all execution
    at DurableExecutionStateMachine.failIfClosed (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:606:19)
    at DurableExecutionStateMachine.checkIfInReplay (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:594:14)
    at DurableExecutionStateMachine.handleSetStateMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:561:14)
    at DurableExecutionStateMachine.onIncomingMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:485:22)
    at Transform.<anonymous> (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/connection/restate_duplex_stream.js:26:13)
    at Transform.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at Transform.transform [as _transform] (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/io/decoder.js:64:38)
[restate] [2023-05-20T15:23:34.074Z] WARN: Error in input stream: Error: State machine is closed. Canceling all execution
    at DurableExecutionStateMachine.failIfClosed (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:606:19)
    at DurableExecutionStateMachine.checkIfInReplay (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:594:14)
    at DurableExecutionStateMachine.handleSetStateMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:561:14)
    at DurableExecutionStateMachine.onIncomingMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:485:22)
    at Transform.<anonymous> (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/connection/restate_duplex_stream.js:26:13)
    at Transform.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at Transform.transform [as _transform] (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/io/decoder.js:64:38)
[restate] [2023-05-20T15:23:34.124Z] WARN: Error in input stream: Error: State machine is closed. Canceling all execution
    at DurableExecutionStateMachine.failIfClosed (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:606:19)
    at DurableExecutionStateMachine.checkIfInReplay (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:594:14)
    at DurableExecutionStateMachine.handleSetStateMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:561:14)
    at DurableExecutionStateMachine.onIncomingMessage (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/state_machine.js:485:22)
    at Transform.<anonymous> (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/connection/restate_duplex_stream.js:26:13)
    at Transform.emit (node:events:513:28)
    at addChunk (node:internal/streams/readable:324:12)
    at readableAddChunk (node:internal/streams/readable:297:9)
    at Readable.push (node:internal/streams/readable:234:10)
    at Transform.transform [as _transform] (/home/igal/work/jepsen.restate/service/node_modules/@restatedev/restate-sdk/dist/io/decoder.js:64:38)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions