If a client cancels SuspendActor while atelet is still checkpointing, ateapi releases the actor's lease.
A retry then starts a second checkpoint on a sandbox that has just saved and exited. ateapi marks the actor
CRASHED, and the first checkpoint completes a second later.
Seen with 1000 actors suspended at once by a client with a 30 s per-attempt deadline:
- The client calls
SuspendActor. atelet's Checkpoint takes 30.85 s: 4.24 s checkpoint, 2.75 s upload,
and about 24 s queued behind 125 other checkpoints on the node.
- At 30 s the client cancels. ateapi logs
workflow failed at step CallAteletSuspend: context canceled
and releases the lease.
- The client retries 50 ms later, and ateapi starts a second suspend.
- The second
runsc checkpoint fails with exit status 128 on the exited sandbox. ateapi logs
Setting Actor to crashed.
- 1.1 s later the first checkpoint finishes and its snapshot is persisted. The actor is already CRASHED.
In one run whose suspend took 30.1 s, 45 of 1000 actors ended CRASHED this way.
Fix
ateapi must not start a second checkpoint while one is in flight for the actor. A client cancel should not
release the lease while the atelet call continues.
If a client cancels
SuspendActorwhile atelet is still checkpointing, ateapi releases the actor's lease.A retry then starts a second checkpoint on a sandbox that has just saved and exited. ateapi marks the actor
CRASHED, and the first checkpoint completes a second later.
Seen with 1000 actors suspended at once by a client with a 30 s per-attempt deadline:
SuspendActor. atelet's Checkpoint takes 30.85 s: 4.24 s checkpoint, 2.75 s upload,and about 24 s queued behind 125 other checkpoints on the node.
workflow failed at step CallAteletSuspend: context canceledand releases the lease.
runsc checkpointfails with exit status 128 on the exited sandbox. ateapi logsSetting Actor to crashed.In one run whose suspend took 30.1 s, 45 of 1000 actors ended CRASHED this way.
Fix
ateapi must not start a second checkpoint while one is in flight for the actor. A client cancel should not
release the lease while the atelet call continues.