[Feature Request] Write spec for graceful worker shutdown

[bergundy]

Different SDKs implement graceful shutdown differently.

TypeScript traditionally treated the graceful shutdown period as "give up on shutdown after this period" and recently changed the semantics to be inline with Python / .NET where graceful shutdown refers to when to issue cancellations to running activities.

By default all Core based SDKs now will wait for activities to complete before returning from Worker.run.

Go does not wait for activities to complete today and should change to be consistent with Core based SDKs.
Need to verify what Java does here.

[edmondop]

Do you have by chance the issue/review on Typescript so I can start to look there and write a draft for the spec? Also if you have spec of other features that I could use as a reference that would be useful

[Quinn-With-Two-Ns]

Most of our specs are just written on the feature test for them for example https://github.com/temporalio/features/blob/main/features/activity/cancel_try_cancel/README.md\

[edmondop]

@Quinn-With-Two-Ns after our discussion, I am confused by the title of the issue. It looks like the spec already exist, and we are just missing the feature tests? https://github.com/temporalio/features/blob/main/features/activity/shutdown/README.md

Maybe the right thing to do is just to:

• write the tests
• fix the non compliant SDK

[cretz]

Note, https://github.com/temporalio/features/blob/main/features/activity/shutdown/README.md may be out of date and may not accurately reflect how we want to approach this with regards to graceful shutdown. It may be revisited as part of this issue.

[saward]

I've experienced an issue where when a worker is given the kill signal (SIGTERM I'm guessing), the server doesn't seem to know that the activity is no longer running in the worker, so doesn't retry until it times out. Depending on timeout settings, this can take a long time. I think it's related to this feature request.

Is there a way right now with Go SDK to handle the killing of a worker, so that the temporal server knows immediately that the in-progress activities now need to be given to new workers?

[cretz]

@saward - in Go, activity contexts are canceled on graceful worker shutdown (after optional grace period) and there are other values in the activity package that show a worker is shutting down. The retry policy governs when failed activities are retried (regardless of reason for failure, including caused by worker shutdown's context cancel). There are ways to capture this situation and instead return a special error that retries sooner.

But this GitHub issue is just about properly documenting. For Go-specific questions/support, can ask on Slack or our forums (see https://temporal.io/community), or open a cloud ticket if you're a cloud user.

[JCMais]

Reading the link above I see no mention of a grace period being given to activities before the activity gets the WORKER SHUTDOWN cancellation signal, is that really the case?

What is the grace period used for in that case? (e.g on TypeScript, where the worker has a shutdownGracePeriod option)

[cretz]

Reading the link above I see no mention of a grace period being given to activities before the activity gets the WORKER SHUTDOWN cancellation signal, is that really the case?

https://typescript.temporal.io/api/interfaces/worker.WorkerOptions#shutdowngracetime mentions "In-flight activities will be cancelled after this period" (emphasis mine)

This is a language specific question at this time (though we are hoping to write consistent spec/docs as part of this issue). Can ask in Slack or forums for the specific language concern.