Allow workers to terminate gracefully by sending `WorkerTerminate` message

[mhoeger]

We should send a WorkerTerminate message to from host to worker to allow the worker to terminate itself gracefully instead of immediately killing it.

Step 1 to allowing #2153.

cc: @pragnagopa

[peterhuene]

Is there any update to this issue?

I notice that my worker process is rudely terminated with SIGKILL when gracefully shutting down the host with Ctrl-C.

Unfortunately, the host notices that the process errored and attempts to restart it. The restarted worker attempts to connect but fails because RPC was shut down and exits with an error. This causes the host to notice again that the process errored and restarts it a second time. This could repeat a number of times depending on how fast the rest of the host shuts down.

Example output with host version 2.0.12285.0:

[2/12/19 3:31:14 AM] Shuttingdown Rpc Channels Manager
[2/12/19 3:31:14 AM] Shutting down language worker channel for runtime:Rust
[2/12/19 3:31:14 AM] Language Worker Process exited.
[2/12/19 3:31:14 AM] workers/rust/rust_worker exited with code 137
[2/12/19 3:31:14 AM]  .
[2/12/19 3:31:14 AM] Starting language worker process:workers/rust/rust_worker  run --host 127.0.0.1 --port 58568 --workerId 8830d120-0f09-4304-8310-88286e4fc6b8 --requestId 85ef34b2-7879-4049-ba30-ca6863bc05a1 --grpcMaxMessageLength 134217728
[2/12/19 3:31:14 AM] workers/rust/rust_worker process with Id=46248 started
[2/12/19 3:31:14 AM] Connecting to Azure Functions host at 127.0.0.1:58568.
[2/12/19 3:31:14 AM] thread '<unnamed>thread 'main' panicked at '' panicked at 'failed to receive message: RpcFailure(RpcStatus { status: Unavailable, details: Some("Connect Failed") }).failed to send message to host: RpcFinished(Some(RpcStatus { status: Unavailable, details: Some("Connect Failed") }))', ', /Users/peterhuene/src/azure-functions-rs/azure-functions/src/rpc/client.rs:src/libcore/result.rs:1361009::335
[2/12/19 3:31:14 AM] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[2/12/19 3:31:14 AM] Language Worker Process exited.
[2/12/19 3:31:14 AM] workers/rust/rust_worker exited with code 101
[2/12/19 3:31:14 AM]  thread '<unnamed>thread 'main' panicked at '' panicked at 'failed to receive message: RpcFailure(RpcStatus { status: Unavailable, details: Some("Connect Failed") }).failed to send message to host: RpcFinished(Some(RpcStatus { status: Unavailable, details: Some("Connect Failed") }))', ', /Users/peterhuene/src/azure-functions-rs/azure-functions/src/rpc/client.rs:src/libcore/result.rs:1361009::335.
[2/12/19 3:31:14 AM] Starting language worker process:workers/rust/rust_worker  run --host 127.0.0.1 --port 58568 --workerId fbd1f024-2f43-457e-adca-6d29a1dd7e45 --requestId d6465cec-b7bf-4ea9-b0f3-3895a195bf47 --grpcMaxMessageLength 134217728
[2/12/19 3:31:14 AM] workers/rust/rust_worker process with Id=46249 started
[2/12/19 3:31:14 AM] Stopping host...
[2/12/19 3:31:14 AM] Connecting to Azure Functions host at 127.0.0.1:58568.
[2/12/19 3:31:14 AM] thread 'thread 'main<unnamed>' panicked at 'failed to send message to host: RpcFinished(Some(RpcStatus { status: Unavailable, details: Some("Connect Failed") }))', ' panicked at 'failed to receive message: RpcFailure(RpcStatus { status: Unavailable, details: Some("Connect Failed") }).src/libcore/result.rs', :/Users/peterhuene/src/azure-functions-rs/azure-functions/src/rpc/client.rs1009::1365:
[2/12/19 3:31:14 AM] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[2/12/19 3:31:14 AM] Language Worker Process exited.
[2/12/19 3:31:14 AM] workers/rust/rust_worker exited with code 101
[2/12/19 3:31:14 AM]  .
[2/12/19 3:31:14 AM] Host shutdown completed.

The result is a lot of errors in the shutdown output that confuses my users.

Would it be possible to at least stop watching for worker process termination (and thus restarting it unnecessarily) prior to killing it when shutting down the worker channel?

Thanks!

[liliankasem]

Duplicate of #2308