Expected behavior

USR2 should not lead to a heartbeat fight, and only the new process should heartbeat.

Current behavior

USR2 without a graceful shutdown leads to a situation where both the new and the old process heartbeat. When trying to apply new labels, this causes a situation where the node flips between the old and new labels.

Background

To minimize disruption to active users when updating a node label, a USR2 signal can be sent. This forks a new process, but does not initiate the graceful shutdown of the parent process.

In a deployment scenario where the Teleport process must remain in the foreground, and not fork/exit, the HUP signal is not appropriate. When the original process exits, the entire process group (including the new child process) can be killed. This means that USR2 is preferred to load configuration changes while minimizing user disruptions.

The downside is that USR2 only passes new connections to the child process. Both the new and old processes continue to heartbeat. This leads to the new label appearing, and then disappearing until a full restart is performed, so there is only one remaining teleport process.

Info

• Teleport version - 13.4.2
• Recreate Steps - change labels in teleport.yaml on any teleport process that heartbeats (ssh node is easiest). Send USR2. Watch the output of tctl get node/name