Router - Local Dev Fast Ctrl-C Handling

[huntharo]

Motivations

• Currently the router hangs on Ctrl-C for up to a minute
• There is no cancellation token to cancel all long-running tasks gracefully
• Note: graceful exit under ECS works fine because ECS is deregistering tasks with the ALB before shutting down the containers

Overview

• Add signal handlers that signal a cancellation token when other processes have wrapped up
• Start failing health checks after receiving SIGTERM
• Wait a bit then signal the cancel token to break out of the scheduled loops like metrics, queue dispatcher, and rebalancer

[huntharo]

Testing with ECS, scaling up and down the containers using oha for load testing, shows no dropped requests or errors while running at 12,000 RPS.

./oha-linux-amd64 -z 10sm -c 400 https://lambdadispatch.ghpublic.pwrdrvr.com/ping