manage locked events during scale-down

[tirumaraiselvan]

If hasura instances are scaled down, then it is possible that events that were being processed by aborting instances stay locked and stuck in pending.

[tirumaraiselvan]

A potential fix could be to unlock events during managed scale down i.e. graceful shutdown.

@0x777 @jberryman Thoughts?

[jberryman]

Yep I think we want to have a graceful shutdown that unlocks events.

Then additionally we could either

• document that users might wish to clean up old messages stuck in pending manually (in case of unclean shutdown, network failures etc)
• automatically "redeliver" after a particular time period elapses

I'm thinking for prior art about how rabbitMQ works in the case of un-ACKed messages