SYSTEM PRESHUTDOWN command for graceful shutdown swarm node

[ianton-ru]

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

SYSTEM PRESHUTDOWN command for graceful shutdown swarm node

Documentation entry for user-facing changes

Solved #759
New command SYSTEM PRESHUTDOWN.
Scenario:
We want to scale down swarm cluster. On node which we want to shutdown we call SYSTEM PRESHUTDOWN, after that node stops to accept new distributed commands. It can still processed objects which is started processed before SYSTEM PRESHUTDOWN. When all that objects successfully processed, we can kill that node without any errors or lost data in responses on initiator.

After SYSTEM PRESHUTDOWN on swarm node:

• unregister node from autodiscovery clusters if exists
• stop getting new tasks for objectStorageCluster-family functions (s3Cluster/icebergCluster/etc.)

On initiator node:

• for all distributed requests with setting skip_unavailable_shards=true unexpected closing of socket is legal if no data packets were accepted before. This allow to shutdown non-autodiscovery node too.

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All with Aarch64
• All Regression
• Disable CI Cache

[ilejn]

Is it true that operations prohibited in PreShutdown phase (e.g getting new tasks), are allowed in Shutdown phase?
If yes, is it correct?

[ianton-ru]

As I understand ClickHouse does not have specific shutdown phase. On SYSTEM SHUTDOWN just calls kill(0, SIGTERM). Without PRESHUTDOW this caused error on initiator as well as already taken but unfinished tasks.

[ilejn]

Then what is the purpose of shutdown_called flag?

From the first glance I would expect that all checks that are true for preshutdown_called should be true for shutdown_called as well.

[ianton-ru]

Ah, understood, flag set in destructor.
Yes, make sense to set preshutdown there too.

[arthurpassos]

As I understand, regular queries would still be processed. Literally the only thing that's stopped is the "swarm node" work. Wouldn't it make more sense to rename the command to something more meaningful? (e.g UNREGISTER FROM SWARM)

This is just a question, not a change request

[ianton-ru]

As I understand, regular queries would still be processed. Literally the only thing that's stopped is the "swarm node" work. Wouldn't it make more sense to rename the command to something more meaningful? (e.g UNREGISTER FROM SWARM)

It's a topic to discussion.
For now I don't have other use cases when we need and can do graceful shutdown from clickhouse side (shutdown main node is not that case, because errors are on client side in code which are out our control. May be possible to add special error "New query can't be executed, node prepared to shutdown", but it is useless without support on client side). But if we have one, I think that better to have single command PRESUTDOWN to prepare shutdown instead of several commands UNREGISTER FROM SWARM, STOP DOING SOMETHING, CLOSE SOMETHING ELSE, because all this hypothetical command make sense only when called in single pack. And better to add new logic under PRESHUTDOWN command.