Unable to completely delete machine when delete cluster

[pavlelee]

What happened:
When i delete cluster cr, tkestack sync delete machine, but the finalize immediately emptied. make machine delete k8s node error by cluster cr.

tke/pkg/platform/controller/cluster/deletion/cluster_deleter.go

Line 123 in c13ac23

err = d.deleteAllContent(ctx, cluster)

tke/pkg/platform/controller/cluster/deletion/cluster_deleter.go

Line 255 in c13ac23

deleteMachine,

tke/pkg/platform/controller/machine/deletion/machine_deleter.go

Line 294 in 5dcd1c2

cluster, err := typesv1.GetClusterByName(context.Background(), deleter.platformClient, machine.Spec.ClusterName)

What you expected to happen:
delete ok

How to reproduce it (as minimally and precisely as possible):
delete cluster maybe reproduce it

Anything else we need to know?:

Environment:

• TKE version:
• Global or business cluster:
• Kubernetes version (use kubectl version):
• Install addons:
• Others:

[pavlelee]

Design:
1、remove deleteMachine from cluster deletion
2、add machine ownerReference && ownerReference.blockOwnerDeletion=true
3、normal delete cluster，k8s gc will auto delete machine. if machine deleting, cluster keep terminating.

@huxiaoliang

[huxiaoliang]

@pavlelee I am ok on you design

[huxiaoliang]

@TomatoAres， please take a look this.

[pavlelee]

#1378

@TomatoAres I found new problem: call sdk without deleteOptions.propagationPolicy to delete cluster, clustercredentials/machines have not garbage collection. but @huxiaoliang confirm that tkestack work good without options, I test not work, you should confirm this.

[huxiaoliang]

@TomatoAres please update the latest status to this issue

[huxiaoliang]

@TomatoAres please append the root case for this issue

[TomatoAres]

@TomatoAres please append the root case for this issue

TKE machine controller doesn't watch delete machine operator.

Machine controller use kubernetes add terminating status to machine and this will trigger update operator.

Then machine controller to deal with this update callback, and the callback is linked to machine.status.phase.

So we need to check the machine.status.phase Not Terminating->Terminating, to execute a delete machine callback function.

[huxiaoliang]

@TomatoAres above analysis make sense, but still don't don't know why delete cluster xxx doesn't work but delete machine xxx works well, please correct me, thanks.

[TomatoAres]

@TomatoAres above analysis make sense, but still don't don't know why delete cluster xxx doesn't work but delete machine xxx works well, please correct me, thanks.

Root cause is deleting cluster make cluster terminating.

Delete machine callback has deleteNode function, but some cluster terminating will block node delete.

Sometimes time check will forbid events enqueue.