Universal dataplane objects are not cleaned up during graceful terminating

[jijiechen]

Kuma Version

2.10.1

Describe the bug

On a universal zone deployment, when the DP is gracefully terminated, the dataplane object is not removed automatically and it is marked as offline immediately.

When there is no service probe policies, this can cause the DP to still be included as an endpoint in the Envoy Cluster of its containing service: eventually causing traffic to down for long.

To Reproduce

1. Run a CP container, connecting to a postgres store
2. Run a DP
3. Send SIGTERM to the kuma-dp process into the DP container
4. Check
   1. Mesh GUI
   2. CP logs

Expected behavior

The dataplane object is automatically removed as described in the document https://kuma.io/docs/2.10.x/production/dp-config/dpp-on-universal/#leaving-the-mesh

Additional context (optional)

CP logs:

2025-04-15T03:15:52.943Z	ERROR	xds.dp-lifecycle	could not check if proxy connected to another CP	{"proxyType": "dataplane", "proxyKey": {"Mesh":"upgrade","Name":"echo-server-1"}, "streamID": 1, "error": "could not get insight to determine if we can delete proxy object: failed to execute query: SELECT spec, version, creation_time, modification_time, labels, status FROM resources WHERE name=$1 AND mesh=$2 AND type=$3;: context canceled", "errorVerbose": "context canceled\nfailed to execute query: SELECT spec, version, creation_time, modification_time, labels, status FROM resources WHERE name=$1 AND mesh=$2 AND type=$3;\ngithub.com/kumahq/kuma/pkg/plugins/resources/postgres.(*pgxResourceStore).Get\n\tgithub.com/kumahq/kuma/pkg/plugins/resources/postgres/pgx_store.go:264\ngithub.com/kumahq/kuma/pkg/core/resources/store.(*paginationStore).Get\n\tgithub.com/kumahq/kuma/pkg/core/resources/store/pagination_store.go:42\ngithub.com/kumahq/kuma/pkg/metrics/store.(*MeteredStore).Get\n\tgithub.com/kumahq/kuma/pkg/metrics/store/store.go:76\ngithub.com/kumahq/kuma/pkg/core/resources/store.(*customizableResourceStore).Get\n\tgithub.com/kumahq/kuma/pkg/core/resources/store/customizable_store.go:36\ngithub.com/kumahq/kuma/pkg/core/resources/manager.(*resourcesManager).Get\n\tgithub.com/kumahq/kuma/pkg/core/resources/manager/manager.go:44\ngithub.com/kumahq/kuma/pkg/core/resources/manager.(*customizableResourceManager).Get\n\tgithub.com/kumahq/kuma/pkg/core/resources/manager/customizable_manager.go:43\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).proxyConnectedToAnotherCP\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:251\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).deregister\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:200\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).OnProxyDisconnected\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:104\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*xdsCallbacks).OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_callbacks.go:71\ngithub.com/kumahq/kuma/pkg/util/xds/v3.(*adapterCallbacks).OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/util/xds/v3/callbacks.go:35\ngithub.com/kumahq/kuma/pkg/util/xds/v3.CallbacksChain.OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/util/xds/v3/callbacks_chain.go:30\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*streamWrapper).shutdown\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:134\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/xds.go:49\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:168\ngithub.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:186\ngithub.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:190\ngithub.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/envoy/service/discovery/v3/ads_grpc.pb.go:137\ngithub.com/kumahq/kuma/pkg/metrics.(*metrics).GRPCServerInterceptors.(*ServerMetrics).StreamServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1695\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1809\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1029\nruntime.goexit\n\truntime/asm_arm64.s:1223\ncould not get insight to determine if we can delete proxy object\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).proxyConnectedToAnotherCP\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:258\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).deregister\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:200\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).OnProxyDisconnected\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:104\ngithub.com/kumahq/kuma/pkg/xds/server/callbacks.(*xdsCallbacks).OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_callbacks.go:71\ngithub.com/kumahq/kuma/pkg/util/xds/v3.(*adapterCallbacks).OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/util/xds/v3/callbacks.go:35\ngithub.com/kumahq/kuma/pkg/util/xds/v3.CallbacksChain.OnStreamClosed\n\tgithub.com/kumahq/kuma/pkg/util/xds/v3/callbacks_chain.go:30\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*streamWrapper).shutdown\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:134\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/xds.go:49\ngithub.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:168\ngithub.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:186\ngithub.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:190\ngithub.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler\n\tgithub.com/envoyproxy/go-control-plane@v0.13.0/envoy/service/discovery/v3/ads_grpc.pb.go:137\ngithub.com/kumahq/kuma/pkg/metrics.(*metrics).GRPCServerInterceptors.(*ServerMetrics).StreamServerInterceptor.func1\n\tgithub.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1695\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1809\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/grpc@v1.67.1/server.go:1029\nruntime.goexit\n\truntime/asm_arm64.s:1223"}
github.com/kumahq/kuma/pkg/xds/server/callbacks.(*DataplaneLifecycle).OnProxyDisconnected
	github.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go:104
github.com/kumahq/kuma/pkg/xds/server/callbacks.(*xdsCallbacks).OnStreamClosed
	github.com/kumahq/kuma/pkg/xds/server/callbacks/dataplane_callbacks.go:71
github.com/kumahq/kuma/pkg/util/xds/v3.(*adapterCallbacks).OnStreamClosed
	github.com/kumahq/kuma/pkg/util/xds/v3/callbacks.go:35
github.com/kumahq/kuma/pkg/util/xds/v3.CallbacksChain.OnStreamClosed
	github.com/kumahq/kuma/pkg/util/xds/v3/callbacks_chain.go:30
github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*streamWrapper).shutdown
	github.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:134
github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).process
	github.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/xds.go:49
github.com/envoyproxy/go-control-plane/pkg/server/sotw/v3.(*server).StreamHandler
	github.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/sotw/v3/server.go:168
github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamHandler
	github.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:186
github.com/envoyproxy/go-control-plane/pkg/server/v3.(*server).StreamAggregatedResources
	github.com/envoyproxy/go-control-plane@v0.13.0/pkg/server/v3/server.go:190
github.com/envoyproxy/go-control-plane/envoy/service/discovery/v3._AggregatedDiscoveryService_StreamAggregatedResources_Handler
	github.com/envoyproxy/go-control-plane@v0.13.0/envoy/service/discovery/v3/ads_grpc.pb.go:137
github.com/kumahq/kuma/pkg/metrics.(*metrics).GRPCServerInterceptors.(*ServerMetrics).StreamServerInterceptor.func1
	github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121
google.golang.org/grpc.(*Server).processStreamingRPC
	google.golang.org/grpc@v1.67.1/server.go:1695
google.golang.org/grpc.(*Server).handleStream
	google.golang.org/grpc@v1.67.1/server.go:1809
google.golang.org/grpc.(*Server).serveStreams.func2.1
	google.golang.org/grpc@v1.67.1/server.go:1029

[jijiechen]

It's is caused by the context passed into proxyConnectedToAnotherCP and resManager.Delete has been canceled because the DP is disconnecting.

kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go

Lines 190 to 198 in b1c0b13

	if connected, err := d.proxyConnectedToAnotherCP(ctx, proxyType, proxyKey, log); err != nil {
	log.Error(err, "could not check if proxy connected to another CP")
	return
	} else if connected {
	return
	}
	log.Info("deregister proxy")
	if err := d.resManager.Delete(ctx, proxyResource(proxyType), store.DeleteBy(proxyKey)); err != nil {

[jijiechen]

In addition, in the deregister method, the cleanup of proxyInfos.delete can be skipped when the proxy is connected to another CP instance. This can cause unnecessary memory leak when the CP runs long.

kuma/pkg/xds/server/callbacks/dataplane_lifecycle.go

Lines 202 to 203 in b1c0b13

	d.proxyInfos.Delete(proxyKey)
	info.deleted = true