bug: failed to reflect pluginConfig delete to cache: still in use

[dickens7]

Issue description

An exception occurs when ApisixPluginConfig is used by multiple ApisixRoute or Ingress

https://github.com/apache/apisix-ingress-controller/blob/v1.5.0/pkg/ingress/apisix_route.go#L329-L336

https://github.com/apache/apisix-ingress-controller/blob/v1.5.0/pkg/kube/translation/apisix_route.go#L924-L927

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long):
  Version: 1.5.0
  Git SHA: no-git-module
  Go Version: go1.19.2
  Building OS/Arch: linux/amd64
  Running OS/Arch: linux/amd64
• your Kubernetes cluster version (output of kubectl version):
  v1.18.4+k3s1
• if you run apisix-ingress-controller in Bare-metal environment, also show your OS version (uname -a):

Minimal test code / Steps to reproduce

1. create ApisixPluginConfig

apiVersion: apisix.apache.org/v2
kind: ApisixPluginConfig
metadata:
  name: cors-plugins
spec:
  plugins:
    - name: cors
      enable: true

2. create ApisixRoute httpserver and httpserver2

apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: httpserver
spec:
  http:
  - name: httpserver
    match:
      hosts:
        - local.httpbin.org
      paths:
        - /
    backends:
     - serviceName: httpbin
       servicePort: 80
    plugin_config_name: cors-plugins

---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: httpserver2
spec:
  http:
  - name: httpserver
    match:
      hosts:
        - local.httpbin.org
      paths:
        - /
    backends:
     - serviceName: httpbin
       servicePort: 80
    plugin_config_name: cors-plugins

Actual result

ApisixPluginConfig will be deleted when ApisixRoute or Ingress is updated and deleted

Error log

2022-10-27T18:20:23+08:00       info    utils/manifest.go:361   plugin_config was referenced by other routes    {"plugin_config_id": "9c092931", "plugin_config_name": ""}
2022-10-27T18:20:23+08:00       debug   ingress/apisix_route.go:507     ApisixRoute update event arrived        {"key": "xxx-env-dev/oa2-orgfront", "new object": {"metadata":{"name":"oa2-orgfront","namespace":"xxx-env-dev","selfLink":"/apis/apisix.apache.org/v2/namespaces/xxx-env-dev/apisixroutes/oa2-orgfront","uid":"2f2036ed-f973-4961-9534-3ce24627cec5","resourceVersion":"181495831","generation":4,"creationTimestamp":"2022-06-24T03:06:34Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"oa2-orgfront","meta.helm.sh/release-namespace":"xxx-env-dev"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}}},{"manager":"aslan","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{}}},{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2022-10-27T10:20:22Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:http":{}},"f:status":{"f:conditions":{}}}}]},"spec":{"http":[{"name":"oa2-orgfront","match":{"paths":["/","/*"],"hosts":["org.k3s-jcyfb-dev.xxx.cn"]},"backends":[{"serviceName":"oa2-orgfront","servicePort":80,"weight":null}],"websocket":false,"plugin_config_name":"oa2-front-plugins","authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":4,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}, "old object": {"metadata":{"name":"oa2-orgfront","namespace":"xxx-env-dev","selfLink":"/apis/apisix.apache.org/v2/namespaces/xxx-env-dev/apisixroutes/oa2-orgfront","uid":"2f2036ed-f973-4961-9534-3ce24627cec5","resourceVersion":"181495839","generation":4,"creationTimestamp":"2022-06-24T03:06:34Z","labels":{"app.kubernetes.io/managed-by":"Helm"},"annotations":{"meta.helm.sh/release-name":"oa2-orgfront","meta.helm.sh/release-namespace":"xxx-env-dev"},"managedFields":[{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{}}},{"manager":"aslan","operation":"Update","apiVersion":"apisix.apache.org/v2beta3","time":"2022-10-15T10:05:53Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app.kubernetes.io/managed-by":{}}},"f:spec":{}}},{"manager":"apisix-ingress-controller","operation":"Update","apiVersion":"apisix.apache.org/v2","time":"2022-10-27T10:20:23Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:http":{}},"f:status":{"f:conditions":{}}}}]},"spec":{"http":[{"name":"oa2-orgfront","match":{"paths":["/","/*"],"hosts":["org.k3s-jcyfb-dev.xxx.cn"]},"backends":[{"serviceName":"oa2-orgfront","servicePort":80,"weight":null}],"websocket":false,"plugin_config_name":"oa2-front-plugins","authentication":{"enable":false,"type":"","keyAuth":{},"jwtAuth":{}}}]},"status":{"conditions":[{"type":"ResourcesAvailable","status":"True","observedGeneration":4,"lastTransitionTime":null,"reason":"ResourcesSynced","message":"Sync Successfully"}]}}}
2022-10-27T18:20:23+08:00       debug   apisix/pluginconfig.go:47       try to look up pluginConfig     {"name": "xxx-env-dev_oa2-plugins", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugin_configs", "cluster": "default"}
2022-10-27T18:20:23+08:00       debug   ingress/apisix_route.go:304     translated ApisixRoute  {"routes": [{"id":"1eedc198","name":"xxx-env-dev_oa2-org_oa2-org","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"hosts":["oa2.k3s-jcyfb-dev.xxx.cn"],"uris":["/org/api/","/org/api/*","/workflow/api/","/workflow/api/*"],"upstream_id":"f9f67824","plugin_config_id":"a7a986af"}], "upstreams": [{"id":"f9f67824","name":"xxx-env-dev_oa2-org_8888","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","labels":{"managed-by":"apisix-ingress-controller"},"type":"roundrobin","nodes":[{"host":"10.42.1.156","port":8888,"weight":100}],"scheme":"http"}], "apisix_route": {}, "pluginConfigs": null}
2022-10-27T18:20:23+08:00       debug   apisix/route.go:47      try to look up route    {"name": "xxx-env-dev_oa2-org_oa2-org", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes", "cluster": "default"}
2022-10-27T18:20:23+08:00       debug   apisix/upstream.go:203  try to update upstream  {"id": "f9f67824", "name": "xxx-env-dev_oa2-org_8888", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams"}
2022-10-27T18:20:23+08:00       debug   apisix/upstreamservicerelation.go:95    try to create upstreamService in cache  {"cluster": "default"}
2022-10-27T18:20:23+08:00       debug   apisix/upstream.go:223  updating upstream       {"body": "{\"id\":\"f9f67824\",\"name\":\"xxx-env-dev_oa2-org_8888\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"type\":\"roundrobin\",\"nodes\":[{\"host\":\"10.42.1.156\",\"port\":8888,\"weight\":100}],\"scheme\":\"http\"}", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/upstreams/f9f67824"}
2022-10-27T18:20:23+08:00       debug   apisix/resource.go:123  got upstream: {"pass_host":"pass","nodes":[{"port":8888,"host":"10.42.1.156","priority":0,"weight":100}],"hash_on":"vars","desc":"Created by apisix-ingress-controller, DO NOT modify it manually","name":"xxx-env-dev_oa2-org_8888","update_time":1666866023,"labels":{"managed-by":"apisix-ingress-controller"},"id":"f9f67824","create_time":1656040035,"scheme":"http","type":"roundrobin"}
2022-10-27T18:20:23+08:00       debug   apisix/route.go:205     try to update route     {"id": "1eedc198", "name": "xxx-env-dev_oa2-org_oa2-org", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes"}
2022-10-27T18:20:23+08:00       debug   apisix/route.go:219     updating route  {"body": "{\"id\":\"1eedc198\",\"name\":\"xxx-env-dev_oa2-org_oa2-org\",\"desc\":\"Created by apisix-ingress-controller, DO NOT modify it manually\",\"labels\":{\"managed-by\":\"apisix-ingress-controller\"},\"hosts\":[\"oa2.k3s-jcyfb-dev.xxx.cn\"],\"uris\":[\"/org/api/\",\"/org/api/*\",\"/workflow/api/\",\"/workflow/api/*\"],\"upstream_id\":\"f9f67824\",\"plugin_config_id\":\"a7a986af\"}", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/routes/1eedc198"}
2022-10-27T18:20:23+08:00       debug   apisix/resource.go:93   got route: {"hosts":["oa2.k3s-jcyfb-dev.xxx.cn"],"desc":"Created by apisix-ingress-controller, DO NOT modify it manually","upstream_id":"f9f67824","plugin_config_id":"a7a986af","update_time":1666866023,"create_time":1658209424,"priority":0,"labels":{"managed-by":"apisix-ingress-controller"},"id":"1eedc198","status":1,"uris":["\/org\/api\/","\/org\/api\/*","\/workflow\/api\/","\/workflow\/api\/*"],"name":"xxx-env-dev_oa2-org_oa2-org"}
2022-10-27T18:20:23+08:00       debug   apisix/pluginconfig.go:180      try to delete pluginConfig      {"id": "a7a986af", "name": "", "cluster": "default", "url": "http://apisix-internal.ingress-apisix.svc.cluster.local:9180/apisix/admin/plugin_configs"}
2022-10-27T18:20:23+08:00       error   apisix/pluginconfig.go:191      failed to reflect pluginConfig delete to cache: still in use```


### Expected result

ApisixPluginConfig is not deleted when ApisixRoute or Ingress are updated and deleted

[tao12345666333]

If pluginconfig is still used, we should not delete it

[tao12345666333]

I see you describe the actual result. Can you describe the expected result?

[dickens7]

apisix-ingress-controller/pkg/ingress/apisix_route.go

Lines 329 to 339 in 21f39e9

	oldCtx, _ := c.controller.translator.TranslateOldRoute(obj.OldObject)
	om := &utils.Manifest{
	Routes: oldCtx.Routes,
	Upstreams: oldCtx.Upstreams,
	StreamRoutes: oldCtx.StreamRoutes,
	PluginConfigs: oldCtx.PluginConfigs,
	}
	added, updated, deleted = m.Diff(om)
	}
	return c.controller.syncManifests(ctx, added, updated, deleted)

• oldCtx
  

  apisix-ingress-controller/pkg/kube/translation/apisix_route.go

  Lines 924 to 928 in 21f39e9

  	if r.PluginConfigId != "" {
  	pc := apisixv1.NewDefaultPluginConfig()
  	pc.ID = r.PluginConfigId
  	oldCtx.AddPluginConfig(pc)
  	}

After the above code m.Diff(om) is executed, deleted.PluginConfigs is not empty;

Expected result

ApisixPluginConfig is not deleted when ApisixRoute or Ingress are updated and deleted

[tao12345666333]

I saw your PR #1413 want to fix v1.5 branch.

But we should follow "upstream first", we should fix it in master branch.

apisix-ingress-controller/pkg/providers/apisix/translation/apisix_route.go

Lines 984 to 988 in 53f26c1

	if r.PluginConfigId != "" {
	pc := apisixv1.NewDefaultPluginConfig()
	pc.ID = r.PluginConfigId
	oldCtx.AddPluginConfig(pc)
	}

Could you open another PR to fix it (master branch)? Thanks