Concurrent creation of conflicting resources may pass validating webhook and cause invalid Kong configuration

[randmonkey]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

From FTI-6318.

When I create multiple conflicting resources concurrently, for example two Secrets used as JWT credentials with the same key and attached to two KongConsumers, the webhook could not detect the conflict because it is using a cached client for listing resources, so the conflicting KongConsumer and Secret may not be listed in the client if the interval of creating them too quickly.
Then the resources will be translated to invalid Kong configuration and rejected by Kong gateways:

{"message": "declarative config is invalid: {jwt_secrets={\"uniqueness violation: 'jwt_secrets' entity with key set to 'abc123' already declared\"}}"}

This may also happen for other resources with unique restraints on some fields, like username of consumers or prefix of vaults.

Expected Behavior

No invalid Kong configuration generated in such scenario.
The creation of conflicting resource is rejected, or detect the conflicts in translation phase then remove them in translated Kong configuration and generate TranslationError events.

Steps To Reproduce

Create resources by the following go snippet:
	consumer1 := &kongv1.KongConsumer{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "consumer-1",
			Namespace: "default",
			Annotations: map[string]string{
				"kubernetes.io/ingress.class": "kong",
			},
		},
		Username:    "consumer-1",
		Credentials: []string{"jwt-1"},
	}

	consumer2 := &kongv1.KongConsumer{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "consumer-2",
			Namespace: "default",
			Annotations: map[string]string{
				"kubernetes.io/ingress.class": "kong",
			},
		},
		Username:    "consumer-2",
		Credentials: []string{"jwt-2"},
	}

	secretJwt1 := &corev1.Secret{
		ObjectMeta: metav1.ObjectMeta{
			Name:      "jwt-1",
			Namespace: "default",
			Labels: map[string]string{
				"konghq.com/credential": "jwt",
			},
		},
		StringData: map[string]string{
			"rsa_public_key": `-----BEGIN PUBLIC KEY-----
...`,
			"key":       "abc123",
			"algorithm": "HS256",
			"secret":    "def456",
		},
	}

	secretJwt2 := secretJwt1.DeepCopy()
	secretJwt2.Name = "jwt-2"

	wg := sync.WaitGroup{}
	wg.Add(2)
	go func() {
		defer wg.Done()
		_, err := cl.CoreV1().Secrets("default").Create(ctx, secretJwt1, metav1.CreateOptions{})
		if err != nil {
			fmt.Println("Failed to create secret for jwt credential jwt-1:", err)
			return
		}
		_, err = clKong.ConfigurationV1().KongConsumers("default").Create(ctx, consumer1, metav1.CreateOptions{})
		if err != nil {
			fmt.Println("Failed to create consumer consumer-1:", err)
			return
		}
		fmt.Println("Created consumer consumer-1 and secret jwt-1")
	}()

	go func() {
		defer wg.Done()
		_, err := cl.CoreV1().Secrets("default").Create(ctx, secretJwt2, metav1.CreateOptions{})
		if err != nil {
			fmt.Println("Failed to create secret for jwt credential jwt-2:", err)
			return
		}
		_, err = clKong.ConfigurationV1().KongConsumers("default").Create(ctx, consumer2, metav1.CreateOptions{})
		if err != nil {
			fmt.Println("Failed to create consumer consumer-2:", err)
			return
		}
		fmt.Println("Created consumer consumer-2 and secret jwt-2")
	}()
	wg.Wait()

Kong Ingress Controller version

3.3.1

Kubernetes version

Not related to the issue

Anything else?

No response