[🐛 Bug]: Helm post-upgrade hook fails when recreating selenium-grid-edge-scaledobject

[oranlom]

What happened?

I'm deploying selenium-grid helm chart on AKS cluster with an existing keda (helm chart version 2.12.1) installed using Azure pipelines. The helm upgrade job fails with the following error:

Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "selenium-grid-edge-scaledobject" already exists

Although after rerunning the failed job, the helm upgrade is successful. The selenium-grid-chrome-scaledobject gets deployed without any issues at the first time, with the selenium-grid-edge-scaledobject there's always this rerun needed. Is there any way how to control the post-upgrade hook deployments, to set some kind of delay on this particular hook? Maybe setting the hook-weight and changing the order of the chrome (by default is installed first) and edge scaled object deployment could solve the issue. With the current selenium-grid helm chart I'm able to set only identical annotations for both scaled objects.

Command used to start Selenium Grid with Docker (or Kubernetes)

helm upgrade --install --values deployments/selenium-grid/envs/DEV/deployment.yaml --wait --timeout 5m0s --create-namespace -n platform-test selenium-grid repo_selenium-grid/selenium-grid --version 0.25.3

deployments/selenium-grid/envs/DEV/deployment.yaml

basicAuth:
  enabled: false
ingress:
  enabled: false
autoscaling:
  enableWithExistingKEDA: true
  scalingType: deployment
  scaledOptions:
    minReplicaCount: 0
    maxReplicaCount: 8
    advanced:
      horizontalPodAutoscalerConfig:
        behavior:
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
              - type: Percent
                value: 100
                periodSeconds: 15

hub:
  serviceType: ClusterIP
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret

# Configuration for chrome nodes
chromeNode:
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-chrome-node

# Configuration for firefox nodes
firefoxNode:
  # Enable firefox nodes
  enabled: false

# Configuration for edge nodes
edgeNode:
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-edge-node

Relevant log output

Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: object is being deleted: scaledobjects.keda.sh "selenium-grid-edge-scaledobject" already exists

Operating System

AKS 1.27.9

Docker Selenium version (image tag)

4.15.0-20231129

Selenium Grid chart version (chart version)

0.25.3

[github-actions]

@oranlom, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[VietND96]

Looks like the issue is in backward chart versions. Can you try 2 possible ways?

1. In your deployments/selenium-grid/envs/DEV/deployment.yaml, add these configs for KEDA sub-chart

keda:
  additionalAnnotations:
    "helm.sh/hook": pre-install

2. Upgrade to use recent chart versions (as of now 0.29.0+)

[VietND96]

Ah ok, I see you are using enableWithExistingKEDA: true. Probably above suggestions might not work. Let me try to reproduce this one

[oranlom]

Yes, those keda.additionalAnnotations are already there by default. Additionally, our selenium helm deploy starts in the different step of the infrastructure pipeline, when keda is already deployed. I'm currently trying to deploy the 0.29.0 version, but there's a lot of changes introduced so I need to tweak my helm-values first to avoid issues in other pipelines with some dependencies on selenium.

[oranlom]

So I upgraded the chart to version 0.29.0 but I got the same error.

Updated helm-values.yaml file:

basicAuth:
  enabled: false
ingress:
  enabled: false
serviceAccount:
  create: true
  nameOverride: selenium-serviceaccount
autoscaling:
  enableWithExistingKEDA: true
  scalingType: deployment
  scaledOptions:
    minReplicaCount: 0
    maxReplicaCount: 8
    advanced:
      horizontalPodAutoscalerConfig:
        behavior: # Optional. Use to modify HPA's scaling behavior
          scaleDown:
            stabilizationWindowSeconds: 300
            policies:
              - type: Percent
                value: 100
                periodSeconds: 15
secrets:
  create: false
  nameOverride: selenium-kv-secret
components:
  subPath: /

hub:
  nameOverride: selenium-hub
  serviceType: ClusterIP
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret

# Configuration for chrome nodes
chromeNode:
  nameOverride: selenium-chrome-node
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-chrome-node

# Configuration for firefox nodes
firefoxNode:
  # Enable firefox nodes
  enabled: false

# Configuration for edge nodes
edgeNode:
  nameOverride: selenium-edge-node
  nodeSelector: { agentpool: "seleniumpool" }
  tolerations:
    - effect: NoSchedule
      key: role
      operator: Equal
      value: seleniumNode
  extraVolumeMounts:
    - name: qa-file-pv
      mountPath: /mnt/testautomation-files
    - name: selenium-kv-secrets-store-inline
      mountPath: "/mnt/secrets-store"
      readOnly: true
  extraVolumes:
    - name: qa-file-pv
      persistentVolumeClaim:
        claimName: qa-file-pvc
    - name: selenium-kv-secrets-store-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: selenium-kv-secrets-provider
  extraEnvFrom:
    - secretRef:
        name: selenium-kv-secret
  scaledObjectOptions:
    scaleTargetRef:
      name: selenium-edge-node

Error message: Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade selenium-grid/templates/edge-node-hpa.yaml failed: scaledobjects.keda.sh "selenium-edge-node" already exists

Any ideas, or suggestions are more than welcome

[VietND96]

Looks like I also could reproduce the issue, but with the object scaledjobs. With using an existing KEDA and upgrade scenario, so I guess all KEDA crds already there, we can update the hook for post-install only. Can you try this config?

autoscaling:
  annotations:
    "helm.sh/hook": post-install

[oranlom]

Correct me if I'm wrong but this would actually mean that the helm-upgrade --install command in our pipeline will not trigger the redeployment of scaledobjects even if their configuration in values.yaml is changed. This is not really a desired solution :/

[VietND96]

As the usage of the chart hooks, without post-upgrade specification, it doesn't mean the redeployment does not happen even their configs changed. In a short description, the hooks allow us to configure an object that will be deployed (before or after) a release's life cycle (install, upgrade, or rollback).
For example: during the upgrade, all objects (configmap, svc, secrets, deployment, etc.) are deployed in parallel, once upgrade completed, the object scaledobject is deployed

[oranlom]

I understand the concept of chart hooks, but when the hook is set to post-install only, the deployment of object scaledobject will happen only at first time installation or clean install, not in the case of an upgrade, right? At least that's what my observation and testing showed.
Also, I don't seem to understand why the helm validation (helm-upgrade --install dry-run) outputs the manifests of an updated scaledobjects with the

  annotations:
    "helm.sh/hook": post-install

but in fact the scaledobject stay untouched with the original
annotations: helm.sh/hook: post-install, post-upgrade

So again, it seems like the redeployment of scaledobjects when upgrading will not happen when having post-install hook set. With post-upgrade hook on scaledobjects I'm getting the error shown in the description of this bug.

[VietND96]

How about if you validate by changing a configuration and rendering YAML output using the helm template? The new config is present in the output YAML?

[oranlom]

I used the following command helm template selenium-grid repo_selenium-grid/selenium-grid --create-namespace --namespace platform-test --version=0.29.0 -f deployments/selenium-grid/envs/DEV/deployment.yaml
The new config is indeed present in the output YAML.

Unfortunately after that with the following command:
helm upgrade --install selenium-grid repo_selenium-grid/selenium-grid --create-namespace --namespace platform-test --version 0.29.0 --values deployments/selenium-grid/envs/DEV/deployment.yaml
the change was not deployed.