Helm upgrade fails when you install spark-operator chart for the first time with webhook enabled

[danielyahn]

I have a use case where I include spark-operator chart as a dependency of my chart, and control its deployment with a Helm variable.

For example, with

# Chart.yaml
apiVersion: v2
name: hello-world
description: A Helm chart for Hello World
dependencies:
  - name: spark-operator
    version: 1.1.6
    repository: https://googlecloudplatform.github.io/spark-on-k8s-operator
    condition: spark-operator.enabled

# values.yaml
spark-operator:
  serviceAccounts:
    spark:
      create: true
      name: spark-service-user
    sparkoperator:
      create: true
      name: spark-operator-service-user
  sparkJobNamespace: spark
  webhook:
    enable: true

It works fine when I helm install my chart with spark-operator.enabled=true.

However, if I helm install the chart with spark-operator.enabled=false then try to helm upgrade with spark-operator.enabled=true, it errors out because of the missing service account, spark-operator-service-user. I think service account are not being created properly during upgrade because the ServiceAccount manifest only has pre-install hook, not pre-upgrade.

For the actual error message, when I helm upgrade, it just times out because the spark-operator-webhook-cleanup job cannot execute. I can see the following error message when I inspect the event log of the job (with kubernetes describe command).

Events:
  Type     Reason        Age                  From            Message
  ----     ------        ----                 ----            -------
  Warning  FailedCreate  67s (x5 over 3m37s)  job-controller  Error creating: pods "hello-world-spark-operator-webhook-cleanup-" is forbidden: error looking up service account spark/spark-operator-service-user: serviceaccount "spark-operator-service-user" not found

[julienlau]

Did you retry since #1359 is fixed in 1.1.27 ?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.