[Bug]: Internal error occurred: failed calling webhook "accountings.slinky.slurm.net

[katrinm86]

Description

I'm trying to deploy a test setup of slinky 1.0.0 but I keep hitting the following error

helm install -f ./slurm_values.yaml slurm oci://ghcr.io/slinkyproject/charts/slurm --namespace=slurm --create-namespace
Pulled: ghcr.io/slinkyproject/charts/slurm:1.0.0
Digest: sha256:a7d3de4096b5ff32261ae064e3c45e223ea2e706a2cc23d3366aeafab713cb87
Error: INSTALLATION FAILED: Internal error occurred: failed calling webhook "accountings.slinky.slurm.net": failed to call webhook: the server could not find the requested resource
Internal error occurred: failed calling webhook "controllers.slinky.slurm.net": failed to call webhook: the server could not find the requested resource
Internal error occurred: failed calling webhook "restapis.slinky.slurm.net": failed to call webhook: the server could not find the requested resource

Went through the installation guide, the operator values file, the slurm values file but couldn't find anything related.

Also tried disabling the webhook/cert-manager installation but then i'm just hitting "no endpoints available for service "slurm-operator-webhook""

Maybe it is something obvious but I cannot see it at the moment ...

Steps to Reproduce

• (Cluster v1.32.5+rke2r1 comes with cert-manager 0.17 installed)
• Install crds: helm install slurm-operator-crds oci://ghcr.io/slinkyproject/charts/slurm-operator-crds
• Install operator: helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator --namespace=slinky --create-namespace
• Install slurm: helm install -f ./slurm_values.yaml slurm oci://ghcr.io/slinkyproject/charts/slurm --namespace=slurm --create-namespace

Expected Behavior

helm install works

Additional Context

clusterName: test-cluster

slurmKeyRef: null
jwtHs256KeyRef: null

controller:
  external: false

  slurmctld:
    image:
      tag: 25.05-rockylinux9

  reconfigure:
    image:
      tag: 25.05-rockylinux9

  persistence:
    enabled: true
    storageClassName:
      storageClassName: csi-cephfs-sc

  metrics:
    enabled: true
    serviceMonitor:
      enabled: true

restapi:
  slurmrestd:
    image:
      tag: 25.05-rockylinux9

accounting:
  enabled: true

  slurmdbd:
    image:
      tag: 25.05-rockylinux9

loginsets:
  slinky:
    enabled: false

nodesets:
  slinky:
    enabled: false

[klairecodes]

I was unable to reproduce those errors on my end. I recommend trying updated versions of RKE and cert-manager if possible (rke v1.34.X and cert-manager v1.19.2). See this section for version compatibility. I suspect cert-manager is the culprit.

The stripped-down values.yaml provided does not indicate much to me. If you continue to run into issues on updated versions of your services, please provide a complete values.yaml.

[katrinm86]

Thanks for looking into it!

The values are actually the complete values.yaml from the test installation I did. Is there something specific that would be missing (totally possible that I missed something obvious)?

The compatibility section doesn't tell me much since cert-manager is not listed there and the minimum requirement for kubernetes would be 1.29 (if I read it correctly rke2 1.32 directly translates to kubernetes 1.32). Are you referring to something specific? I saw that the slurm version should be newer but I also tried it with 25.11 and got the same result back then ...

I guess you are right about the version though ... However we still have to live with this rke version for now so I can only retest this at a later point :/

[klairecodes]

My apologies, after looking more I found I have misdiagnosed this. It would be very helpful to see logs of your slurm-operator-webhook (for potential cert-manager issues) and your slurm-operator pod, and others. Your YAML should work. A fresh install, double-checking that CRDs are properly installed, from a release like 1.0.1 would be a good retest.