litmus-agent Helm chart - Subscriber and Event-tracker CrashLoopBackOff: ACCESS_KEY / VERSION missing values

[gustavoromerobenitez]

What happened:

1. Deployed litmus Helm chart v3.9.0 on a private GKE cluster in the litmus-operator namespace via ArgoCD.
2. The control plane components are working fine and I can access the Chaos Center UI.
3. Created an Environment.
4. Configured the litmus-agent Helm chart v3.9.0 as follows (credentials are redacted)

litmus-agent:    
  global:
    infraConfigName: test-chaos-infra-config  # This gets created but it is empty
    infraSecretName: test-chaos-infra-secret  # This gets created but it is empty
    INFRA_MODE: "cluster"
  INFRA_NAME: "test-chaos-infrastructure"
  INFRA_DESCRIPTION: "Test chaos infrastructure"
  SA_EXISTS: "false"
  NS_EXISTS: "true"
  INFRA_TYPE: "internal"
  APP_VERSION: "3.9.0"
  SKIP_SSL: "true"
  LITMUS_URL: "http://litmus-operator-app-frontend-service.litmus-operator.svc.cluster.local:9091"
  LITMUS_BACKEND_URL: "http://litmus-operator-app-server-service.litmus-operator.svc.cluster.local:9002"
  LITMUS_USERNAME: "admin"
  LITMUS_PASSWORD: "******"
  LITMUS_PROJECT_ID: "***"
  LITMUS_ENVIRONMENT_ID: "testenvironment"

5. Deployed litmus-agent Helm chart 3.9.0 on the same private GKE cluster in the litmus-agent namespace via Argo CD.
6. The agent installation Job finishes with an error about not being able to process the request header, but I can see the new infrastructure as PENDING in the Chaos Center UI. It never connects.
7. The subscriber workload is in a CrashLoopBackOff state and the logs show a fatal error: "required key ACCESS_KEY missing value"
8. The event-tracker workload is in a CrashLoopBackOff state and the logs show a fatal error: "required key VERSION missing value"
9. The litmus-agent namespace only contains an empty infra secret. There is no 'agent secret' as mentioned in other Issues.

What you expected to happen:
I expected the infra created by the litmus-agent Helm chart to connect successfully to the Chaos Center.

How to reproduce it (as minimally and precisely as possible):
See above in "What happened"

Anything else we need to know?:
I have searched through the issues and have followed the steps mentioned in issue 3531 and issue 3528 amongst others.

[t-singtel]

is this issue resolved?

[rakesh123822]

Im also facing this exact issue with the latest installation.

[celinegrn]

I have the same problem using flux with the latest version 3.11.0.
Has anyone found a solution?

[jemlog]

Can I take this issue?

[namkyu1999]

go ahead, @jemlog

[thiha-min-thant]

After switching to the Helm charts litmus-agent:3.11.2 and litmus:3.11.0, the issues I was encountering have been resolved. Everything now appears to be functioning smoothly on my end.

[seanocca]

I am seeing the same issue on litmus-agent:3.13.0 and litmus:3.13.0.
We install via ArgoCD and let the chart create all it's own secrets and configmaps.
We also set the INFRA_TYPE to external use an ingress via LITMUS_URL, LITMUS_FRONTEND_URL and LITMUS_BACKEND_URL

[phonkd]

Does this issue occur only when using skip_ssl: true ?
Edit: skip_ssl false doesnt fix the issue

[phonkd]

I looked into this and found the following:
In version 3.16.0 the dockerfille seemingly requires the environment Variable access_key (subscriber.go)

type Config struct {
	AccessKey        string `required:"true" split_words:"true"`
...
}

In the helm chart secret template it doesnt exist however (secret.yaml)..

[phonkd]

Only workaround is setting
useExistingHookSecret: false
useExistingInfraConfigMap: true
useExistingInfraSecret: true
and creating the secrets manually

The automatically created secret and configmap are empty by default.