Random timeouts for GraphQL getKubeNamespace/getKubeObject

[stollero]

What happened:
I create a new experiment and want to select the namespace and app label.
But the dropdown does not list a namespace or if the namespace is available, the app labels.

Watching the network I can see that the graphql query ends with 504 gateway timeout after a minute.
Reloading the page helps sometimes.

What you expected to happen:
The drop down for namespace and app label selection always works.

Where can this issue be corrected? (optional)

How to reproduce it (as minimally and precisely as possible):
If I run just the graphql query getKubeObject or getKubeNamespace with a standalone client I can reproduce the issue.
Running the query multiple times it might return nothing (6 out of 10 requests), the correct data (2 out of 10 requests) or runs into the timeout (2 out of 10 requests).

I tried other Queries, e.g. to list the experiments and these work every time.
Only the query that pulls information from the kubernetes cluster runs into timeouts.

Anything else we need to know?:
Cluster currently has 109 namespaces, but kubeobject also fails for namespaces only having a single app label.

Maybe related issues:

• I can't list namespaces when setting up my experiment #4308
• Seperate call to retrieve namespace list and retrieve KubeObject for only a single namespace #4680
• Reaching timeout when getting KubeObject on big cluster #4616

[KelvinVenancio]

We have the same problem here, on one of our clusters with 190 namespaces. Did you figure this out @stollero?

[stollero]

We have the same problem here, on one of our clusters with 190 namespaces. Did you figure this out @stollero?

Unfortunately not yet. We stopped using Litmus and evaluated other tools. But we will try again this week. 🤞

[KelvinVenancio]

Ok, thank you @stollero. Would you mind telling me what tool you are currently using?

[stollero]

Ok, thank you @stollero. Would you mind telling me what tool you are currently using?

We are currently evaluating tools and tried commercial (Gremlin) and OSS (Litmus, Chaos-Mesh). Litmus ist the most promising candidate for us so far.

[KelvinVenancio]

Litmus ist the most promising candidate for us so far.

Agree!

I figured out the problem. For me, the problem was running litmus-server with more than one replica; it didn't make sense to me, and I honestly didn't understand why. However, when I deleted the HPA and reduced it to a single replica, everything started working fine.