Experiment stuck due to hitting `Suggestion` custom resource size limits

[nielsmeima]

/kind bug

What steps did you take and what happened:
Submitting a large (i.e. resulting in a large number of trials, in this case ~14500 with 4 hyperparameters with 10/11 values per hyperparameter) experiment results in the Suggestion custom resource reaching the size limits of custom resources dictated by Kubernetes due to all suggestions being stored in this resource. This results in the following error being output by the Katib controller when trying to update the Suggestion custom resource: Request entity too large and the experiment not being able to progress. This issue seems to describe the exact problem.

Argo Workflows seems to have encountered the same problem, described here and solved it by allowing for 1) compression of the data stored in the status field of the custom resource and 2) storage of information under the status field in a relational database as described here.

What did you expect to happen:
I expected Katib to be able to handle search spaces or arbitrary size.

Anything else you would like to add:
A workaround would be to manually split the experiment into smaller subexperiments to circumvent the size limits of custom resources. Ideally, this is solved by following a similar approach as Argo does for their Workflow custom resources.

---

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

[johnugeorge]

Thanks for creating this issue. Can you provide more info about the experiment yaml and other relevant information for reproducibility?

MaxTrials - 14500
Parameter1, Parameter2, Parameter3, Parameter4 - 11 values each
ParallelTrials - ?

[nielsmeima]

Yes, please see below for the experiment yaml as well as the other dependencies to be able to reproduce the experiment. I create a configmap k create configmap script --from-file=run.py=run.py based on a run.py file to mock my actual implementation (but this also results in the same issues). The configmap then gets mounted in the main container of the experiment taking in the parameters and producing a value for the objective of interest. I have also attached the run.py file below.

Experiment

apiVersion: kubeflow.org/v1beta1
kind: Experiment
metadata:
  name: debug
spec:
  objective:
    type: maximize
    goal: 500
    objectiveMetricName: cost
  algorithm:
    algorithmName: grid
  parallelTrialCount: 20
  maxTrialCount: 14641
  maxFailedTrialCount: 2000
  parameters:
    - name: a
      parameterType: double
      feasibleSpace:
        min: "1.30"
        max: "1.41"
        step: "0.01"
    - name: b
      parameterType: categorical
      feasibleSpace:
        list: ["0.0010", "0.0025", "0.0063", "0.0158", "0.0398", "0.1000", "0.2512", "0.6310", "1.5849", "3.9811", "10.0000"]
    - name: c
      parameterType: double
      feasibleSpace:
        min: "1.30"
        max: "1.41"
        step: "0.01"
    - name: d
      parameterType: categorical
      feasibleSpace:
        list: ["0.0010", "0.0025", "0.0063", "0.0158", "0.0398", "0.1000", "0.2512", "0.6310", "1.5849", "3.9811", "10.0000"]
  trialTemplate:
    retain: false
    primaryContainerName: training-container
    trialParameters:
      - name: a
        reference: a
        description: ""
      - name: b
        reference: b
        description: ""
      - name: c 
        reference: c
        description: ""
      - name: d 
        reference: d
        description: ""
    trialSpec:
      apiVersion: batch/v1
      kind: Job
      spec:
        template:
          spec:
            containers:
              - name: training-container
                image: docker.io/python:alpine3.15
                volumeMounts:
                - name: script
                  mountPath: /app/run.py
                  subPath: run.py
                command:
                  - "python3"
                  - "/app/run.py"
                  - "${trialParameters.a}"
                  - "${trialParameters.b}"
                  - "${trialParameters.c}"
                  - "${trialParameters.d}"
            restartPolicy: Never
            volumes:
              - name: script
                configMap:
                  name: script

The mock implementation run.py

import sys
import time
time.sleep(4)
cost = sum([float(x) for x in sys.argv[1:]])
print(f"cost={cost}")

EDIT: and to provide you an idea of the error messages arising from the katib-controller (this is a slightly different experiment, but the errors are identical)

{"level":"info","ts":1649925772.8911624,"logger":"suggestion-controller","msg":"Sync assignments","Suggestion":"simulation/simulation-nr-fb","Suggestion Requests":8165,"Suggestion Count":8143}
{"level":"info","ts":1649925774.5568578,"logger":"suggestion-client","msg":"Getting suggestions","Suggestion":"simulation/simulation-nr-fb","endpoint":"simulation-nr-fb-grid.simulation:6789","Number of current request parameters":22,"Number of response parameters":22}
{"level":"info","ts":1649925775.6414711,"logger":"suggestion-controller","msg":"Update suggestion instance status failed, reconciler requeued","Suggestion":"simulation/simulation-nr-fb","err":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (2100613 vs. 2097152)"}

[robertzsun-dev]

I have the same problem running Sobol suggestions. You basically just need enough trials and it will collapse on you. First time its with the etcd request size being too large, which you can "fix" by increasing the etcd max request size, but then you run into the issue in this thread, which is that we are hitting CRD limits. I don't think you can try to work around this.

{"level":"info","ts":1684216616.817069,"logger":"suggestion-controller","msg":"Update suggestion instance status failed, reconciler requeued","Suggestion":"argo/katib-hyperparam-opt-feature-update","err":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (2103483 vs. 2097152)"} 

[andreyvelich]

Hi @robertzsun-dev, can it be limitation of Goptuna algorithm that we use for Sobol ?
@c-bata Do we have any limitation of maximum number of Trials in Goptuna ?

Also, related issue: #1058.

[robertzsun-dev]

@andreyvelich
No, I don't think it's a limitation of Goptuna. I am seeing the suggestion controller offer valid new suggestions in the suggestion loop.

I think issue is due to Katib architecture :

katib/pkg/controller.v1beta1/suggestion/suggestionclient/suggestionclient.go

Line 191 in b9dc63e

instance.Status.Suggestions = append(instance.Status.Suggestions, trialAssignments...)

We can see that new suggestions are simply appended to the suggestionsv1beta1.Suggestion resource's Status field (under Status.Suggestions).

I don't think its K8S-kosher to just add data to this Status field in a K8S resource. This YAML list gets extremely long when you have 1000s of Suggestions, and especially when you have lots of Parameter Assignments, this list actually gets to megabytes of size, eventually exceeding the limit that K8S has on resource size in Etcd. And that is error I posted above.

[robertzsun-dev]

With the related issue of #1058.

I don't think its directly related, but indirectly related in the fact that Katib is not really built for running mass-scale experiments.
I do think for ML, the number of hyperparameters may not be high, and you will not need to run 1000's to 10000's of experiments. But for other things (like what I am doing), we do want to run tons of experiments.

Both the fact that Suggestions/Trials are stored as an Etcd resource, as well as the other issue linked, which prevents long-running suggestions (necessary because 1000s of trials takes a while to compute the next suggestion) from working, prevent users from achieving scale with Katib.

[nielsmeima]

I think @robertzsun-dev is correct in his assessment. At work we moved to a custom system (non-Katib) for performing large scale (> 20k) experiments which does not rely on the Kubernetes custom resource model.

[andreyvelich]

Thanks for the information @robertzsun-dev. Yes, you are right the etcd default size is 1.5 MiB, which makes it impossible to store large chunk of data in the Custom Resource.

I understand that ordinary HP Tuning Experiments might not require 10000 Trials, but for some cases it might be useful. Since, Katib allows you to use optimisation algorithms for any type of the task (as long as Trial is set), we can find work-around for it.

@robertzsun-dev @nielsmeima Please can you describe your use-case, when you need to run Experiments for more than 10000 Trials.

As a solution, we can store such information in Katib DB instead of Suggestion CR or Experiment CR.

cc @johnugeorge @tenzen-y @gaocegege

[tenzen-y]

I would propose we set the limit for the number of AlgorithSettings and Suggestions.
And then, if exceeded the limitation, the katib-controller creates configMaps to write the results of experiments and clean up AlgorithSettings and Suggestions.

katib/pkg/apis/controller/suggestions/v1beta1/suggestion_types.go

Lines 46 to 49 in b9dc63e

	// AlgorithmSettings defines HP or NAS algorithm settings which suggestion gRPC service returns.
	// These settings overwrites Experiment's settings before the gRPC request.
	// It can be empty if settings haven't been changed.
	AlgorithmSettings []common.AlgorithmSetting `json:"algorithmSettings,omitempty"`

katib/pkg/apis/controller/suggestions/v1beta1/suggestion_types.go

Lines 54 to 55 in b9dc63e

	// Suggestion results
	Suggestions []TrialAssignment `json:"suggestions,omitempty"`

wdyt?

[andreyvelich]

And then, if exceeded the limitation, the katib-controller creates configMaps to write the results of experiments and clean up AlgorithSettings and Suggestions.

I think, storing such information in the ConfigMap might also be a problem, since the limit is 1 Mb: https://kubernetes.io/docs/concepts/configuration/configmap/#motivation

That is why, I suggested to store that info in the Katib DB if that is possible.

[tenzen-y]

I think, storing such information in the ConfigMap might also be a problem, since the limit is 1 Mb: https://kubernetes.io/docs/concepts/configuration/configmap/#motivation

Yes, that's right.
I was thinking of creating multiple configMaps.

[andreyvelich]

I see. @tenzen-y any objections you see to store that info in the MySQL/Postgres DB ?

[tenzen-y]

I see. @tenzen-y any objections you see to store that info in the MySQL/Postgres DB ?

As per the opinions of the on-prem cluster administrator side.

Currently, even if the katib-db crashes, it is easy to check the result of experiments since CRs have the results for the experiments in etcd.

But, Storing the result of experiments to the katib-db increases the importance of katib-db. I wouldn't increase the number of storage with high importance.

[robertzsun-dev]

Thanks for the information @robertzsun-dev. Yes, you are right the etcd default size is 1.5 MiB, which makes it impossible to store large chunk of data in the Custom Resource.

I understand that ordinary HP Tuning Experiments might not require 10000 Trials, but for some cases it might be useful. Since, Katib allows you to use optimisation algorithms for any type of the task (as long as Trial is set), we can find work-around for it.

@robertzsun-dev @nielsmeima Please can you describe your use-case, when you need to run Experiments for more than 10000 Trials.

As a solution, we can store such information in Katib DB instead of Suggestion CR or Experiment CR.

cc @johnugeorge @tenzen-y @gaocegege

My use case is for heuristic-driven learning or other types of probabilistic models. Training or learning behaviors from scratch for complex behaviors (for me in robotics) is not really feasible or takes a long time. We introduce a set of heuristics to inform the algorithms. We can do this in multi-step processes, with heuristics only approaches and then training with heuristics, and so on. Generally, evaluating the heuristics takes a very short amount of time, maybe 5 - 20 minutes, and we want to hyperparameter search over this heuristic space.

One such case is say we will do something if the distance to an object is < X meters. What should X be? There might be many of such heuristics, each with their own < Y, > Z, etc... Together, the combination of heuristics and hyper-parameters chosen can greatly affect the quality of the algorithm. We can even search over how to weigh these heuristics against each other.

So we use katib and mass-scale hyperparameter search to just close-in on a good set of hyperparameters - we can even do a sort of distributed coordinate descent by first searching the X, Y, Z's above, then searching the weights, then going back and doing X, Y, Z, etc.... Once we arrive at a good set of heuristics and weights, we can do some more learning on top. The possibilities are endless.

This is just one of many use cases, but highlights the value of mass-scale hyperparameter search.