Add support for entire kubeflow pipelines as trial target (in addition to containers)

[romeokienzler]

/kind feature

Describe the solution you'd like
Hyper parameters not only affect the training step but also upstream pipeline components like feature transformation for example (e.g. parameters of a normalization transformation). In addition, transformation and training steps should be able to make use of kfp's parallel components (e.g. SparkJob, TFJob, ...). It would be helpful to not only allow to specify containers as trial targets but also complete kubeflow pipelines. As the latter also expose parameters they can be either set directly (non-hyperparameters) or added to the hyper parameter space.

Anything else you would like to add:
I've started to create simple container image which can be used as trial target which acts as a proxy and downstream triggers parameterized Kubeflow pipeline executions with the respective hyper parameters. A Kubernetes Custom Resource can be created as well down the line.

---

Love this feature? Give it a 👍 We prioritize the features with the most 👍

[romeokienzler]

Just digged in a little more - actually IMHO we need a new TrialTemplate - as Argo and Tekton are already supported I'll go for one of those initially...

[skliarpawlo]

Hi @romeokienzler , I'm also very interested in this feature and it's surprising actually that it wasn't provided already as it sounds natural. I have a question though, I see there is an example of integration argo workflows with katib here kubeflow pipelines are implemented via argo workflows aren't they? If so why can we do this already? I gave it a try and I'm constantly getting some new errors which makes me think they are not compatible.

[votti]

I think I have kubeflow pipelines with Katib - almost - working:
Unfortunately I have not a minimal example yet I set out to translate an existing ML example to run as pipeline with Katib - just assuming that this works - before discovering that this was not straight forward.

I followed the Argo/Katib installation instruction (https://github.com/kubeflow/katib/tree/master/examples/v1beta1/argo). Then I for now manually generated a manifest based on the example provided there, using a Kubeflow pipeline.

Adaptions

• replacing the trialSpec from a pipeline into: https://github.com/kubeflow/katib/blob/master/examples/v1beta1/argo/argo-workflow.yaml
• setting up the trialParameters
• manually adding the primaryPodLabel katib.kubeflow.org/model-training: "true" to the main training step

My example file is attached. To reproduce in your own env, also adapt the serviceAccountName and namespace.

I found that currently the reason this does not work is, that when the pod with the metrics collector sidecar is setup, the run command is re-written to be a single command.

In the Argo example, Katib re-writes:

            command:
            - python3
            - /opt/mxnet-mnist/mnist.py
            - --lr=${trialParameters.learningRate}
            - --num-examples={{inputs.parameters.num-examples}}

into

      /var/run/argo/argoexec emissary -- python3 /opt/mxnet-mnist/mnist.py --lr=0.025074661653484223 --num-examples=3157 1>/var/log/katib/metrics.log 2>&1 && echo completed > /var/log/katib/$$$$.pid

Now as for pipelines this commands are much more complex, I believe this re-writing breaks them.
For my example this leads to a syntax error:

sh: 2: cannot create : Directory nonexistent
/opt/conda/bin/python3: can't find '__main__' module in '/home/jovyan/'
sh: 4: Syntax error: "(" unexpected

Removing the katib.kubeflow.org/model-training: "true" label from the training step makes this step run correctly, but Katib failing (as no metrics can be collected.

Anyways, I think if this re-writing of the command for metrics collection would be fixed, combining pipelines with Katib should work.

Run with:

• Argo v3.3.8
• Katib v0.14.0

Example pipeline: katib-pipeline.txt
Based on:
original_pipeline.txt

Sorry for making this hasty/sloppy report. I am working on another deadline but thought reporting this even in this state may be interesting.

[votti]

Hey all,

Now managed to get the Workflows running:
I worked-around my issue with complex commands being broken by the adaptation for the metrics collector sidecar: writing the metrics out into a file and then have a separate, super simple pipeline step to just print the file did the trick

Two issues I run into:

• the container to print was starting faster than the metrics collector sidecar, causing errors. Dirty workaround: using sleep before printing with cat
• Caching: the pipeline step printing the metrics needs to have caching turned of to be guaranteed to run every time

A working example can be found here:
working_workflow.txt

What I found nice is that caching works as one would hope to expect: so only the steps affected by the parameter tuning and downstream will be re-run. Eg initial data loading/parsing/... will be only run once.
To me for prototyping this could come handy.

For me the next steps will be:

• compile my example in a python script (currently I directly modified the yaml to play around)
• generate a minimal example that generates and runs a Katib parameter tuning on a pipeline
• generate an issue that sidecar injection should not break complex commands -> also with minimal example

[votti]

Hi there,
Just an update: I have now a working example that can be run from a notebook that:

1. Defines a Kubeflow Pipeline
2. Runs Katib over the whole pipeline, tuning the pipeline parameters.

I am currently re-writing this to a simple MNIST example.
Would it make sense to prepare directly a branch with such a documented Example in the Example folder, or would you prefer me to prepare the example in a separate repository?

Something I was wondering is, if it wouldnt make sense, that to have a metrics collector that works with Kubeflow pipeline metrics artifacts.
This would make it even more seamless to put a Katib job over an existing pipeline (will open a new issue).

[votti]

I started here a separate repo where I am developing the mnist example:
https://github.com/votti/katib-exploration/blob/main/notebooks/mnist_pipeline_v1.ipynb

The general approach seems to work and the pipeline produces a yaml that can be submitted to Katib.

Unfortunately I need to adapt things for v2 kubeflow pipeline syntax, thus this is with v1.
Note that this requires the Argo integration setup as described: https://github.com/kubeflow/katib/tree/master/examples/v1beta1/argo/README.md

[johnugeorge]

@votti This is great start. Thank you for taking this. We can add this MNIST example to https://github.com/kubeflow/katib/tree/master/examples/v1beta1/kubeflow-pipelines

Few items to be completed before merge

1. Have a clean solution for A metrics collector for Kubeflow Pipeline Metrics artifacts #2019
2. v2 kfp support
3. Submit using sdk instead of yaml

[votti]

Cool, I am happy to push a bit more on this.
@1 As discussed #2019 I think something analogous to the TFevent metrics collector could work
@2 I need to investigate how the parameters would be passed in v2
@3 This definitely also sounds feasible

[andreyvelich]

Thank you for this great examples @votti!
We recently made changes to Katib SDK: #2075, so it should be easier to update your examples.
Do you have some bandwidth to contribute your example before 0.15 Katib release (we will cut the first RC on Jan 25th) ?
If not, we can post-pone this to the next release.

[votti]

Hi @andreyvelich I will try to find time, but currently am a bit stopped as I lost access to my previous Kubeflow environment and am now moving to a local installation. Lets see.