Support a container flow inside one pod

[bassel-z]

I have noticed that each ContainerOp creates a pod not a container. It would be great if we can define a flow based on a series of steps that are launched as containers within the same pod. Would that be possible?

The idea behind it is that containers share the same disk storage volume. So for big data use cases and if multiple steps in the flow require the same data, it would make sense to load the data once to the node rather than loading the data for each step.

Is there a way to overcome the multiple loading issue currently across different steps (pods)?

[elikatsis]

Hello @bassel-z,
As far as I know, such pipelining inside a pod is not supported by Kubernetes. There are initContainers, which run before the main container but are not supported by the DSL, yet. Also, there are sidecars which are supported by the DSL, but they run in parallel with the main container.

However, what you could use is persistent storage which is local to your cluster. You can mount PVCs under some path, download data, write the results there, read from a subpath etc.
There are a couple of ways to do this. An easy one, if you are using KFP SDK v0.1.18 or later, is to use the pvolumes argument in the ContainerOp's constructor: check this sample for example, or these lines as a mini documentation.

Note: VolumeOp is an op (i.e. a task of the pipeline) which makes the creation of PVCs easy, however it is not yet supported since it requires Argo v2.3 which is soon to be released.

[Ark-kun]

Generally this goes against the parallelism aspect of the pipelines.

The idea behind it is that containers share the same disk storage volume.

This is already possible to do multiple ways.

• ContainerOp.add_volume + ContainerOp.add_volume_mount
• kfp.onprem.mount_pvc
• Arrikto's VolumeOp

Is there a way to overcome the multiple loading issue currently across different steps (pods)?

We're leaning to standardizing on system-managed volume-based approach in future.