Containers should not be allowed to directly create K8s objects

[Ark-kun]

Currently our containers are not really isolated. They have full access to network, GCS, other GCP services and can even create, delete and modify other K8s objects as in #231

This was discussed during a design meeting and as far as I remember the consensus was that it's too dangerous and insecure. Worse still, its really hard to put this genie back into the bottle once it's out and people start using this.

As we discussed, all CRDs should be created through the pipelines system backend endpoint or admission controller and the Pod needs to identify itself.

Technical solution:

• Pod should not be able to directly manipulate K8s in any way.
• To launch CRD, a Pod should send the CRD to a backend endpoint after identifying itself (e.g. send its name. The Pod should not be able to see any information about other pods and steal their identity).
• Backend must verify that the Pod's container image is in whitelist (a quick implementation can just check for the gcr.io/ml-pipelines/ container image bucket).
• Backend must verify that the container is running the intended code: We can check that the entry-point wasn't tampered with - the command property of the container should not be set.

[jlewi]

Isn't this what RBAC and namespaces are for?

What does it mean to send a CRD to some backend?

Why not just let pipelines run in a user selected namespace with suitable, user controlled RBAC permissions to limit what pipelines can do?

[vicaire]

+1 on @jlewi's comment.

I am not in favor of containers calling the Pipeline API Server.

I think we have two options:

• The one @jlewi mentioned
• Containers could also interact with the outside world only through local files, local ports, and local pipes, which the system would wire appropriately depending on the workflow configuration.

[vicaire]

Resolving since there is no outstanding item.