Silent failure to trigger Argo Workflow from CLI

[notablegrace]

If the workflow exists in the given namespace, it seems that python flow.py argo-workflows trigger can yield this output: Workflow 'foo' triggered on Argo Workflows (run-id 'bar')., even if the workflow was not triggered.

We use port forwarding to connect to our Argo Workflows server. If someone forget to port forward and attempts to run this command, the output is misleading because a run will not get triggered. I confirmed that I was able to create the run as expected when I used port forwarding.

[savingoyal]

@notablegrace, the run ID is the output from the kubernetes server. As long as the kubernetes API server is reachable - metaflow will register a workflow object with kubernetes. That workflow object may or may not result in an actual metaflow run object - usually you would need to introduce some supervisor process that can check if there are workflow objects that haven't resulted in a run object and alert accordingly.

also, the port forwards (as introduced through the deployment templates) are more for rapidly evaluating metaflow and are not recommended for a production setup.

[notablegrace]

Really appreciate the speedy explanation @savingoyal !