Error running pipeline: cannot create tfjobs.kubeflow.org 403

[lluunn]

Steps:

1. followed wiki to deploy pipeline.
2. I took the code from @amygdala's blog, ran it in jupyter notebook to build the tarball
3. Upload the tarball, and graph looks correct:

4. Running the pipeline, got error:

INFO:root:Getting credentials for GKE cluster kfp1.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for kfp1.
INFO:root:Generating training template.
INFO:root:Start training.
ERROR:root:Exception when calling DefaultApi->apis_fqdn_v1_namespaces_namespace_resource_post: tfjobs.kubeflow.org is forbidden: User "system:serviceaccount:kubeflow:pipeline-runner" cannot create tfjobs.kubeflow.org in the namespace "kubeflow"
Traceback (most recent call last):
  File "/ml/train.py", line 230, in <module>
    main()
  File "/ml/train.py", line 188, in main
    create_response = tf_job_client.create_tf_job(api_client, content_yaml, version=kf_version)
  File "/tf-operator/py/tf_job_client.py", line 56, in create_tf_job
    raise e
kubernetes.client.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 15 Nov 2018 22:32:05 GMT', 'Audit-Id': '316cc166-62f2-43aa-926e-8014f66f4b2e', 'Content-Length': '318', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"tfjobs.kubeflow.org is forbidden: User \"system:serviceaccount:kubeflow:pipeline-runner\" cannot create tfjobs.kubeflow.org in the namespace \"kubeflow\"","reason":"Forbidden","details":{"group":"kubeflow.org","kind":"tfjobs"},"code":403

[amygdala]

I think the issue is that you need to run this command or similar before running any pipelines:
kubectl create clusterrolebinding sa-admin --clusterrole=cluster-admin --serviceaccount=kubeflow:pipeline-runner
(there is another issue filed related to this... let me find it).

[amygdala]

Here's the related issue: #220

[lluunn]

Thanks!
I created the clusterrolebinding.
Now the error is:
py.util.JobTimeoutError: Timeout waiting for job trainer-s46w4 in namespace kubeflow to enter one of the conditions [{u'status': u'True', u'lastUpdateTime': u'2018-11-16T04:11:13Z', u'lastTransitionTime': u'2018-11-16T04:11:13Z', u'reason': u'TFJobCreated', u'message': u'TFJob trainer-s46w4 is created.', u'type': u'Created'}, {u'status': u'True', u'lastUpdateTime': u'2018-11-16T04:11:59Z', u'lastTransitionTime': u'2018-11-16T04:11:13Z', u'reason': u'TFJobRunning', u'message': u'TFJob trainer-s46w4 is running.', u'type': u'Running'}]

[amygdala]

Hmm, I have not seen this (and can't repro). :(
Do the logs of the tf-job-operator* pod have any clues?

Jeremy, I have a vague memory of your saying there was some recent issue with tf-job -- could this be related? @jlewi

[jlewi]

@amygdala Only issues I'm aware of are RBAC and IAM.

@IronPan might know more.

[jlewi]

@IronPan @vicaire Any update on this?

[vicaire]

@qimingj @gaoning777, could you please have a look?

[jlewi]

@qimingj @gaoning777 any update on this?

[IronPan]

The RBAC issue should be resolved with the latest version.
https://github.com/kubeflow/kubeflow/blob/master/kubeflow/pipeline/pipeline-apiserver.libsonnet#L322

@lluunn Could you give another try and if issue still persist, could you share me the pipeline definition and the kubeflow version?

[jlewi]

@IronPan Is there a pipelines test that covers firing off TFJob from pipelines? If not can we open up an issue to add such at test and use that to verify the fix is working?