Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy a TFX pipeline from the command line #1127

Closed
ianhellstrom opened this issue Apr 10, 2019 · 7 comments
Closed

Deploy a TFX pipeline from the command line #1127

ianhellstrom opened this issue Apr 10, 2019 · 7 comments

Comments

@ianhellstrom
Copy link

Is it possible to deploy a TFX pipeline from the command line? The examples in notebooks all show the notebook has to run inside the Kubernetes cluster that the pipelines are supposed to run.

It would be helpful to not have to go through the UI for deploying pipelines (and experiments/runs).

@swiftdiaries
Copy link
Member

You can do this now from the Pipelines CLI
Look here for instructions to install the CLI

@ianhellstrom
Copy link
Author

That does not seem to work on my system:

go get -u github.com/kubeflow/pipelines/backend/src/cmd/ml
go: finding github.com/kubeflow/pipelines/backend/src/cmd/ml latest
go: finding github.com/kubeflow/pipelines/backend/src/cmd latest
go: finding github.com/kubeflow/pipelines/backend/src latest
go: finding github.com/kubeflow/pipelines/backend latest
go: finding github.com/kubeflow/pipelines latest
go: finding github.com/modern-go/concurrent latest
go: finding github.com/gopherjs/gopherjs latest
go: finding golang.org/x/time latest
go: finding github.com/VividCortex/mysqlerr latest
go: finding github.com/globalsign/mgo latest
go: finding github.com/smartystreets/assertions latest
go: finding k8s.io/apimachinery latest
go: finding github.com/docker/spdystream latest
go: finding golang.org/x/net latest
go: finding github.com/lann/builder latest
go: finding k8s.io/apiextensions-apiserver latest
go: finding github.com/ugorji/go/codec latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/sync latest
go: finding golang.org/x/crypto latest
go: finding github.com/golang/groupcache latest
go: finding github.com/erikstmartin/go-testdb latest
go: finding gopkg.in/tomb.v1 latest
go: finding github.com/smartystreets/goconvey latest
go: finding github.com/elazarl/goproxy latest
go: finding github.com/robfig/cron latest
go: finding github.com/lann/ps latest
go: finding github.com/asaskevich/govalidator latest
go: finding github.com/prometheus/procfs latest
go: finding github.com/mxk/go-flowrate latest
go: finding google.golang.org/genproto latest
go: finding github.com/jinzhu/inflection latest
go: finding github.com/mailru/easyjson latest
go: finding github.com/armon/consul-api latest
go: finding github.com/denisenkom/go-mssqldb latest
go: finding github.com/mattbaird/jsonpatch latest
go: finding k8s.io/apiserver latest
go: finding github.com/gregjones/httpcache latest
go: finding github.com/golang/glog latest
go: finding golang.org/x/exp latest
go: finding golang.org/x/lint latest
go: finding golang.org/x/tools latest
go: finding github.com/alecthomas/template latest
go: finding github.com/golang/lint latest
go: finding k8s.io/kube-openapi latest
go: finding golang.org/x/arch latest
go: finding github.com/beorn7/perks latest
go: finding golang.org/x/image latest
go: finding github.com/google/pprof latest
go: finding honnef.co/go/tools latest
go: finding github.com/kr/logfmt latest
go: finding golang.org/x/mobile latest
go: finding k8s.io/api latest
go: finding github.com/mwitkow/go-conntrack latest
go: finding github.com/pquerna/ffjson latest
go: finding golang.org/x/oauth2 latest
go: finding github.com/pquerna/cachecontrol latest
go: finding gopkg.in/check.v1 latest
go: finding golang.org/x/build latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/tmc/grpc-websocket-proxy latest
go: finding k8s.io/utils latest
go: finding github.com/Azure/go-ansiterm latest
go: finding github.com/BurntSushi/xgb latest
go: finding github.com/gliderlabs/ssh v0.1.3
go: finding github.com/rcrowley/go-metrics latest
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding k8s.io/component-base latest
go: finding github.com/coreos/pkg latest
go: finding github.com/prometheus/client_model latest
go get: error loading module requirements

After that I try the ml command but it's not in my $GOPATH because the installation failed. Any ideas on how to fix that. I'm running go1.12.3 darwin/amd64 on MacOS.

@ianhellstrom
Copy link
Author

I tried it without setting GO111MODULE=on and it worked.

@ianhellstrom
Copy link
Author

However, when trying to create a pipeline from a local tar.gz file it fails with an internal server error.

I tried: ml pipeline create -n kubeflow --name 'Whatever' --url $(pwd)/pipeline.tar.gz --debug

The response:

{"error":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\"","message":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\"","code":13,"details":[{"@type":"type.googleapis.com/api.Error","error_message":"Internal Server Error","error_details":"InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme \"\""}]}
InternalServerError: Failed to download the pipeline from /Users/ian/tfx/pipeline.tar.gz Please double check the URL is valid and can be accessed by the pipeline system.: Get /Users/ian/tfx/pipeline.tar.gz: unsupported protocol scheme "" (code: 13)

@ianhellstrom
Copy link
Author

Sorry, for the spam: I should have checked the upload command. That works.

@swiftdiaries
Copy link
Member

Awesome ! Glad it works :)

@hamedhsn
Copy link
Contributor

@swiftdiaries the link is broken can you point me to the right url. thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants