Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update dockerfile and build steps #562

Merged
merged 1 commit into from
Dec 18, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 3 additions & 11 deletions backend/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,23 +1,15 @@
FROM golang:alpine as builder

RUN apk add --no-cache curl git
RUN curl -fsSL -o /bin/dep https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 && chmod +x /bin/dep

# Download and install the latest release of dep
ADD https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 /usr/bin/dep
RUN chmod +x /usr/bin/dep

WORKDIR /go/src/github.com/kubeflow/pipelines
COPY . .

# Needed for github.com/mattn/go-sqlite3
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you keep this comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, accidently removed this line.

will add it back in next pr.

RUN apk add --update gcc musl-dev
RUN apk add --no-cache gcc musl-dev
RUN go build -o /bin/apiserver backend/src/apiserver/*.go

FROM python:3.5 as compiler

RUN apt-get update -y && \
apt-get install --no-install-recommends -y -q default-jdk wget
apt-get install --no-install-recommends -y -q default-jdk wget

RUN pip3 install setuptools==40.5.0

Expand All @@ -42,7 +34,7 @@ COPY ./samples .
RUN find . -maxdepth 2 -name '*.py' -type f | while read pipeline; do dsl-compile --py "$pipeline" --output "$pipeline.tar.gz"; done


FROM alpine
FROM alpine:3.8

ARG COMMIT_SHA=unknown
ENV COMMIT_SHA=${COMMIT_SHA}
Expand Down
11 changes: 1 addition & 10 deletions backend/Dockerfile.persistenceagent
Original file line number Diff line number Diff line change
@@ -1,26 +1,17 @@
FROM golang:alpine as builder

RUN apk add --no-cache curl git
RUN curl -fsSL -o /bin/dep https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 && chmod +x /bin/dep


# Download and install the latest release of dep
ADD https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 /usr/bin/dep
RUN chmod +x /usr/bin/dep

WORKDIR /go/src/github.com/kubeflow/pipelines
COPY . .

# Needed for github.com/mattn/go-sqlite3
RUN apk add --update gcc musl-dev
RUN go build -o /bin/persistence_agent backend/src/agent/persistence/*.go

FROM alpine
FROM alpine:3.8
WORKDIR /bin

COPY --from=builder /bin/persistence_agent /bin/persistence_agent
COPY --from=builder /go/src/github.com/kubeflow/pipelines/third_party/license.txt /bin/license.txt
RUN chmod +x /bin/persistence_agent

CMD persistence_agent \
--alsologtostderr=true
10 changes: 1 addition & 9 deletions backend/Dockerfile.scheduledworkflow
Original file line number Diff line number Diff line change
@@ -1,25 +1,17 @@
FROM golang:alpine as builder

RUN apk add --no-cache curl git
RUN curl -fsSL -o /bin/dep https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 && chmod +x /bin/dep

# Download and install the latest release of dep
ADD https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 /usr/bin/dep
RUN chmod +x /usr/bin/dep

WORKDIR /go/src/github.com/kubeflow/pipelines
COPY . .

# Needed for github.com/mattn/go-sqlite3
RUN apk add --update gcc musl-dev
RUN go build -o /bin/controller backend/src/crd/controller/scheduledworkflow/*.go

FROM alpine
FROM alpine:3.8
WORKDIR /bin

COPY --from=builder /bin/controller /bin/controller
COPY --from=builder /go/src/github.com/kubeflow/pipelines/third_party/license.txt /bin/license.txt
RUN chmod +x /bin/controller

CMD /bin/controller -alsologtostderr=true

22 changes: 11 additions & 11 deletions developer_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,28 +38,28 @@ $ cd ml-pipeline && ks delete default
### GKE
To be able to use GKE, the Docker images need to be uploaded to a public Docker repository, such as [GCR](https://cloud.google.com/container-registry/)

To build the API server image and upload it to GCR:
To build the API server image and upload it to GCR:
```bash
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/api-server:latest -f backend/Dockerfile ./backend
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/api-server:latest -f backend/Dockerfile .
# Push to GCR
$ gcloud auth configure-docker
$ docker push gcr.io/<your-gcp-project>/api-server:latest
```

To build the scheduled workflow controller image and upload it to GCR:
To build the scheduled workflow controller image and upload it to GCR:
```bash
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/scheduledworkflow:latest -f backend/Dockerfile.scheduledworkflow ./backend
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/scheduledworkflow:latest -f backend/Dockerfile.scheduledworkflow .
# Push to GCR
$ gcloud auth configure-docker
$ docker push gcr.io/<your-gcp-project>/scheduledworkflow:latest
```

To build the persistence agent image and upload it to GCR:
To build the persistence agent image and upload it to GCR:
```bash
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/persistenceagent:latest -f backend/Dockerfile.persistenceagent ./backend
# Run in the repository root directory
$ docker build -t gcr.io/<your-gcp-project>/persistenceagent:latest -f backend/Dockerfile.persistenceagent .
# Push to GCR
$ gcloud auth configure-docker
$ docker push gcr.io/<your-gcp-project>/persistenceagent:latest
Expand All @@ -68,7 +68,7 @@ $ docker push gcr.io/<your-gcp-project>/persistenceagent:latest
### Minikube
Minikube can pick your local Docker image so you don't need to upload to remote repository.

For example, to build API server image
For example, to build API server image
```bash
$ docker build -t ml-pipeline-api-server backend/src
```
Expand Down Expand Up @@ -138,7 +138,7 @@ API server logs are located at /tmp directory of the pod. To SSH into the pod, r
kubectl exec -it -n ${NAMESPACE} $(kubectl get pods -l app=ml-pipeline -o jsonpath='{.items[0].metadata.name}' -n ${NAMESPACE}) -- /bin/bash
```

**Q: How to check my cluster status if I am using Minikube?**
**Q: How to check my cluster status if I am using Minikube?**

Minikube provides dashboard for deployment
```bash
Expand Down