Skip to content

Commit

Permalink
Cicd delete old prs (kubeflow#575)
Browse files Browse the repository at this point in the history
* Define a script to close obsolete PRs to update an application.

* In the event there are multiple PRs open to update a Kubeflow application
  we want to close the older PRs; so there is a single open PR updating
  the application to the newest code.

Related to kubeflow#571

* Define a script to close obsolete PRs to update an application.

* In the event there are multiple PRs open to update a Kubeflow application
  we want to close the older PRs; so there is a single open PR updating
  the application to the newest code.

  Related to kubeflow#571

* Setup a def namespace for use with apps-cd.

* Update update_kf_apps.py to close old PRs on each sync.

* Bake the source code into the docker image rather than using a wrapper
  script to sync the code from git.

  * Sync'ing the code from git became to difficult to reason about once
    we start splitting the source code across multiple repositories
    * We now depend on github/kubeflow/code-intelligence for utilities
      for working with GitHub Apps

    * Using a docker image also ensures we don't get broken suddenly when
      new changes are in place

    * In the future we could use github actions to automate updating the
      deployment on postsubmits

* Turn app-pipeline.template.yaml into a ConfigMap
  * This allows better versioning
  * We can rely on kustomize to create a configmap with a hash based on the
    contents
  * kustomize will then reference the config map using its hash. As
    a result a rolling update is triggered whenever the hash contents changes.
  * This makes it easier to handle rollous and updates.

Define a dev instance of the update KF apps infrastructure to facilitate development
  * Use profiles in skaffold.

  * update_kf_apps.py in dev uses a config map now to ubtain app-pipeline.template.yaml
    rather than fetching it from git

  * This makes it much easier to test changes in the dev instance

Fix a bunch of bugs preventing update_kf_apps.py from working
   * Update requirements.txt with a bunch of missing packages.
   * Fix some imports in update_kf_apps.py

  * Need to set resource requests for the build pods otherwise builds get
    CPU starved and take forever.

Miscellaneous

* Create a tool to copy secrets between namespaces from GCS

* Fix lint.

* Due to kubeflow#460 we need to disable pylint.
  • Loading branch information
jlewi authored and k8s-ci-robot committed Jan 28, 2020
1 parent 7464a22 commit 65c84fd
Show file tree
Hide file tree
Showing 22 changed files with 645 additions and 358 deletions.
28 changes: 19 additions & 9 deletions apps-cd/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Build the docker image used to run the scripts
# to continuously update our docker files.
#
# The context for this docker file should be the root of the kubeflow/testing repository.
FROM ubuntu:18.04

RUN apt-get update -y && \
Expand All @@ -22,22 +26,28 @@ RUN export KUSTOMIZE_VERSION=3.2.0 && \
mv kustomize_${KUSTOMIZE_VERSION}_linux_amd64 /usr/local/bin/kustomize && \
chmod a+x /usr/local/bin/kustomize


RUN python -m pip install fire \
google-api-python-client \
google-cloud-storage \
kubernetes \
watchdog

COPY apps-cd/requirements.txt /tmp
RUN python -m pip install \
-r /tmp/requirements.txt

# Create go symlinks
RUN ln -sf /usr/local/go/bin/go /usr/local/bin && \
ln -sf /usr/local/go/bin/gofmt /usr/local/bin && \
ln -sf /usr/local/go/bin/godoc /usr/local/bin

RUN mkdir -p /app
COPY update_launcher.py /app
COPY run_with_auto_restart.py /app

RUN cd /app && \
mkdir -p /app/src/kubeflow && \
cd /app/src/kubeflow && \
git clone https://github.com/kubeflow/code-intelligence code-intelligence && \
cd code-intelligence && \
git checkout db1230d

COPY apps-cd/run_with_auto_restart.py /app
COPY py /app/src/kubeflow/testing/py

ENV PYTHONPATH /app/src/kubeflow/testing/py:/app/src/kubeflow/code-intelligence/py:$PYTHONPATH

# See(https://github.com/tektoncd/pipeline/issues/1271): Tekton will put ssh
# credentials in /tekton/home. We can't change the home directory
Expand Down
21 changes: 12 additions & 9 deletions apps-cd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,7 @@ and open PRs to update Kubeflow kustomize manifests to use the newly built image

* The kubeflow-bot GitHub account is used to create the PRs

* Continuous building is achieved by running `update_launcher.py` in a Kubernetes deployment

* This script periodically fetches the latest code in `kubeflow/testing` to pick up any changes
to the code or config

* It then launches `update_kf_apps.py` to create Tekton PipelineRuns for any applications that
need to be updated.
* Continuous building is achieved by running `update_kf_apps.py` in a Kubernetes deployment

### Adding Applications to continuous delivery

Expand Down Expand Up @@ -150,12 +144,21 @@ This is a Kubeflow cluster (v0.6.2) and we rely on that to configure certain thi

## Developer Guide

You can use skaffold to build a docker image and auto update the deployment running `update_launcher.py`
You can use skaffold to build a docker image and auto update the deployment running `update_kf_apps.py`

* The namespace `kf-releasing-dev` is intended for trying out your local changes
* You can update `apps-cd/pipelines/base/config/app-pipeline.template` to pull in your branch of changes to the CD tooling

* The template defines a PipelineRun that is used to generate the PipelineRun's to update each application
* The template defines a Git resource which points to the kubeflow/testing repository to obtain the scripts to update the manifests
* You can change this to point to your fork of kubeflow/testing to test your changes
* You can also deploy any changes to tekton resources to that namespace before trying out
in prod.

1. Run skaffold

```
skaffold dev -v info --cleanup=false --trigger=polling
skaffold dev -p dev -v info --cleanup=false --trigger=polling
```

1. During development you can take advantage of skaffold's continuous file sync mode to update
Expand Down
23 changes: 0 additions & 23 deletions apps-cd/applications.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -119,26 +119,3 @@ versions:
value: v1.0-branch
- name: url
value: git@github.com:kubeflow/manifests.git
# Define a v0-8 release
# This is primarily so we can test that the CI/CD infrastructure is workin
# with release branches. We don't plan on actually releasing 0.8
- name: v0-8
# A tag to prefix image names with
tag: "v0.8.0"
repos:
- name: kubeflow
resourceSpec:
type: git
params:
- name: revision
value: v0.8-branch
- name: url
value: git@github.com:kubeflow/kubeflow.git
- name: manifests
resourceSpec:
type: git
params:
- name: revision
value: v0.8-branch
- name: url
value: git@github.com:kubeflow/manifests.git
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,13 @@ spec:
# https://github.com/kubeflow/testing/pull/572
# is merged
- name: revision
value: master
value: cicd_delete_old_prs
- name: url
value: git@github.com:kubeflow/testing.git
value: git@github.com:jlewi/testing.git
#- name: revision
# value: master
#- name: url
# value: git@github.com:kubeflow/testing.git
# The image we want to build
- name: image
resourceSpec:
Expand Down
52 changes: 29 additions & 23 deletions apps-cd/pipelines/base/deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,41 +17,47 @@ spec:
- name: app
image: gcr.io/kubeflow-releasing/update_kf_apps
command:
# Uncomment the following lines when running with skaffold
# and you want to use skaffold's auto-sync feature wto pick up changes
# to update_launcher.
# Begin skaffold
- python
- run_with_auto_restart.py
- --dir=/app
- --
# End skaffold
#
- python
- update_launcher.py
- run
# repo_dir is the directory where kubeflow testing should be checked out to
- --repo_dir=/launcher_src/kubeflow/testing
# repo is the code to check out to get the configuration as well as code for
# launching the applications
#
# To test changes before they are checked in you can set this to the repo containing your PR
# e.g. - --repo=https://github.com/jlewi/testing.git?ref=cicd_0.8
- --repo=https://github.com/kubeflow/testing.git
- -m
- kubeflow.testing.cd.update_kf_apps
- sync
# Extra arguments to be passed to update_kf_apps.py
- --namespace=kf-releasing
- --config=/launcher_src/kubeflow/testing/apps-cd/applications.yaml
- --namespace=kf-releasing
# TODO(jlewi): Should we just put this in a config map as well and then check in into source control?
# Would a tool like Anthos CM be smart enough to detect changes in that file and trigger
# updates with kustomize?
- --config=https://raw.githubusercontent.com/kubeflow/testing/master/apps-cd/applications.yaml
- --output_dir=/tmp/runs
- --src_dir=/src
- --template=/launcher_src/kubeflow/testing/apps-cd/runs/app-pipeline.template.yaml
- --template=/app/config/app-pipeline.template.yaml
env:
# TODO(jlewi): We should stop using a GITHUB_TOKEN and switch to using
# the github APP. Is this only used by the Hub CLI?
- name: GITHUB_TOKEN
valueFrom:
secretKeyRef:
name: github-token
key: github_token
- name: GITHUB_APP_PEM_KEY
value: /var/secrets/github/kubeflow-auto-bot.2020-01-24.private-key.pem
resources:
requests:
cpu: 4
memory: 8Gi
workingDir: /app
volumeMounts:
- name: github-app
mountPath: /var/secrets/github
volumeMounts:
- name: pipelinerun-template
mountPath: /app/config
volumes:
# Secret containing the PEM key for the GitHub app used to close PRs.
- name: github-app
secret:
secretName: github-app-pem
- name: pipelinerun-template
configMap:
# Kustomize will automatically replace the name with the unique name given
# to the configmap based on the config contents.
name: pipelinerun-template
6 changes: 6 additions & 0 deletions apps-cd/pipelines/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,9 @@ resources:
- task.yaml
- pipeline.yaml
namespace: kf-releasing
# Create a configMap containing the template for the pipeline run
configMapGenerator:
- name: pipelinerun-template
files:
# key will be name of the file
- ./config/app-pipeline.template.yaml
22 changes: 11 additions & 11 deletions apps-cd/pipelines/base/task.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,11 @@ spec:
- --digest-file=/workspace/image-digest
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /secret/user-gcp-sa.json
value: /secret/user-gcp-sa.json
resources:
requests:
cpu: 7
memory: 16Gi
volumeMounts:
- mountPath: /secret
name: gcp-credentials
Expand Down Expand Up @@ -88,8 +92,8 @@ spec:
- -m
- kubeflow.testing.cd.create_manifests_pr
- apply
- --image_url=${IMAGE_URL}
- --src_image_url=${SRC_IMAGE_URL}
- --image_url=$(inputs.resources.image.url)
- --src_image_url=$(inputs.params.src_image_url)
- --manifests_dir=/workspace/$(inputs.resources.manifests.name)/$(inputs.params.path_to_manifests_dir)
- --manifests_base=$(inputs.resources.manifests.revision)
env:
Expand All @@ -102,14 +106,10 @@ spec:
secretKeyRef:
name: github-token
key: github_token
- name: SRC_IMAGE_URL
value: $(inputs.params.src_image_url)
- name: IMAGE_URL
value: $(inputs.resources.image.url)
- name: MANIFESTS_REPO_REVISION
value: $(inputs.resources.manifests.revision)
- name: PATH_TO_MANIFESTS_DIR
value: $(inputs.params.path_to_manifests_dir)
resources:
requests:
cpu: 4
memory: 4Gi
volumeMounts:
- mountPath: /secret
name: gcp-credentials
Expand Down
33 changes: 33 additions & 0 deletions apps-cd/pipelines/overlays/dev/deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: server
spec:
template:
spec:
containers:
- name: app
image: gcr.io/kubeflow-releasing/update_kf_apps
command:
# In the dev environment we run with auto_restart for compatibility with
# skaffold
- python
- run_with_auto_restart.py
- --dir=/app
- --
- python
- -m
- kubeflow.testing.cd.update_kf_apps
- sync
# repo is the code to check out to get the configuration as well as code for
# launching the applications
#
# To test changes before they are checked in you can set this to the repo containing your PR
# e.g.
# Extra arguments to be passed to update_kf_apps.py
- --namespace=kf-releasing-dev
- --config=https://raw.githubusercontent.com/jlewi/testing/cicd_delete_old_prs/apps-cd/applications.yaml
- --output_dir=/tmp/runs
- --src_dir=/src
- --template=/app/config/app-pipeline.template.yaml

9 changes: 9 additions & 0 deletions apps-cd/pipelines/overlays/dev/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
commonLabels:
environment: dev
namespace: kf-releasing-dev
patchesStrategicMerge:
- deployment.yaml
7 changes: 7 additions & 0 deletions apps-cd/pipelines/overlays/prod/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../base
commonLabels:
environment: prod
namespace: kf-releasing
20 changes: 20 additions & 0 deletions apps-cd/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
-i https://pypi.org/simple
asn1crypto==0.24.0
cffi==1.13.2
fire==0.2.1
github3.py==1.3.0
google-api-core==1.14.2
google-api-python-client==1.7.10
google-auth==1.6.3
google-auth-httplib2==0.0.3
google-cloud-core==1.0.3
google-cloud-storage==1.17.0
json-log-formatter==0.2.0
jwcrypto==0.6.0
kubernetes==9.0.0
pyjwt==1.7.1
requests==2.22.0
requests-oauthlib==1.2.0
requests-toolbelt==0.9.1
tqdm==4.42.0
watchdog==0.9.0
4 changes: 0 additions & 4 deletions apps-cd/runs/README.md

This file was deleted.

54 changes: 0 additions & 54 deletions apps-cd/runs/profile_controller_v1795828.yaml

This file was deleted.

Loading

0 comments on commit 65c84fd

Please sign in to comment.