From 69363e4bc693d4f9f847fb5babcc09c53b607cef Mon Sep 17 00:00:00 2001 From: pratap0007 Date: Thu, 3 Mar 2022 21:26:13 +0530 Subject: [PATCH] Adds a doc for upgrading Hub v1.6.0 to v1.7.0 This doc defines the steps for upgrading Hub v1.6.0 to v1.7.0 as metadata name and labels are changed for db, api and ui configs. Updates deployment and development docs. Signed-off-by: Shiv Verma --- docs/DEPLOYMENT.md | 113 +++++++------ docs/DEVELOPMENT.md | 2 + docs/UPGRADE_V1.6.0_TO_V1.7.0.md | 278 +++++++++++++++++++++++++++++++ 3 files changed, 339 insertions(+), 54 deletions(-) create mode 100644 docs/UPGRADE_V1.6.0_TO_V1.7.0.md diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 143e23265c..fe383fefa8 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -44,10 +44,10 @@ Lets build and push all images to a registry and then create deployments. ### UI Image -Ensure you are in `ui` directory and logged in you image registry. +Ensure you are in the root of project and logged in you image registry. ``` -docker build -t . && docker push +docker build -f images/ui.Dockerfile -t . && docker push ``` Replace `` with the registry and image name. @@ -55,12 +55,12 @@ eg. `quay.io//ui` ### API & DB Migration Image -Ensure you are in `api` directory and logged in you image registry. +Ensure you are in the root of project and logged in you image registry. Build the API image using below command ``` -docker build -t . && docker push +docker build -f images/api.Dockerfile -t . && docker push ``` Replace `` with the registry and image name. @@ -69,7 +69,7 @@ eg. `quay.io//api` Now, Build the Db migration image using below command ``` -docker build -f db.Dockerfile -t . && docker push +docker build -f images/db.Dockerfile -t . && docker push ``` Replace `` with the registry and image name. @@ -79,6 +79,12 @@ Make sure all images are public before creating deployments. --- +> ### NOTE: If You have deployed Hub v1.6.0 and want to upgrade to Hub v1.7.0 then please follow [this][upgrade-v1.6.0-to-v1.7.0] doc for next steps. As we have renamed config metadata name and labels in the configs of db, api and ui. + +--- + +--- + > ### NOTE: In case of using Github or Gitlab Enterprise make sure you deploy Hub in the same network where the Enterprise server is running. Example if your Github/Gitlab Enterprise is running behind VPN then your Kubernetes Cluster should also be behind VPN. --- @@ -105,7 +111,7 @@ Wait till the pod comes in a running state kubectl get pod -n tekton-hub -w ``` -## Run Db migration +## Run DB migration Once the pod is in running state now we can run db migration. This will create all the tables required in the database. @@ -120,17 +126,17 @@ This will create a job which will read the db credentials from the secret create ```bash $ kubectl get pods -n tekton-hub NAME READY STATUS RESTARTS AGE -db-589d44fdd5-ksf8v 1/1 Running 0 50s -db-migration-8vhpd 0/1 Completed 0 17s +tekton-hub-db-589d44fdd5-ksf8v 1/1 Running 0 50s +tekton-hub-db-migration-8vhpd 0/1 Completed 0 17s ``` -The `Pod` will go into `Completed` state if the job succeeds. You can also check the logs using `kubectl logs -n tekton-hub db-migration-8vhpd`. In case of any failure the `Pod` will go into `Error` state. +The `Pod` will go into `Completed` state if the job succeeds. You can also check the logs using `kubectl logs -n tekton-hub tekton-hub-db-migration-8vhpd`. In case of any failure the `Pod` will go into `Error` state. ```bash $ kubectl get pods -n tekton-hub NAME READY STATUS RESTARTS AGE -db-589d44fdd5-ksf8v 1/1 Running 0 50s -db-migration-8vhpd 0/1 Error 0 17s +tekton-hub-db-589d44fdd5-ksf8v 1/1 Running 0 50s +tekton-hub-db-migration-8vhpd 0/1 Error 0 17s ``` ## Deploy API Service @@ -145,22 +151,23 @@ db-migration-8vhpd 0/1 Error 0 17s - If deploying on Kubernetes:- - - Create the secret containing tls cert and tls key both for API and Oauth server. Example as follows: + - Create the secret containing tls cert and tls key both for API and Oauth server. Example as follows: - ```bash - kubectl create secret tls api-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub - ``` + ```bash + kubectl create secret tls api-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub + ``` - - Apply the Ingress + - Apply the Ingress - ```bash - kubectl apply -f 04-kubernetes/40-api-ingress.yaml -f 04-kubernetes/40-auth-ingress.yaml -n tekton-hub - ``` + ```bash + kubectl apply -f 04-kubernetes/40-api-ingress.yaml -f 04-kubernetes/40-auth-ingress.yaml -n tekton-hub + ``` ### Create Git Oauth Applications -- **Github** - Create a GitHub OAuth with `Homepage URL` and `Authorization callback URL` as ``. Follow the steps given [here][github-oauth-steps] to create a GitHub OAuth. -- **Gitlab** - Create a Gitlab Oauth with `REDIRECT_URI` as `/auth/gitlab/callback`. Follow the steps given [here][gitlab-oauth-steps] to create a Gitlab Oauth -- **BitBucket** - Create a BitBucket Oauth with `Callback URL` as ``. Follow the steps given [here][bitbucket-oauth-steps] to create a BitBucket Oauth + +- **Github** - Create a GitHub OAuth with `Homepage URL` and `Authorization callback URL` as ``. Follow the steps given [here][github-oauth-steps] to create a GitHub OAuth. +- **Gitlab** - Create a Gitlab Oauth with `REDIRECT_URI` as `/auth/gitlab/callback`. Follow the steps given [here][gitlab-oauth-steps] to create a Gitlab Oauth +- **BitBucket** - Create a BitBucket Oauth with `Callback URL` as ``. Follow the steps given [here][bitbucket-oauth-steps] to create a BitBucket Oauth ### Update API Secret @@ -176,7 +183,7 @@ Edit `02-api/20-api-secret.yaml` and update the configuration apiVersion: v1 kind: Secret metadata: - name: api + name: tekton-hub-api namespace: tekton-hub type: Opaque stringData: @@ -189,7 +196,7 @@ stringData: JWT_SIGNING_KEY: a-long-signing-key ACCESS_JWT_EXPIRES_IN: time such as 15m REFRESH_JWT_EXPIRES_IN: time such as 15m - AUTH_BASE_URL: Oauth route + AUTH_BASE_URL: auth route GHE_URL: Add Github Enterprise URL in case of authenticating through Github Enterprise (Example (https|http)://myghe.com) --> Do not provide the catalog URL GLE_URL: Add Gitlab Enterprise URL in case of authenticating through Gitlab Enterprise (Example (https|http)://mygle.com) --> Do not provide the catalog URL ``` @@ -198,9 +205,9 @@ stringData: - Update the [config.yaml][config-yaml] to add at least one user with all scopes such as - - refresh a catalog, - - refresh config file - - create an agent token + - refresh a catalog, + - refresh config file + - create an agent token - For example @@ -222,10 +229,9 @@ scopes: apiVersion: v1 kind: ConfigMap metadata: - name: api - namespace: tekton-hub + name: tekton-hub-api labels: - app: api + app: tekton-hub-api data: CONFIG_FILE_URL: https://raw.githubusercontent.com/tektoncd/hub/master/config.yaml ## Change the file URL here ``` @@ -242,10 +248,10 @@ please follow the below steps: 1. Create the ssh keys if not already present in `~/.ssh` dir. 1. Create kubernetes secret which will contain the ssh keys using the following command - ```sh - kubectl create secret generic tekton-hub-api-ssh-crds --from-file=id_rsa="/path/to/id_rsa" --from-file=id_rsa.pub="/path/to/id_rsa.pub" --from-file=known_hosts="/path/to/known_hosts" - ``` - Please make sure that secrets are created with the name `tekton-hub-api-ssh-crds` + ```sh + kubectl create secret generic tekton-hub-api-ssh-crds --from-file=id_rsa="/path/to/id_rsa" --from-file=id_rsa.pub="/path/to/id_rsa.pub" --from-file=known_hosts="/path/to/known_hosts" + ``` + Please make sure that secrets are created with the name `tekton-hub-api-ssh-crds` ### Update API Image @@ -255,7 +261,7 @@ Edit the `02-api/22-api-deployment.yaml` and replace the image with the one crea kubectl apply -f 02-api/ -n tekton-hub ``` -This will create the deployment, secret, configmap and a NodePort service to expose the API server. +This will create the pvc, deployment, secret, configmap and a NodePort service to expose the API server. #### Verify if api route is accessible @@ -283,26 +289,26 @@ After the deployment is done successfully, we need to expose the URL to access t - If deploying on Kubernetes:- - - Create the secret containing tls cert and tls key + - Create the secret containing tls cert and tls key - ```bash - kubectl create secret tls ui-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub - ``` + ```bash + kubectl create secret tls ui-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub + ``` - - Apply the Ingress + - Apply the Ingress - ```bash - kubectl apply -f 04-kubernetes/41-ui-ingress.yaml -n tekton-hub - ``` + ```bash + kubectl apply -f 04-kubernetes/41-ui-ingress.yaml -n tekton-hub + ``` ### Update UI ConfigMap -Edit `config/03-ui/30-ui-configmap.yaml` and by setting the API URL, Oauth URL and Redirect URL. +Edit `config/03-ui/30-ui-configmap.yaml` and by setting the API URL, auth URL and Redirect URL. You can get the API URL using below command (OpenShift) ``` - kubectl get -n tekton-hub routes api --template='https://{{ .spec.host }}' + kubectl get -n tekton-hub routes tekton-hub-api --template='https://{{ .spec.host }}' ``` ```yaml @@ -310,12 +316,11 @@ You can get the API URL using below command (OpenShift) apiVersion: v1 kind: ConfigMap metadata: - name: ui - namespace: tekton-hub + name: tekton-hub-ui data: API_URL: API URL <<< Update this by API url API_VERSION: API VERSION <<< Update this by API version (For e.g-"v1") - AUTH_BASE_URL: OAUTH URL << Update this by oauth url + AUTH_BASE_URL: AUTH URL << Update this by auth url REDIRECT_URI: UI URL << Update this by ui url ``` @@ -339,10 +344,10 @@ eg. wait till status of UI pod is running ``` NAME READY STATUS RESTARTS AGE -api-6dfc6f97d9-vk66r 1/1 Running 3 21m -db-9bd4cdf99-zsq89 1/1 Running 0 21m -db-migration-26ngs 0/1 Completed 0 21m -ui-55fc66cc6b-69dsp 1/1 Running 0 58s +tekton-hub-api-6dfc6f97d9-vk66r 1/1 Running 3 21m +tekton-hub-db-9bd4cdf99-zsq89 1/1 Running 0 21m +tekton-hub-db-migration-26ngs 0/1 Completed 0 21m +tekton-hub-ui-55fc66cc6b-69dsp 1/1 Running 0 58s ``` #### Verify if ui route is accessible @@ -350,7 +355,7 @@ ui-55fc66cc6b-69dsp 1/1 Running 0 58s For `OpenShift`:- ``` - kubectl get routes -n tekton-hub ui --template='https://{{ .spec.host }} ' + kubectl get routes -n tekton-hub tekton-hub-ui --template='https://{{ .spec.host }} ' ``` Open the URL in a browser. @@ -435,7 +440,6 @@ NOTE: The catalog refresh will add new resources or update existing resource if kind: Secret metadata: name: catalog-refresh - namespace: tekton-hub type: Opaque stringData: HUB_TOKEN: hub token @@ -526,3 +530,4 @@ Replace `` with your JWT token. you must have `config-refresh` sco [catalog]: https://github.com/tektoncd/catalog [catalog-tep]: https://github.com/tektoncd/community/blob/main/teps/0003-tekton-catalog-organization.md [hub-config]: https://github.com/tektoncd/hub/blob/main/config.yaml +[upgrade-v1.6.0-to-v1.7.0]: https://github.com/tektoncd/hub/blob/main/docs/UPGRADE_V1.6.0_TO_V1.7.0.md diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 3b23a01b53..96d62e24d8 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -112,6 +112,8 @@ For `JWT_SIGNING_KEY`, you can use any random word. For `ACCESS_JWT_EXPIRES_IN` and `REFRESH_JWT_EXPIRES_IN` you can set the time as per your convenience. Example `15m`, `10y`. +For `AUTH_BASE_URL`, use the `http://localhost:4200` + ### Running API Service Once the database is setup and the migration has been run, you can run api service by diff --git a/docs/UPGRADE_V1.6.0_TO_V1.7.0.md b/docs/UPGRADE_V1.6.0_TO_V1.7.0.md new file mode 100644 index 0000000000..c1d0947c80 --- /dev/null +++ b/docs/UPGRADE_V1.6.0_TO_V1.7.0.md @@ -0,0 +1,278 @@ +# Upgrade Hub v1.6.0 to v1.7.0 + +This doc defines the steps to follow for upgrading from Hub v1.6.0 to Hub v1.7.0. + +Navigate to `config` directoy in the root of the project. + +## Get the backup of Hub v1.6.0 db + +Follow these steps sequentially to take the backup of Hub v1.6.0 DB:- + +Execute the below command to get the Hub v1.6.0 db pod + +``` +kubectl get pods -l app=db -n tekton-hub +``` + +Exec into the db pod + +``` +kubectl exec -it /bin/bash +``` + +Dump the Hub v1.6.0 db by running below commands inside the db pod + +``` +pg_dump -Ft -h localhost -U postgres hub -f /tmp/db.dump +``` + +Exit from the pod and copy the Hub v1.6.0 db dump file from the db pod to your system + +``` +kubectl cp :/tmp/db.dump ./backup_db.dump +``` + +## Deploy Databse + +Execute the below command + +``` +kubectl apply -f 00-init/ +``` + +This will create `tekton-hub` namespace, db deployment with `postgres` image, secret to save db credentials, pvc for the db and a ClusterIP service to expose the db internally. + +All resources are created in `tekton-hub` namespace. + +Wait till the pod comes in a running state + +``` +kubectl get pod -l app=tekton-hub-db -n tekton-hub -w +``` + +## Restore Hub v1.6.0 db backup into Hub v1.7.0 db + +Access the Hub v1.7.0 db pod and exec into the pod and restore the db backup + +``` +kubectl get pods -l app=tekton-hub-db -n tekton-hub +``` + +Copy the db backup file from your system to the Hub v1.7.0 db pod + +``` + +kubectl cp backup_db.dump :/tmp/backup_db.dump + +``` + +Now restore the dump file into Hub v1.7.0 db pod by running below command inside the pod + +``` +pg_restore -d hub -h localhost -U postgres /tmp/backup_db.dump + +``` + +To check succesful backup of hub v1.6.0 db, run the following commands inside the Hub v1.7.0 db pod and resources count shouldn't be zero + +``` + +psql -U postgres + +\c hub; + +select count(*) from resources; + +``` + +Exit from the db pod + +## Run the DB Migration + +Edit the `01-db/10-db-migration.yaml` and replace the image with Hub v1.7.0 DB migration image and apply the yaml. + +``` + +kubectl apply -f 01-db/10-db-migration.yaml -n tekton-hub + +``` + +## Deploy the API service + +### Setup Route or Ingress + +- If deploying on OpenShift:- + + ```bash + kubectl apply -f 04-openshift/40-api-route.yaml -f 04-openshift/40-auth-route.yaml -n tekton-hub + ``` + +```` + +- If deploying on Kubernetes:- + + - Create the secret containing tls cert and tls key both for API and Oauth server. Example as follows: + + ```bash + kubectl create secret tls api-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub + ``` + + - Apply the Ingress + + ```bash + kubectl apply -f 04-kubernetes/40-api-ingress.yaml -f 04-kubernetes/40-auth-ingress.yaml -n tekton-hub + ``` + +### Update API Secret + +Eexcute the below command to get the secret of Hub v1.6.0 api + +``` +kubectl get secret api -o yaml +``` + +Copy the secret values and update the `02-api/20-api-secret.yaml` file with same value + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: tekton-hub-api +type: Opaque +stringData: + GH_CLIENT_ID: Github Oauth client id + GH_CLIENT_SECRET: Github Oauth secret + GL_CLIENT_ID: Gitlab Oauth client id + GL_CLIENT_SECRET: Gitlab Oauth secret + BB_CLIENT_ID: BitBucket Oauth client id + BB_CLIENT_SECRET: BitBucket Oauth secret + JWT_SIGNING_KEY: a-long-signing-key + ACCESS_JWT_EXPIRES_IN: time such as 15m + REFRESH_JWT_EXPIRES_IN: time such as 15m + AUTH_BASE_URL: auth route of Hub v1.7.0 + GHE_URL: Add Github Enterprise URL in case of authenticating through Github Enterprise (Example (https|http)://myghe.com) --> Do not provide the catalog URL + GLE_URL: Add Gitlab Enterprise URL in case of authenticating through Gitlab Enterprise (Example (https|http)://mygle.com) --> Do not provide the catalog URL +``` + +### Update API ConfigMap + +Execute the below command to get the `CONFIG_FILE_URL` of Hub v1.6.0 api config + +``` +kubectl get cm/api -o=jsonpath="{.data['CONFIG_FILE_URL']}" +``` + +Copy the `CONFIG_FILE_URL` url path from Hub v1.6.0 configMap and update the `02-api/21-api-configmap.yaml` with same url path + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: tekton-hub-api + labels: + app: tekton-hub-api +data: + CONFIG_FILE_URL: https://raw.githubusercontent.com/tektoncd/hub/master/config.yaml ## Change the file URL here with Hub v1.6.0 configMap's CONFIG_FILE_URL +``` + +### Update API Image + +Edit the `02-api/22-api-deployment.yaml` and replace the image with Hub v1.7.0 API image + +### Apply API configs + +```bash +kubectl apply -f 02-api/ -n tekton-hub +``` + +This will create the pvc, deployment, secret, configmap and a NodePort service to expose the API server. + +## Deploy the UI service + +### Setup Route or Ingress + +- If deploying on OpenShift:- + + ```bash + kubectl apply -f 04-openshift/41-ui-route.yaml -n tekton-hub + ``` + +- If deploying on Kubernetes:- + + - Create the secret containing tls cert and tls key + + ```bash + kubectl create secret tls ui-hub-tekton-dev-tls --cert=path/to/cert/file --key=path/to/key/file -n tekton-hub + ``` + + - Apply the Ingress + + ```bash + kubectl apply -f 04-kubernetes/41-ui-ingress.yaml -n tekton-hub + ``` + +### Update UI ConfigMap + +Edit `config/03-ui/30-ui-configmap.yaml` and by setting the API URL, API version auth URL and Redirect URL. + +You can get the API URL,AUTH URL and UI URL for Hub v1.7.0 by using below command (OpenShift) + +``` + kubectl get routes tekton-hub-api --template='https://{{ .spec.host }}' -n tekton-hub +``` + +``` + kubectl get routes tekton-hub-auth --template='https://{{ .spec.host }}' -n tekton-hub +``` + +``` + kubectl get routes tekton-hub-ui --template='https://{{ .spec.host }}' -n tekton-hub +``` + +```yaml +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: tekton-hub-ui +data: + API_URL: API URL <<< Update this by Hub v1.7.0 API url + API_VERSION: API VERSION <<< Update this by API version (For e.g-"v1") + AUTH_BASE_URL: AUTH URL << Update this by hub v1.7.0 auth url + REDIRECT_URI: UI URL << Update this by hub v1.7.0 ui url +``` + +### Update UI Image + +Edit the `03-ui/31-ui-deployment.yaml` by replacing the image with Hub v1.7.0 UI image + +### apply API configs + +Execute follow command + +```bash +kubectl apply -f 03-ui/ -n tekton-hub +``` + +### Update the Git OAuth Configuration with Hub v1.7.0 auth routes + +- **Github** - Update GitHub OAuth with `Homepage URL` and `Authorization callback URL` as Huv v1.7.0 ``. +- **Gitlab** - Update Gitlab Oauth with `REDIRECT_URI` as `/auth/gitlab/callback`,use the Hub v1.7.0 auth route as ``. +- **BitBucket** - Update BitBucket Oauth with `Callback URL` as Hub v1.7.0 ``. + +## Delete the db, api and ui resoruces for Hub v1.6.0 + +Verify the Hub v1.7.0 API, DB and UI if it's working fine then delete the all resources for Hub v1.6.0 by exucuting the following commands + +``` +kubectl delete secret,pvc,deployment,service -l app=db -n tekton-hub +``` + +``` +kubectl delete secret,pvc,cm,deployment,service,route -l app=api -n tekton-hub +``` + +``` +kubectl delete cm,deployment,service,route -l app=ui -n tekton-hub +``` +````