Update exercises 13 and 14 to use YAML plugin instead of gomatic

dtsato · dtsato · commit a2efbc9d2ae4 · 2019-05-08T21:29:29.000+01:00
diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@ sharing, DevOps is showing that developers and system engineers have much to
 learn from each other. Through a series of hands-on exercises, Danilo Sato will
 use a sample web application to demonstrate how to automate its build and
 deployment pipeline, using infrastructure and pipeline as code techniques. You
-will learn how to combine tools such as Docker, Terraform, GoCD, Gomatic, and
+will learn how to combine tools such as Docker, Terraform, GoCD, and
 Kubernetes to create deployment pipelines for your infrastructure, your services,
 and applications. But even if your company is not using any of these tools, we
 will discuss alternative technologies and highlight the patterns and principles
diff --git a/SETUP.md b/SETUP.md
@@ -111,7 +111,6 @@ can create a project for the workshop and name it "DevOps Workshop".
   `sudo` on Linux):
   * `docker pull openjdk:8-jdk-alpine`
   * `docker pull mysql:5.7`
-  * `docker pull dtsato/gomatic`
 2. **Maven Dependencies**: have Maven download all dependencies by running:
   * `./mvnw clean install`
 3. **Minikube VM**: ensure you have the minikube VirtualBox VM image by running:
diff --git a/instructions/13-pipeline-as-code.md b/instructions/13-pipeline-as-code.md
@@ -3,193 +3,112 @@
 ## Goals
 
 * Learn about Pipeline as Code
-* Learn about GoMatic
-* Create a new pipeline for updating GoCD pipelines
+* Learn about GoCD YAML Config Plugin
+* Learn about GoCD Environments
+* Move PetClinic pipeline definition to YAML
 
 ## Acceptance Criteria
 
-* Pipeline code should be placed under a new `pipelines` folder at the root of
-the project
-* Exclude the `pipelines` folder as a trigger to the "PetClinic" pipeline
-* Create a new "Meta" pipeline triggered on changes to the `pipelines` folder
-only, that executes the python scripts using GoMatic
-* Use GoMatic locally to export the current pipeline configurations and convert
-them into pipeline scripts
+* Pipeline configuration should be placed under a new `PetClinic.gocd.yaml` file
+at the root of the project
+* Exclude the `PetClinic.gocd.yaml` file as a trigger to the "PetClinic" pipeline
+* Setup a new Configuration Repo using the YAML Config Plugin
+* Export the current pipeline configuration to YAML and move to the repository
+* Extract the common environment variables and secret variables to a new `gcp`
+GoCD environment
+* Display test results and artifacts on job "Tests" tab
 
 ## Step by Step Instructions
 
-### Creating placeholder scripts
+### Extracting and configuring the PetClinic Pipeline as Code
 
-First, let's create the folder and a few dummy scripts for each of the
-pipelines, to serve as placeholder test scripts:
+First, let's export the current configuration to the YAML configuration format.
+Click on the "ADMIN" menu and select "Pipelines". Click on "Export using" for
+"PetClinc" pipeline and select the "YAML Configuration Plugin" option.
+tab. This will download a file called `PetClinic.gocd.yaml`, that you can move
+to the root of the project.
 
-```shell
-$ mkdir pipelines
-$ echo "print \"Updating Meta Pipeline...\"" > pipelines/meta_pipeline.py
-$ echo "print \"Updating PetClinic Pipeline...\"" > pipelines/pet_clinic_pipeline.py
-```
-
-Let's also create a `pipelines/update.sh` script that will be executed by the
-meta pipeline to invoke the dummy scripts inside a Docker container with
-Python and GoMatic:
-
-```bash
-#!/bin/bash
-set -xe
-
-CWD=$(cd $(dirname $0) && pwd)
-for pipeline in $CWD/*.py; do
-  docker run -i --rm -v "$CWD":/usr/src/meta -w /usr/src/meta -e GO_SERVER_URL=$GO_SERVER_URL dtsato/gomatic /bin/bash -c "python $(basename $pipeline)"
-done
-```
-
-Make sure the new script is executable:
-
-```shell
-$ chmod a+x pipelines/update.sh
-```
-
-Now let's update the current "PetClinic" pipeline configuration to not trigger
-when changes occur to the `pipelines` folder. Click on the "ADMIN" menu and select
-"Pipelines". Click on "Edit" for "PetClinc" pipeline and go to the "Materials"
-tab. Opening the Git material, add the following configuration to the Blacklist:
-
-* Paths to be excluded: `pipelines/*`
-
-Now we can commit and push our changes to introduce the placeholder scripts and
-it should not trigger a pipeline execution.
+Fix some of the formatting on the file to make sure all the tasks are defined in
+a single line. Also update the `format_version` to version `4`.
 
-### Creating the meta pipeline
+In order to use this file, we need to create a new Configuration Repository,
+that will poll for changes and update the pipeline definitions when the YAML
+file changes. Click on the "ADMIN" menu and select "Config Repositories". Click
+the "Add" button and create the new configuration repository:
 
-From the "ADMIN" menu, select "Pipelines", and click on "Create a new pipeline
-within this group". The first step will setup the pipeline name to "Meta" and
-click "NEXT". Then on the next step, use the following configuration:
-
-* Material Type: `Git`
+* Plugin ID: `YAML Configuration Plugin`
+* Material type: `Git`
+* Config repository ID: `sample`
 * URL: Same as before, use the Git URL for your repository
 * Branch: `master`
-* Blacklist: `pipelines/*`
-* Invert the file filter: Checked
-
-Once again, you can test by clicking on "CHECK CONNECTION" before proceeding
-with clicking "NEXT". In the final step, use the following configuration:
-
-* Stage Name: `update-pipelines`
-* Job Name: `update-pipelines`
-* Task Type: `More...`
-* Command: `pipelines/update.sh`
 
-We can then click on "FINISH". Going back to the "Job Settings" tab, we can add
-the `docker-jdk` Elastic Profile Id to ensure the agent will run. We can now
-un-pause the pipeline and test that it runs successfully.
+Once again, you can test by clicking on "Test Connection" before proceeding
+with clicking "Save".
 
-### Pipeline as code for Meta pipeline
+Now we can commit and push our changes to introduce the `PetClinic.gocd.yaml`
+pipeline configuration file and it should be picked up by the YAML configuration
+plugin.
 
-Once the build succeeds, let's update the dummy script to setup the pipeline
-based on its current state. GoMatic has a feature to export the current pipeline
-configuration as code, so we can run this command locally, replacing the IP
-address with your GoCD Server public IP from GKE:
+When it first executes, you will see an error because the `PetClinic` pipeline
+already exists. In order for GoCD to use the YAML definition, we need to delete
+the existing pipeline. Don't worry, once the YAML plugin re-scans the repo, it
+will recreate our pipeline. Click on the "ADMIN" menu and select "Pipelines".
+Click the "Delete" link and on the "Proceed" button to confirm the pipeline
+deletion.
 
-```shell
-$ docker run -i --rm dtsato/gomatic python -m gomatic.go_cd_configurator -s 35.190.56.218 -p Meta
-#!/usr/bin/env python
-from gomatic import *
-...
-```
+Now wait for the YAML Config Plugin to re-scan the repository and recreate our
+pipeline.
 
-You can see that at the end of the execution, there is some Python code that we
-can copy and paste and use as the template for our pipeline configuration as code.
-Paste it on the `pipelines/meta_pipeline.py` script and make a few tweaks:
-
-```python
-#!/usr/bin/env python
-from gomatic import *
-import os, re
-
-print "Updating Meta Pipeline..."
-
-go_server_host = re.search('https?://([a-z0-9.\-._~%]+)', os.environ['GO_SERVER_URL']).group(1)
-go_server_url = "%s:%s" % (go_server_host, "8153")
-configurator = GoCdConfigurator(HostRestClient(go_server_url))
-pipeline = configurator\
-	.ensure_pipeline_group("sample")\
-	.ensure_replacement_of_pipeline("Meta")\
-	.set_git_material(GitMaterial("https://github.com/dtsato/devops-in-practice-workshop.git", branch="master", ignore_patterns=set(['pipelines/*']), invert_filter="True"))
-stage = pipeline.ensure_stage("update-pipelines")
-job = stage.ensure_job("update-pipelines").set_elastic_profile_id('docker-jdk')
-job.add_task(ExecTask(['pipelines/update.sh']))
-
-configurator.save_updated_config()
-```
+### Improving the PetClinic pipeline
 
-We are adding some code to parse the GoCD Server URL from the environment
-variable and removed the named arguments from the last line to actually save the
-configuration, and not just do a dry-run.
+Now, to test that the pipeline configuration is really defined in code, on the
+`PetClinic.gocd.yaml` file, let's make a few changes to improve it. First, let's
+configure our `build-and-publish` job, to collect and publish the test report
+artifacts from surefire, by adding the `artifacts` section:
 
-Once you commit and push this code, the Meta pipeline should trigger again, and
-this time the above code will invoke GoMatic.
-
-### Pipeline as code for PetClinic pipeline
-
-Finally, let's extract the pipeline configuration for the "PetClinic" pipeline,
-by executing the same GoMatic command locally, changing the pipeline name and
-using your GoCD Server URL:
-
-```shell
-$ docker run -i --rm dtsato/gomatic python -m gomatic.go_cd_configurator -s 35.190.56.218 -p PetClinic
-#!/usr/bin/env python
-from gomatic import *
+```yaml
+...
+jobs:
+  build-and-publish:
+    timeout: 0
+    environment_variables:
+      MAVEN_OPTS: -Xmx1024m
+      GCLOUD_PROJECT_ID: devops-workshop-123
+    secure_variables:
+      GCLOUD_SERVICE_KEY: AES:kb7KQ/gJ1VTtYGU6SLUJjA==...
+    elastic_profile_id: docker-jdk
+    artifacts:
+      - test:
+          source: target/surefire-reports
+    tasks:
 ...
 ```
 
-At the end of the execution, you can once again copy and paste the Python code
-into the `pipelines/pet_clinic_pipeline.py` script, and make the same tweaks:
-
-```python
-#!/usr/bin/env python
-from gomatic import *
-import os, re
-
-print "Updating PetClinic Pipeline..."
-go_server_host = re.search('https?://([a-z0-9.\-._~%]+)', os.environ['GO_SERVER_URL']).group(1)
-go_server_url = "%s:%s" % (go_server_host, "8153")
-configurator = GoCdConfigurator(HostRestClient(go_server_url))
-secret_variables = {'GCLOUD_SERVICE_KEY': 'lKD+DoKDGtCsaToW...'}
-pipeline = configurator\
-	.ensure_pipeline_group("sample")\
-	.ensure_replacement_of_pipeline("PetClinic")\
-	.set_git_material(GitMaterial("https://github.com/dtsato/devops-in-practice-workshop.git", branch="master", ignore_patterns=set(['pipelines/*'])))
-stage = pipeline.ensure_stage("commit")
-job = stage\
-    .ensure_job("build-and-publish")\
-    .ensure_environment_variables({'MAVEN_OPTS': '-Xmx1024m', 'GCLOUD_PROJECT_ID': 'devops-workshop-123'})\
-    .ensure_encrypted_environment_variables(secret_variables)
-job.set_elastic_profile_id('docker-jdk')
-job.add_task(ExecTask(['./mvnw', 'clean', 'package']))
-job.add_task(ExecTask(['bash', '-c', 'docker build --tag pet-app:$GO_PIPELINE_LABEL --build-arg JAR_FILE=target/spring-petclinic-2.0.0.BUILD-SNAPSHOT.jar .']))
-job.add_task(ExecTask(['bash', '-c', 'docker login -u _json_key -p"$(echo $GCLOUD_SERVICE_KEY | base64 -d)" https://us.gcr.io']))
-job.add_task(ExecTask(['bash', '-c', 'docker tag pet-app:$GO_PIPELINE_LABEL us.gcr.io/$GCLOUD_PROJECT_ID/pet-app:$GO_PIPELINE_LABEL']))
-job.add_task(ExecTask(['bash', '-c', 'docker push us.gcr.io/$GCLOUD_PROJECT_ID/pet-app:$GO_PIPELINE_LABEL']))
-stage = pipeline.ensure_stage("deploy")
-job = stage\
-    .ensure_job("deploy")\
-    .ensure_environment_variables({'GCLOUD_ZONE': 'us-central1-a', 'GCLOUD_PROJECT_ID': 'devops-workshop-123', 'GCLOUD_CLUSTER': 'devops-workshop-gke'})\
-    .ensure_encrypted_environment_variables(secret_variables)
-job.set_elastic_profile_id('kubectl')
-job.add_task(ExecTask(['bash', '-c', 'echo $GCLOUD_SERVICE_KEY | base64 -d > secret.json && chmod 600 secret.json']))
-job.add_task(ExecTask(['bash', '-c', 'gcloud auth activate-service-account --key-file secret.json']))
-job.add_task(ExecTask(['bash', '-c', 'gcloud container clusters get-credentials $GCLOUD_CLUSTER --zone $GCLOUD_ZONE --project $GCLOUD_PROJECT_ID']))
-job.add_task(ExecTask(['bash', '-c', './deploy.sh']))
-job.add_task(ExecTask(['bash', '-c', 'rm secret.json']))
-
-configurator.save_updated_config()
+Also, let's extract the common environment variables and secret variables
+configuration to a new [GoCD Environment](https://docs.gocd.org/current/navigation/environments_page.html)
+by adding the following section to the beginning of the `PetClinic.gocd.yaml`
+file, copying the values of the variables from one of the stages/jobs
+(make sure the `GCLOUD_PROJECT_ID` and `GCLOUD_SERVICE_KEY` are yours):
+
+```yaml
+format_version: 4
+environments:
+  gcp:
+    environment_variables:
+      GCLOUD_CLUSTER: devops-workshop-gke
+      GCLOUD_ZONE: us-central1-a
+      GCLOUD_PROJECT_ID: devops-workshop-123
+    secure_variables:
+      GCLOUD_SERVICE_KEY: AES:kb7KQ/gJ1VTtYGU6SLUJjA==...
+    pipelines:
+      - PetClinic
+...
 ```
 
-Please note that we once again added some code to parse the GoCD Server URL, set
-the Elastic Profile IDs for both jobs, and removed the arguments from the last
-line. We also added an extra line to the `build-and-publish` job to collect the
-test report artifacts from surefire. This allows us to test that the pipeline
-configuration is getting updated after the Meta pipeline executes.
+Also, make sure to remove those common environment variables and secret
+variables from the stages and jobs to avoid the duplication.
 
-Once again, when you commit and push these changes, the "Meta" pipeline should
-trigger and reconfigure the PetClinic pipeline.
+Once again, when you commit and push these changes, the YAML configuration
+plugin should pick up the changes and trigger a new pipeline execution, where
+you can check that the test results are now available after a successful
+pipeline run.
diff --git a/instructions/14-canary-release.md b/instructions/14-canary-release.md
@@ -10,7 +10,7 @@
 * Create a `web-canary.yml` kubernetes definition to implement canary releases
 * Update the `deploy.sh` script to deploy the canary release only
 * Create a `complete-canary.sh` script to complete the canary release rollout
-* Extend the PetClinic pipeline (using GoMatic) to add a new stage with a manual
+* Extend the PetClinic pipeline to add a new stage with a manual
 approval to complete the canary release using the above scripts
 
 ## Step by Step Instructions
@@ -129,32 +129,54 @@ Don't forget to make the script executable:
 $ chmod a+x complete-canary.sh
 ```
 
-Finally, we can update our `pipelines/pet_clinic_pipeline.py` script to add a
-new manual stage and job to execute the `complete_canary.sh` script, after the
-definition of the `deploy` stage:
+Finally, we can update our `PetClinic.gocd.yaml` pipeline configuration file to
+add a new manual stage and job to execute the `complete_canary.sh` script, after
+the definition of the `deploy` stage:
 
-```python
+```yaml
 ...
 
-stage = pipeline.ensure_stage("approve-canary")
-stage.set_has_manual_approval()
-job = stage\
-	.ensure_job("complete-canary")\
-    .ensure_environment_variables({'GCLOUD_ZONE': 'us-central1-a', 'GCLOUD_PROJECT_ID': 'devops-workshop-123', 'GCLOUD_CLUSTER': 'devops-workshop-gke'})\
-    .ensure_encrypted_environment_variables(secret_variables)
-job.set_elastic_profile_id('kubectl')
-job.add_task(ExecTask(['bash', '-c', 'echo $GCLOUD_SERVICE_KEY | base64 -d > secret.json && chmod 600 secret.json']))
-job.add_task(ExecTask(['bash', '-c', 'gcloud auth activate-service-account --key-file secret.json']))
-job.add_task(ExecTask(['bash', '-c', 'gcloud container clusters get-credentials $GCLOUD_CLUSTER --zone $GCLOUD_ZONE --project $GCLOUD_PROJECT_ID']))
-job.add_task(ExecTask(['bash', '-c', './complete-canary.sh']))
-job.add_task(ExecTask(['bash', '-c', 'rm secret.json']))
-
-configurator.save_updated_config()
+- approve-canary:
+    fetch_materials: true
+    keep_artifacts: false
+    clean_workspace: false
+    approval:
+      type: manual
+    jobs:
+      complete-canary:
+        timeout: 0
+        elastic_profile_id: kubectl
+        tasks:
+        - exec:
+            arguments:
+            - -c
+            - echo $GCLOUD_SERVICE_KEY | base64 -d > secret.json && chmod 600 secret.json
+            command: bash
+            run_if: passed
+        - exec:
+            arguments:
+            - -c
+            - gcloud auth activate-service-account --key-file secret.json
+            command: bash
+            run_if: passed
+        - exec:
+            arguments:
+            - -c
+            - gcloud container clusters get-credentials $GCLOUD_CLUSTER --zone $GCLOUD_ZONE --project $GCLOUD_PROJECT_ID
+            command: bash
+            run_if: passed
+        - exec:
+            command: ./complete-canary.sh
+            run_if: passed
+        - exec:
+            arguments:
+            - -c
+            - rm secret.json
+            command: bash
+            run_if: passed
 ```
 
-Commit and push the changes to the pipeline definition and wait until GoCD is
-updated. Once the pipeline is updated with the new stage, go ahead and commit and
-push the other remaining changes to the kubernetes files and deployment scripts.
-This should trigger the "PetClinic" pipeline and you should see it deploy the
-new version as a canary release. Then you can test a manual approval to complete
-the release.
+Commit and push the changes and wait until GoCD is updated. Once the pipeline is
+updated with the new stage, it will trigger a new execution.You should see it
+deploy the new version as a canary release. Then you can test a manual approval
+to complete the release.
