Erisch/features/devops/integration tests #75

erikschlegel · 2019-04-23T20:51:24Z

Test Automation Harness:

[YES] Have you followed the guidelines in our Contributing document?
[YES] Have you added an explanation of what your changes do and why you'd like us to include them

Description
This pull request adds a new automated test harness to account for unit and integration tests. We recommend cobalt contributors provide automated test artifacts in the test folder for each terraform template. We're using mage to orchestrate the test run. This harness supports integration test classes where thew class name starts with TestIT and unit tests that start with TestUT.

The test runner will automatically run integration and unit tests based on the git change log of the current working branch. We took a naive approach with re-running all template tests if TF module artifacts are modified. We'll follow up a with a PR to build a module reference <> template dependency graph to follow a more robust and sophisticated approach.

We also put a focus on developer experience with simplifying local automated test runs with including a dockerized test runner. The dockerized test harness will be migrated to azure devops once the base image is multi staged.

Feature List

Integrating the test harness with our Azure DevOPS CI pipeline.
A dockerized test harness that can be invoked via ./test-harness/local-run.sh
Excluding unmodified terraform templates from test run
An environment template file .env.template
dep manifest for terratest dependencies.
Base docker image
An nginx container hello world app service terraform template azure-simple-hw
A backend template for setting up remote terraform state resources in azure. This template references the backend-state module from bedrock.
An golang based integration test for the hello world template. This test asserts that an http connection can be established to the app service resource.
Removed unused files.
A shell script utility to automate the base image build step.
The magefile.go test orchestrator.
Added a lint format step to our test runner.
Added key vault integration with our azure devops pipeline.

[NO] Does this include new/updated documentation?
Details
To avoid tacking more changes onto this mounting PR I opted to submit a follow on PR that focuses on documentation. The user story will be closed once both PR's are merged.

[YES] I have added tests to cover my changes.
[YES] All new and existing tests passed.
[YES] My code follows the code style of this project.
[YES] I ran lint checks locally prior to submission.
[YES] Have you checked to ensure there aren't other open Pull Requests for the same update/change?

Issue Number: #56

Does this introduce a breaking change?

NO

Any relevant logs, error output, etc?

Logs from passing CI build from azure DevOPS

.env.template

nmiodice · 2019-04-23T21:41:32Z

Gopkg.toml

@@ -0,0 +1,42 @@
+# Gopkg.toml example


Are these intended to act as an actual example or can they be removed?

I'm still a golang noob and not sure how much control we have on these files as they were auto generated.

I think the commented parts are eligible for deletion

https://golang.github.io/dep/docs/Gopkg.toml.html

I think they are eligible for deletion

https://golang.github.io/dep/docs/Gopkg.toml.html

devops/providers/azure-devops/templates/azure-pipelines.yml

go.mod

test-harness/Dockerfile

test-harness/init.sh

test-harness/local-run.sh

infra/templates/azure-simple-hw/test/azure_simple_integration_test.go

nmiodice · 2019-04-25T01:30:22Z

magefile.go

+}
+
+// A build step that formats both Terraform code and Go code
+func FormatGolangFiles() error {


[question for the future] Should this be caught by linting and cause a build to fail? I think you may be able to run something like this to check if the terraform is correctly formatted:

# https://www.terraform.io/docs/commands/fmt.html terraform fmt -check=true

Agree that we should create a user story to support a proper TF Lint target within our test harness.

# This is the 1st commit message: Adding docker test base image # This is the commit message #2: Adding test files # This is the commit message #3: testing # This is the commit message #4: Added folder # This is the commit message #5: Added folder # This is the commit message #6: Azure Devops CI Build Pipeline (#67) * Adding azure devops build pipeline * Adding README * Adding azure build badge # This is the commit message #7: saving latest tests # This is the commit message #8: Adding docker test base image (#68) # This is the commit message #9: EOD checkin # This is the commit message #10: posting latest # This is the commit message #11: eod commits # This is the commit message #12: eod commit # This is the commit message #13: refactored test directory structure # This is the commit message #14: Refactored build # This is the commit message #15: Adding hello world azure simple app # This is the commit message #16: Adding hello world azure simple app # This is the commit message #17: refactoring test harness # This is the commit message #18: refactoring test harness # This is the commit message #19: checking latest # This is the commit message #20: committing latest # This is the commit message #21: checking in latest changes # This is the commit message #22: Checking in latest progress # This is the commit message #23: applying final touches before PR submission # This is the commit message #24: updating azure keyvault secret names # This is the commit message #25: Checking in latest progress # This is the commit message #26: pushing latest pipeline definition # This is the commit message #27: updating build pipeline variable format # This is the commit message #28: updating build pipeline variable format # This is the commit message #29: cleaning up build variables # This is the commit message #30: adding minor changes to azure build pipeline # This is the commit message #31: updating azure keyvault secret names # This is the commit message #32: updating azure keyvault secret names # This is the commit message #33: fixing docker login azdo step # This is the commit message #34: fixing docker login azdo step # This is the commit message #35: fixing docker login azdo step # This is the commit message #36: fixing docker login azdo step # This is the commit message #37: updating build pipeline variables # This is the commit message #38: updating build pipeline variables # This is the commit message #39: updating build pipeline variables # This is the commit message #40: updating build pipeline variables # This is the commit message #41: updating build pipeline variables # This is the commit message #42: updating build pipeline variables # This is the commit message #43: applying fix to git diff script # This is the commit message #44: applying fix to git diff script # This is the commit message #45: applying fix to git diff script # This is the commit message #46: applying fix to git diff script # This is the commit message #47: applying fix to git diff script # This is the commit message #48: resolving pipeline bug # This is the commit message #49: fixing build error # This is the commit message #50: fixing build error # This is the commit message #51: fixing build error # This is the commit message #52: fixing build error # This is the commit message #53: fixing build error # This is the commit message #54: fixing build error # This is the commit message #55: fixing build error # This is the commit message #56: fixing build error # This is the commit message #57: fixing build error # This is the commit message #58: fixing build error # This is the commit message #59: fixing build error # This is the commit message #60: fixing build error # This is the commit message #61: fixing build error # This is the commit message #62: fixing build error # This is the commit message #63: fixing build error # This is the commit message #64: fixing build error # This is the commit message #65: fixing build error # This is the commit message #66: fixing build error # This is the commit message #67: fixing build error # This is the commit message #68: fixing build error # This is the commit message #69: fixing build error # This is the commit message #70: fixing build error # This is the commit message #71: fixing build error # This is the commit message #72: fixing build error # This is the commit message #73: trying acr fix # This is the commit message #74: latest changes # This is the commit message #75: removing docker dependency on azure devops pipeline # This is the commit message #76: removing docker dependency on azure devops pipeline # This is the commit message #77: removing docker dependency on azure devops pipeline # This is the commit message #78: removing docker dependency on azure devops pipeline # This is the commit message #79: removing docker dependency on azure devops pipeline # This is the commit message #80: removing docker dependency on azure devops pipeline # This is the commit message #81: removing docker dependency on azure devops pipeline # This is the commit message #82: removing docker dependency on azure devops pipeline # This is the commit message #83: removing docker dependency on azure devops pipeline # This is the commit message #84: removing docker dependency on azure devops pipeline # This is the commit message #85: removing docker dependency on azure devops pipeline # This is the commit message #86: removing docker dependency on azure devops pipeline # This is the commit message #87: removing docker dependency on azure devops pipeline # This is the commit message #88: removing docker dependency on azure devops pipeline # This is the commit message #89: removing docker dependency on azure devops pipeline # This is the commit message #90: removing docker dependency on azure devops pipeline # This is the commit message #91: removing docker dependency on azure devops pipeline # This is the commit message #92: removing docker dependency on azure devops pipeline # This is the commit message #93: removing docker dependency on azure devops pipeline # This is the commit message #94: removing docker dependency on azure devops pipeline # This is the commit message #95: removing docker dependency on azure devops pipeline # This is the commit message #96: removing docker dependency on azure devops pipeline # This is the commit message #97: removing docker dependency on azure devops pipeline # This is the commit message #98: final commit including resource name randomization # This is the commit message #99: Add provider to configure az infra for Terraform

Updated documentation based on review comments Integrate review comments for documentation. Add usage example for each of the providers Adding environment template descriptions removing go version var from azdo pipeline as it's unused removing unused setup folder adding more verbose logging for azure devops failure fixing pull request branch name resolution bug fixing pull request branch name resolution bug addressing nick's pr comments documenting script and docker images cleaning up required env var check fixing base branch bug Added service plan terraform templates to Azure Adding tags for resources Integrate review comments to add random id to name Add tags, capacity and reserved for service plan Minor Fixes for documentation Integrate review comments related to resource naming Remove Optional and Required in variables.tf Update variables.tf Update Service Plan Tier to Isolated squaching commits applied base image suggestions from nick

…test runner

.env.template

nmiodice · 2019-04-25T19:52:23Z

Gopkg.toml

@@ -0,0 +1,42 @@
+# Gopkg.toml example


I think the commented parts are eligible for deletion

https://golang.github.io/dep/docs/Gopkg.toml.html

nmiodice · 2019-04-25T19:52:37Z

Gopkg.toml

@@ -0,0 +1,42 @@
+# Gopkg.toml example


I think they are eligible for deletion

https://golang.github.io/dep/docs/Gopkg.toml.html

devops/providers/azure-devops/templates/azure-pipelines.yml

infra/templates/azure-simple-hw/test/azure_simple_integration_test.go

infra/templates/backend-state-setup/terraform.tfvars

nmiodice

Approved with some comments. Feel free to address as you see fit and merge.

NathanielRose · 2019-04-26T18:52:29Z

devops/providers/azure-devops/templates/azure-pipelines.yml

+- script: |
+    docker login $ACR_HOST -u $ACR_USERNAME -p $(DOCKER-PASSWORD)
+
+- script: |


So checked with AzDO there isnt a way to trigger tasks from repo path and this is a good approach to acheive this. Another approach would have been to have multiple build jobs based on repo file path that would execute different parts of the test suite. Only concern here is it may get tricky to get the logs out running everything inside docker.

* # This is a combination of 99 commits. # This is the 1st commit message: Adding docker test base image # This is the commit message #2: Adding test files # This is the commit message #3: testing # This is the commit message #4: Added folder # This is the commit message #5: Added folder # This is the commit message #6: Azure Devops CI Build Pipeline (#67) * Adding azure devops build pipeline * Adding README * Adding azure build badge # This is the commit message #7: saving latest tests # This is the commit message #8: Adding docker test base image (#68) # This is the commit message #9: EOD checkin # This is the commit message #10: posting latest # This is the commit message #11: eod commits # This is the commit message #12: eod commit # This is the commit message #13: refactored test directory structure # This is the commit message #14: Refactored build # This is the commit message #15: Adding hello world azure simple app # This is the commit message #16: Adding hello world azure simple app # This is the commit message #17: refactoring test harness # This is the commit message #18: refactoring test harness # This is the commit message #19: checking latest # This is the commit message #20: committing latest # This is the commit message #21: checking in latest changes # This is the commit message #22: Checking in latest progress # This is the commit message #23: applying final touches before PR submission # This is the commit message #24: updating azure keyvault secret names # This is the commit message #25: Checking in latest progress # This is the commit message #26: pushing latest pipeline definition # This is the commit message #27: updating build pipeline variable format # This is the commit message #28: updating build pipeline variable format # This is the commit message #29: cleaning up build variables # This is the commit message #30: adding minor changes to azure build pipeline # This is the commit message #31: updating azure keyvault secret names # This is the commit message #32: updating azure keyvault secret names # This is the commit message #33: fixing docker login azdo step # This is the commit message #34: fixing docker login azdo step # This is the commit message #35: fixing docker login azdo step # This is the commit message #36: fixing docker login azdo step # This is the commit message #37: updating build pipeline variables # This is the commit message #38: updating build pipeline variables # This is the commit message #39: updating build pipeline variables # This is the commit message #40: updating build pipeline variables # This is the commit message #41: updating build pipeline variables # This is the commit message #42: updating build pipeline variables # This is the commit message #43: applying fix to git diff script # This is the commit message #44: applying fix to git diff script # This is the commit message #45: applying fix to git diff script # This is the commit message #46: applying fix to git diff script # This is the commit message #47: applying fix to git diff script # This is the commit message #48: resolving pipeline bug # This is the commit message #49: fixing build error # This is the commit message #50: fixing build error # This is the commit message #51: fixing build error # This is the commit message #52: fixing build error # This is the commit message #53: fixing build error # This is the commit message #54: fixing build error # This is the commit message #55: fixing build error # This is the commit message #56: fixing build error # This is the commit message #57: fixing build error # This is the commit message #58: fixing build error # This is the commit message #59: fixing build error # This is the commit message #60: fixing build error # This is the commit message #61: fixing build error # This is the commit message #62: fixing build error # This is the commit message #63: fixing build error # This is the commit message #64: fixing build error # This is the commit message #65: fixing build error # This is the commit message #66: fixing build error # This is the commit message #67: fixing build error # This is the commit message #68: fixing build error # This is the commit message #69: fixing build error # This is the commit message #70: fixing build error # This is the commit message #71: fixing build error # This is the commit message #72: fixing build error # This is the commit message #73: trying acr fix # This is the commit message #74: latest changes # This is the commit message #75: removing docker dependency on azure devops pipeline # This is the commit message #76: removing docker dependency on azure devops pipeline # This is the commit message #77: removing docker dependency on azure devops pipeline # This is the commit message #78: removing docker dependency on azure devops pipeline # This is the commit message #79: removing docker dependency on azure devops pipeline # This is the commit message #80: removing docker dependency on azure devops pipeline # This is the commit message #81: removing docker dependency on azure devops pipeline # This is the commit message #82: removing docker dependency on azure devops pipeline # This is the commit message #83: removing docker dependency on azure devops pipeline # This is the commit message #84: removing docker dependency on azure devops pipeline # This is the commit message #85: removing docker dependency on azure devops pipeline # This is the commit message #86: removing docker dependency on azure devops pipeline # This is the commit message #87: removing docker dependency on azure devops pipeline # This is the commit message #88: removing docker dependency on azure devops pipeline # This is the commit message #89: removing docker dependency on azure devops pipeline # This is the commit message #90: removing docker dependency on azure devops pipeline # This is the commit message #91: removing docker dependency on azure devops pipeline # This is the commit message #92: removing docker dependency on azure devops pipeline # This is the commit message #93: removing docker dependency on azure devops pipeline # This is the commit message #94: removing docker dependency on azure devops pipeline # This is the commit message #95: removing docker dependency on azure devops pipeline # This is the commit message #96: removing docker dependency on azure devops pipeline # This is the commit message #97: removing docker dependency on azure devops pipeline # This is the commit message #98: final commit including resource name randomization # This is the commit message #99: Add provider to configure az infra for Terraform * resolving merge conflicts for squash Updated documentation based on review comments Integrate review comments for documentation. Add usage example for each of the providers Adding environment template descriptions removing go version var from azdo pipeline as it's unused removing unused setup folder adding more verbose logging for azure devops failure fixing pull request branch name resolution bug fixing pull request branch name resolution bug addressing nick's pr comments documenting script and docker images cleaning up required env var check fixing base branch bug Added service plan terraform templates to Azure Adding tags for resources Integrate review comments to add random id to name Add tags, capacity and reserved for service plan Minor Fixes for documentation Integrate review comments related to resource naming Remove Optional and Required in variables.tf Update variables.tf Update Service Plan Tier to Isolated squaching commits applied base image suggestions from nick * squashing commits and slimming down the build image * removing go* path from azure devops pipeline as we removed go mods * refactoring our azure devops pipeline so it's aligned with the local test runner * fixing azure devops pipeline bug * fixing base image build variable reference bug * Addressing nick's PR comments * resolving issue with failed builds after completing branches merges * addressing a bug where test harness directory changes were not getting picked up by the harness * removing local echo statement * Adding directory exists check and app service container command variable * addressing nick's PR comments * adding git diff extension whitelist filter

* # This is a combination of 99 commits. # This is the 1st commit message: Adding docker test base image # This is the commit message #2: Adding test files # This is the commit message #3: testing # This is the commit message #4: Added folder # This is the commit message #5: Added folder # This is the commit message #6: Azure Devops CI Build Pipeline (#67) * Adding azure devops build pipeline * Adding README * Adding azure build badge # This is the commit message #7: saving latest tests # This is the commit message #8: Adding docker test base image (#68) # This is the commit message #9: EOD checkin # This is the commit message #10: posting latest # This is the commit message #11: eod commits # This is the commit message #12: eod commit # This is the commit message #13: refactored test directory structure # This is the commit message #14: Refactored build # This is the commit message #15: Adding hello world azure simple app # This is the commit message #16: Adding hello world azure simple app # This is the commit message #17: refactoring test harness # This is the commit message #18: refactoring test harness # This is the commit message #19: checking latest # This is the commit message #20: committing latest # This is the commit message #21: checking in latest changes # This is the commit message #22: Checking in latest progress # This is the commit message #23: applying final touches before PR submission # This is the commit message #24: updating azure keyvault secret names # This is the commit message #25: Checking in latest progress # This is the commit message #26: pushing latest pipeline definition # This is the commit message #27: updating build pipeline variable format # This is the commit message #28: updating build pipeline variable format # This is the commit message #29: cleaning up build variables # This is the commit message #30: adding minor changes to azure build pipeline # This is the commit message #31: updating azure keyvault secret names # This is the commit message #32: updating azure keyvault secret names # This is the commit message #33: fixing docker login azdo step # This is the commit message #34: fixing docker login azdo step # This is the commit message #35: fixing docker login azdo step # This is the commit message #36: fixing docker login azdo step # This is the commit message #37: updating build pipeline variables # This is the commit message #38: updating build pipeline variables # This is the commit message #39: updating build pipeline variables # This is the commit message #40: updating build pipeline variables # This is the commit message #41: updating build pipeline variables # This is the commit message #42: updating build pipeline variables # This is the commit message #43: applying fix to git diff script # This is the commit message #44: applying fix to git diff script # This is the commit message #45: applying fix to git diff script # This is the commit message #46: applying fix to git diff script # This is the commit message #47: applying fix to git diff script # This is the commit message #48: resolving pipeline bug # This is the commit message #49: fixing build error # This is the commit message #50: fixing build error # This is the commit message #51: fixing build error # This is the commit message #52: fixing build error # This is the commit message #53: fixing build error # This is the commit message #54: fixing build error # This is the commit message #55: fixing build error # This is the commit message #56: fixing build error # This is the commit message #57: fixing build error # This is the commit message #58: fixing build error # This is the commit message #59: fixing build error # This is the commit message #60: fixing build error # This is the commit message #61: fixing build error # This is the commit message #62: fixing build error # This is the commit message #63: fixing build error # This is the commit message #64: fixing build error # This is the commit message #65: fixing build error # This is the commit message #66: fixing build error # This is the commit message #67: fixing build error # This is the commit message #68: fixing build error # This is the commit message #69: fixing build error # This is the commit message #70: fixing build error # This is the commit message #71: fixing build error # This is the commit message #72: fixing build error # This is the commit message #73: trying acr fix # This is the commit message #74: latest changes # This is the commit message #75: removing docker dependency on azure devops pipeline # This is the commit message #76: removing docker dependency on azure devops pipeline # This is the commit message #77: removing docker dependency on azure devops pipeline # This is the commit message #78: removing docker dependency on azure devops pipeline # This is the commit message #79: removing docker dependency on azure devops pipeline # This is the commit message #80: removing docker dependency on azure devops pipeline # This is the commit message #81: removing docker dependency on azure devops pipeline # This is the commit message #82: removing docker dependency on azure devops pipeline # This is the commit message #83: removing docker dependency on azure devops pipeline # This is the commit message #84: removing docker dependency on azure devops pipeline # This is the commit message #85: removing docker dependency on azure devops pipeline # This is the commit message #86: removing docker dependency on azure devops pipeline # This is the commit message #87: removing docker dependency on azure devops pipeline # This is the commit message #88: removing docker dependency on azure devops pipeline # This is the commit message #89: removing docker dependency on azure devops pipeline # This is the commit message #90: removing docker dependency on azure devops pipeline # This is the commit message #91: removing docker dependency on azure devops pipeline # This is the commit message #92: removing docker dependency on azure devops pipeline # This is the commit message #93: removing docker dependency on azure devops pipeline # This is the commit message #94: removing docker dependency on azure devops pipeline # This is the commit message #95: removing docker dependency on azure devops pipeline # This is the commit message #96: removing docker dependency on azure devops pipeline # This is the commit message #97: removing docker dependency on azure devops pipeline # This is the commit message #98: final commit including resource name randomization # This is the commit message #99: Add provider to configure az infra for Terraform * resolving merge conflicts for squash Updated documentation based on review comments Integrate review comments for documentation. Add usage example for each of the providers Adding environment template descriptions removing go version var from azdo pipeline as it's unused removing unused setup folder adding more verbose logging for azure devops failure fixing pull request branch name resolution bug fixing pull request branch name resolution bug addressing nick's pr comments documenting script and docker images cleaning up required env var check fixing base branch bug Added service plan terraform templates to Azure Adding tags for resources Integrate review comments to add random id to name Add tags, capacity and reserved for service plan Minor Fixes for documentation Integrate review comments related to resource naming Remove Optional and Required in variables.tf Update variables.tf Update Service Plan Tier to Isolated squaching commits applied base image suggestions from nick * squashing commits and slimming down the build image * removing go* path from azure devops pipeline as we removed go mods * refactoring our azure devops pipeline so it's aligned with the local test runner * fixing azure devops pipeline bug * fixing base image build variable reference bug * Addressing nick's PR comments * resolving issue with failed builds after completing branches merges * addressing a bug where test harness directory changes were not getting picked up by the harness * removing local echo statement * Adding directory exists check and app service container command variable * addressing nick's PR comments * adding git diff extension whitelist filter * Overhauling main project readme * Adding james to the readme * Update README.md * Updating readme from the test-harness directory

erikschlegel requested a review from nmiodice April 23, 2019 20:51

erikschlegel assigned NathanielRose, code4clouds, James-Nance and manojvazirani Apr 23, 2019

nmiodice reviewed Apr 23, 2019

View reviewed changes

erikschlegel added this to the Sprint 1 milestone Apr 24, 2019

erikschlegel added the effort - large 3 day effort label Apr 24, 2019

nmiodice previously approved these changes Apr 25, 2019

View reviewed changes

erikschlegel added 4 commits April 24, 2019 21:01

squashing commits and slimming down the build image

31cff7a

squashing commits and slimming down the build image

ba30507

erikschlegel dismissed nmiodice’s stale review via ba30507 April 25, 2019 03:11

erikschlegel force-pushed the erisch/features/devops/integration-tests branch from 22995b2 to ba30507 Compare April 25, 2019 03:11

erikschlegel added 4 commits April 24, 2019 22:19

removing go* path from azure devops pipeline as we removed go mods

bfca01a

refactoring our azure devops pipeline so it's aligned with the local …

32639c8

…test runner

fixing azure devops pipeline bug

c5f610e

fixing base image build variable reference bug

adaf6aa

nmiodice reviewed Apr 25, 2019

View reviewed changes

nmiodice previously approved these changes Apr 25, 2019

View reviewed changes

Addressing nick's PR comments

6cb6fd3

erikschlegel dismissed nmiodice’s stale review via 6cb6fd3 April 25, 2019 21:16

erikschlegel merged commit 5070b66 into master Apr 25, 2019

NathanielRose reviewed Apr 26, 2019

View reviewed changes

erikschlegel deleted the erisch/features/devops/integration-tests branch April 27, 2019 23:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Erisch/features/devops/integration tests #75

Erisch/features/devops/integration tests #75

erikschlegel commented Apr 23, 2019 •

edited

Loading

nmiodice Apr 23, 2019

erikschlegel Apr 23, 2019

nmiodice Apr 25, 2019

nmiodice Apr 25, 2019

nmiodice Apr 25, 2019

erikschlegel Apr 25, 2019

nmiodice Apr 25, 2019

nmiodice Apr 25, 2019

nmiodice left a comment

NathanielRose Apr 26, 2019

Erisch/features/devops/integration tests #75

Erisch/features/devops/integration tests #75

Conversation

erikschlegel commented Apr 23, 2019 • edited Loading

Test Automation Harness:

Does this introduce a breaking change?

Any relevant logs, error output, etc?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmiodice left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikschlegel commented Apr 23, 2019 •

edited

Loading