WIP: "tspec" - Add a BDD-style interface to Terratest #730

robmorgan · 2020-12-08T15:00:00Z

This PR adds a BDD-style interface to Terratest which means you can write infrastructure tests in plain-English.

RFC

Terratest is great and really powerful, but since it's written in Go, it can sometimes be a significant learning curve for a lot of people to properly test their infrastructure. Imagine if you could write infrastructure tests in plain-English, but take advantage of all of the Terratest modules under the hood? This PR adds a BDD-style interface to Terratest which means you can write infrastructure tests without writing a single line of code.

Consider the Hello, World test for Terratest. It's 25+ lines for the simplest use case. Using tspec you could write a feature test as follows:

Feature: Terraform Hello World

  Scenario: Run a simple test
    Given the Terraform module at "./examples/terraform-hello-world-example"
    When I run "terraform apply"
    Then the "hello_world" output is "Hello, World!"

Or to test something slightly more complicated you could write a feature test along the lines of:

Feature: Terraform AWS Example

  Scenario: Run a simple test on AWS
    Given the Terraform module at "./examples/terraform-aws-example"
    And an input variable named "instance_name" with the value "test-instance-1"
    And an environment variable named "AWS_DEFAULT_REGION" with the value "us-east-1"
    When I run "terraform apply"
    Then the "instance_id" output should match "i-[0-9a-z]+(\,i-[0-9a-z]+)*"

When you execute tspec it will automatically copy the specified Terraform module to a temporary path, pass in the target parameters and ensure the tests pass as expected using Terratest methods under the hood.

PoC

Under the hood, tspec borrows heavily from the Godog Cucumber library but includes a library of common step definitions so users don't need to write any Go code.

Note: Unfortunately this is a rather large PR, so this RFC should serve as a potential discussion as to how we can "chop" this tool down into the smallest possible product increment.

TODO

Support for Background grammar.
Add more grammar for tools like Packer & Docker.

Open Questions

Should we remove all of the various formatters like junit, event etc from the first version?
What should we name this? tspec, infraspec?

… to a feature struct

brikis98

First, the BDD functionality is FANTASTIC. So cool to see this. I think it'll extend the ability to use Terratest and test infrastructure code to so many more people. Thanks for putting this together Rob 🍺

Second, I started to review this, but quickly ran into a lot of code that seemed to be copied from somewhere, or that I didn't fully understand, so I stopped. Could you provide a bit more context on this stuff? What to review? What the code is based on? Etc?

brikis98 · 2020-12-09T09:16:05Z

cmd/tspec/main.go

+var opts = tspec.Options{
+	Output:      colors.Colored(os.Stdout),
+	Format:      "pretty", // can define default values
+	Concurrency: 1,


What's this do?

brikis98 · 2020-12-09T09:16:54Z

cmd/tspec/main.go

+		Options:              &opts,
+	}.Run()
+
+	logger.Infof("Exit code is: %d", status)


Are you supposed to exit with this exit code? So if tests fail, you exit with a non-zero code?

brikis98 · 2020-12-09T09:17:58Z

cmd/tspec/main.go

+
+	currentDir, err := os.Getwd()
+	if err != nil {
+		logger.Fatalf("Error finding current directory: %s", err)


I believe this will panic under the hood, which I've always found surprising with a logger method... Might be clearer to make this more explicit.

brikis98 · 2020-12-09T09:18:57Z

features/terraform_aws_example.feature

@@ -0,0 +1,8 @@
+Feature: Terraform AWS Example


Is putting these in a features folder a BDD idiom? Or a requirement?

If it's not a requirement, I'd be tempted to put them under test/features instead...

brikis98 · 2020-12-09T09:20:15Z

modules/tspec/assert.go

+// assertion functions where you want to compare an expected and an actual value.
+func assertExpectedAndActual(a expectedAndActualAssertion, expected, actual interface{}, msgAndArgs ...interface{}) error {
+	var t asserter
+	a(&t, expected, actual, msgAndArgs...)


NIT: try to avoid one-letter variable names like a. Even though it's defined just 2 lines above, but the time you read this line, it's completely opaque what a is supposed to be.

brikis98 · 2020-12-09T09:20:59Z

modules/tspec/colors/ansi_others.go

+
+// +build !windows
+
+package colors


Is all this colors package stuff copied from somewhere?

brikis98 · 2020-12-09T09:21:56Z

modules/tspec/fmt.go

+// suite name and io.Writer to record output
+type FormatterFunc = formatters.FormatterFunc
+
+func printStepDefinitions(steps []*models.StepDefinition, w io.Writer) {


What is this formatter stuff? And step definitions? I think I'm missing some context on how this stuff all comes together.

brikis98 · 2020-12-09T09:23:00Z

modules/tspec/internal/formatters/fmt.go

+	file, line := f.FileLine(ptr)
+	dir := filepath.Dir(file)
+
+	fn := strings.Replace(f.Name(), dir, "", -1)


Struggling to understand this code too. In part, due to lots of one and two letter variable names, and in part, as I'm missing some context of how this BDD API is defined...

brikis98 · 2020-12-09T09:23:32Z

modules/tspec/internal/formatters/fmt_base.go

+}
+
+// Snippets ...
+func (f *Basefmt) Snippets() string {


Is this all code you wrote? Or is it copied from somewhere?

robmorgan · 2020-12-09T12:07:36Z

@brikis98 hey Jim, thanks for a first pass! I feel you may have missed my remark in the RFC: the code borrows heavily from the Godog Cucumber library. I think a good approach now is possibly how I've extended this library CLI/tool and maybe we should focus on reviewing my custom parts. An additional step would be figuring out how to reduce the amount of code here. This would be super easy for me to explain on a call, so I'm wondering if we should briefly chat on Friday during Ticket Jam.

p.s I'm not going to respond to your comments in the meantime.

brikis98 · 2020-12-09T12:33:21Z

@brikis98 hey Jim, thanks for a first pass! I feel you may have missed my remark in the RFC: the code borrows heavily from the Godog Cucumber library.

I saw that, but coudn't figure out which parts were borrowed, and which are unique? Also, is it possible to use those parts directly from godog, pulling them in as dependencies?

This would be super easy for me to explain on a call, so I'm wondering if we should briefly chat on Friday during Ticket Jam.

Sure, chatting live is fine too.

p.s I'm not going to respond to your comments in the meantime.

What a jerk 😄

robmorgan · 2020-12-10T02:34:46Z

@brikis98 actually on 2nd thoughts I'll leave a few comments here ahead of the call:

I saw that, but couldn't figure out which parts were borrowed, and which are unique? Also, is it possible to use those parts directly from Godog, pulling them in as dependencies?

Yes definitely! I was unsure how much of the library I'd actually need to modify, but now I have a better sense I'm wondering if we should simply import the parts we need and only sprinkle in our custom logic? The downsides would be:

We need to add additional imports to Terratest.
You've mentioned some parts of the code aren't very clear to you, e.g: their interfaces and two-letter variable names. If we import the code, we'll miss the opportunity to throw away the parts we don't need and also explain things more clearly.

On a more positive note, Godog is MIT-licensed, so we can either import the parts we need or vendor them directly into Terratest.

brikis98 · 2020-12-10T10:55:46Z

I saw that, but couldn't figure out which parts were borrowed, and which are unique? Also, is it possible to use those parts directly from Godog, pulling them in as dependencies?

Yes definitely! I was unsure how much of the library I'd actually need to modify, but now I have a better sense I'm wondering if we should simply import the parts we need and only sprinkle in our custom logic? The downsides would be:

We need to add additional imports to Terratest.

You've mentioned some parts of the code aren't very clear to you, e.g: their interfaces and two-letter variable names. If we import the code, we'll miss the opportunity to throw away the parts we don't need and also explain things more clearly.

On a more positive note, Godog is MIT-licensed, so we can either import the parts we need or vendor them directly into Terratest.

Adding imports is definitely preferred. In fact, this is a standalone CLI tool, so those imports shouldn't affect anyone not directly using the CLI tool.

francis-io · 2022-08-29T11:56:42Z

Hello 👋

Did the idea of using Terratest in a BDD framework get off the ground beyond what's in this issue?

robmorgan · 2022-09-20T00:35:08Z

@francis-io Hi Mark, sorry for the delayed reply. I've been away on vacation and am slowly getting caught up on everything. Short answer - No. Long answer: We've decided to prioritise other products and projects so this idea was temporarily put on hold. Also I don't have an ETA for when we'd pick it up again. You can somewhat try and use or take advantage of what I've pushed in this branch, but we wouldn't be able to offer any official support for it at this stage.

robmorgan added 14 commits December 8, 2020 16:59

ignore compiled tspec binary

317f260

fix .gitignore path

9d517c6

add current tspec wip

17e9c57

parse args as feature dir

3e8e18d

use tspec refs

32d8d5f

refactor keyword grammar

07c473e

fix go import path

a7304fd

go mod tidy

64d822c

rename feature

9aece89

add support for terraform input and env vars

302b3be

ensure the terraform aws feature can pass

f159307

automatically run tf destroy

399ded1

refactor tspec to use the terratest testing module. also move tf code…

b5d0a06

… to a feature struct

build tspec on circleci

70837c7

robmorgan requested review from brikis98 and yorinasub17 as code owners December 8, 2020 15:00

robmorgan added 3 commits December 8, 2020 23:09

go fmt

9b796e5

go fmt

0f64fd4

fix description

7facf40

brikis98 reviewed Dec 9, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: "tspec" - Add a BDD-style interface to Terratest #730

WIP: "tspec" - Add a BDD-style interface to Terratest #730

robmorgan commented Dec 8, 2020 •

edited

Loading

brikis98 left a comment

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

brikis98 Dec 9, 2020

robmorgan commented Dec 9, 2020 •

edited

Loading

brikis98 commented Dec 9, 2020

robmorgan commented Dec 10, 2020

brikis98 commented Dec 10, 2020

francis-io commented Aug 29, 2022

robmorgan commented Sep 20, 2022

WIP: "tspec" - Add a BDD-style interface to Terratest #730

Are you sure you want to change the base?

WIP: "tspec" - Add a BDD-style interface to Terratest #730

Conversation

robmorgan commented Dec 8, 2020 • edited Loading

RFC

PoC

TODO

Open Questions

brikis98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robmorgan commented Dec 9, 2020 • edited Loading

brikis98 commented Dec 9, 2020

robmorgan commented Dec 10, 2020

brikis98 commented Dec 10, 2020

francis-io commented Aug 29, 2022

robmorgan commented Sep 20, 2022

robmorgan commented Dec 8, 2020 •

edited

Loading

robmorgan commented Dec 9, 2020 •

edited

Loading