Integration tests fail intermittently

[petemounce]

Describe the bug

Our tests have external dependencies

• DNS - dnstest.io
• http - httpbin.org (example https://travis-ci.org/github/aelsabbahy/goss/jobs/729563840#L5278 via Mention flag and env-var for alpha #621)

How To Reproduce

Keep retrying.

Expected Behavior

Tests don't fail for external reasons.

Actual Behavior

If an external dependency is unavailable, our tests fail.

Environment:

• Version of goss
• OS/Distribution version (if applicable)

[petemounce]

httpbin

Used within integration-tests/goss/goss-shared.yaml.

Could replace with a trivial golang http server.

dnstest

Used within integration-tests\goss\goss-shared.yaml.

Could replace with a https://coredns.io/ - I don't know very much about DNS server choices, but this looked at a glance easy to configure.

Orchestration

We'd need to spin up both dependencies, then run the integration tests, then tear down the dependencies. I think that could be achieved via a docker-compose.yml with some judicious port-maps.

I think that could form the basis of reworking the linux-based integration tests, since each test-container could maybe become registered into the compose file - I think that might simplify away a bunch of shell script & makefile.

Approach

If I start this before someone else does (please say hi!), I think I'd

1. write the httpbin dep
2. wrap the docker-compose.yml around it
3. wedge in the docker-compose up --daemon to the integration-tests/test.sh at the top, and tear it down in a trap ... EXIT

That would get us to a working loop, even if it's inefficient to spin up and tear down per test-suite rather than for the whole CI run.

I'd probably then

1. add the dnstest slice
2. figure out if I could refactor the Makefile/test.sh to remove the per-suite dependency and turn it into whole CI run.
   • but I'd only do that if it seemed worth the complexity for the speed win
   • ... and really, I'd want to do more in-depth surgery to the integration tests approach, but I haven't got a handle on exactly what yet.

[aelsabbahy]

One thing I would be interested in before actual work is done on this is a brief research on the pros/cons of using docker-compose vs https://github.com/testcontainers/testcontainers-go

The later may be less flexible, but one less build dependency to worry about? shrug

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[aelsabbahy]

Marking this as approved.

[aelsabbahy]

Status update:

• httpbin change was done
• dns needs to be done
• docker-compose needs to be done