Flaky integration tests

[klingerf]

This is a tracking ticket for integration tests that have failed in CI. I think most of these failures could be fixed by tweaking some of our test timeouts, but I wanted to track them in one place to see how frequently they're occurring.

[klingerf]

6/28, https://travis-ci.org/runconduit/conduit/jobs/397919327:

--- FAIL: TestCliTap (95.16s)
    --- PASS: TestCliTap/tap_a_deployment (18.07s)
    --- PASS: TestCliTap/tap_a_service_call (3.99s)
    --- FAIL: TestCliTap/tap_a_pod (60.51s)
    	tap_test.go:137: cmd [/home/travis/gopath/src/github.com/runconduit/conduit/conduit tap pod/t3-6768bd8c64-q7wdn --namespace conduit-git-9f274526-tap-test --conduit-namespace conduit-git-9f274526] Timed out trying to read 10 lines
    --- PASS: TestCliTap/filter_tap_events_by_method (3.49s)
    --- PASS: TestCliTap/filter_tap_events_by_authority (4.00s)
FAIL
FAIL	command-line-arguments	95.189s

This specific test failure is captured in #1227.

[klingerf]

6/29, https://travis-ci.org/runconduit/conduit/jobs/398429935:

Running test [get_test.go]
=== RUN   TestCliGet
=== RUN   TestCliGet/get_pods_from_--all-namespaces
=== RUN   TestCliGet/get_pods_from_the_conduit_namespace
--- FAIL: TestCliGet (4.88s)
    --- FAIL: TestCliGet/get_pods_from_--all-namespaces (0.05s)
    	get_test.go:87: Unexpected error: exit status 1 output:
    		Error: error reading byte stream header: error while reading bytes from message: unexpected EOF
    		Usage:
    		  conduit get [flags] pods
    		
    		Examples:
    		  # get all pods
    		  conduit get pods
    		
    		  # get pods from namespace conduit
    		  conduit get pods --namespace conduit
    		
    		Flags:
    		      --all-namespaces     If present, returns pods across all namespaces, ignoring the "--namespace" flag
    		  -h, --help               help for get
    		  -n, --namespace string   Namespace of pods (default "default")
    		
    		Global Flags:
    		      --api-addr string            Override kubeconfig and communicate directly with the control plane at host:port (mostly for testing)
    		  -c, --conduit-namespace string   Namespace in which Conduit is installed (default "conduit")
    		      --kubeconfig string          Path to the kubeconfig file to use for CLI requests
    		      --verbose                    Turn on debug logging
    		
    --- FAIL: TestCliGet/get_pods_from_the_conduit_namespace (0.04s)
    	get_test.go:100: Unexpected error: exit status 1 output:
    		Error: error reading byte stream header: error while reading bytes from message: unexpected EOF
    		Usage:
    		  conduit get [flags] pods
    		
    		Examples:
    		  # get all pods
    		  conduit get pods
    		
    		  # get pods from namespace conduit
    		  conduit get pods --namespace conduit
    		
    		Flags:
    		      --all-namespaces     If present, returns pods across all namespaces, ignoring the "--namespace" flag
    		  -h, --help               help for get
    		  -n, --namespace string   Namespace of pods (default "default")
    		
    		Global Flags:
    		      --api-addr string            Override kubeconfig and communicate directly with the control plane at host:port (mostly for testing)
    		  -c, --conduit-namespace string   Namespace in which Conduit is installed (default "conduit")
    		      --kubeconfig string          Path to the kubeconfig file to use for CLI requests
    		      --verbose                    Turn on debug logging
    		
FAIL
FAIL	command-line-arguments	4.898s

This seems to indicate that the CLI received a non-200 response from its call to the public API's ListPods endpoint.

[klingerf]

7/3, https://travis-ci.org/runconduit/conduit/jobs/399791442:

Running test [install_test.go]
=== RUN   TestVersionPreInstall
--- PASS: TestVersionPreInstall (0.03s)
=== RUN   TestInstall
--- FAIL: TestInstall (60.81s)
	install_test.go:99: Error validating deploy [grafana]:
		Expected deployment [grafana] in namespace [conduit-git-1c5e9eea] to have [1] replicas, but found [0]
=== RUN   TestVersionPostInstall
--- PASS: TestVersionPostInstall (0.03s)
=== RUN   TestCheck
--- PASS: TestCheck (0.10s)
=== RUN   TestDashboard
--- PASS: TestDashboard (0.55s)
=== RUN   TestInject
--- PASS: TestInject (5.53s)
FAIL
FAIL	command-line-arguments	67.073s

And:

--- FAIL: TestCliStatForConduitNamespace (20.89s)
    --- FAIL: TestCliStatForConduitNamespace/conduit_stat_deploy_-n_conduit-git-1c5e9eea (20.06s)
    	stat_test.go:144: Expected success rate [100.00%] for [grafana], got [-]

These failures appear to indicate that that the grafana pod in the conduit namespace took a really long time to start, exceeding our 1-minute timeout.

[klingerf]

7/4, https://travis-ci.org/runconduit/conduit/jobs/399848343:

Running test [get_test.go]
=== RUN   TestCliGet
=== RUN   TestCliGet/get_pods_from_--all-namespaces
=== RUN   TestCliGet/get_pods_from_the_conduit_namespace
--- FAIL: TestCliGet (31.28s)
	get_test.go:80: Error validating pods for deploy [cli-get-test-d1]:
		Expected deployment [cli-get-test-d1] in namespace [conduit-git-a17177f7-get-test] to have [2] running pods, but found [1]
    --- PASS: TestCliGet/get_pods_from_--all-namespaces (0.05s)
    --- PASS: TestCliGet/get_pods_from_the_conduit_namespace (0.03s)
FAIL
FAIL	command-line-arguments	31.304s

And:

Running test [egress_test.go]
=== RUN   TestEgressHttp
=== RUN   TestEgressHttp/Can_use_egress_to_send_GET_request_to_http_(egress-test-http-get-svc)
=== RUN   TestEgressHttp/Can_use_egress_to_send_POST_request_to_http_(egress-test-http-post-svc)
=== RUN   TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-https-get-svc)
=== RUN   TestEgressHttp/Can_use_egress_to_send_POST_request_to_https_(egress-test-https-post-svc)
=== RUN   TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-not-www-get-svc)
--- FAIL: TestEgressHttp (35.86s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_http_(egress-test-http-get-svc) (4.20s)
    --- FAIL: TestEgressHttp/Can_use_egress_to_send_POST_request_to_http_(egress-test-http-post-svc) (30.01s)
    	egress_test.go:52: Unexpected error: GET request to [http://127.0.0.1:39654/api/v1/namespaces/conduit-git-a17177f7-egress-test/services/egress-test-http-post-svc:http/proxy/] returned status [503]
    		{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"no endpoints available for service \"egress-test-http-post-svc:http\"","reason":"ServiceUnavailable","code":503}
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-https-get-svc) (0.28s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_POST_request_to_https_(egress-test-https-post-svc) (0.26s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-not-www-get-svc) (0.30s)
FAIL
FAIL	command-line-arguments	35.888s

These two failures seem unrelated to me. The first one is a result of the conduit get command not returning the correct number of pods. That test blocks until all pods are in the Ready state, and then makes the call to conduit get. My guess is that there's a race and the public-api's pod cache hasn't been fully updated before the conduit get call is made.

The second failure indicates that a call to a service that proxies traffic externally failed, but there's not enough additional information to indicate where the failure occurred.

[klingerf]

7/5, https://travis-ci.org/runconduit/conduit/jobs/400541164:

--- FAIL: TestEgressHttp (37.30s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_http_(egress-test-http-get-svc) (3.22s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_POST_request_to_http_(egress-test-http-post-svc) (1.17s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-https-get-svc) (0.48s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_POST_request_to_https_(egress-test-https-post-svc) (0.48s)
    --- FAIL: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-not-www-get-svc) (31.01s)
    	egress_test.go:52: Unexpected error: GET request to [http://127.0.0.1:39587/api/v1/namespaces/conduit-git-d42d7cb0-egress-test/services/egress-test-not-www-get-svc:http/proxy/] returned status [500]
    		error handling http request: unexpected status returned by [https://httpbin.org/get]for requestUID [in:http-sid:http-egress-grpc:-1-h1:8080-389108836]: 503
FAIL
FAIL	command-line-arguments	37.327s

In this test it appears that the request from our cluster to https://httpbin.org/get returned a 503.

[klingerf]

7/5, https://travis-ci.org/runconduit/conduit/jobs/400652210:

--- FAIL: TestEgressHttp (36.15s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_http_(egress-test-http-get-svc) (3.18s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_POST_request_to_http_(egress-test-http-post-svc) (1.11s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-https-get-svc) (0.31s)
    --- PASS: TestEgressHttp/Can_use_egress_to_send_POST_request_to_https_(egress-test-https-post-svc) (0.25s)
    --- FAIL: TestEgressHttp/Can_use_egress_to_send_GET_request_to_https_(egress-test-not-www-get-svc) (30.31s)
    	egress_test.go:52: Unexpected error: GET request to [http://127.0.0.1:35777/api/v1/namespaces/conduit-git-fd1aecfa-egress-test/services/egress-test-not-www-get-svc:http/proxy/] returned status [500]
    		error handling http request: unexpected status returned by [https://httpbin.org/get]for requestUID [in:http-sid:http-egress-grpc:-1-h1:8080-617159869]: 503
FAIL
FAIL	command-line-arguments	36.175s

This appears to be identical to the failure in the previous comment.

[klingerf]

I think the egress test might be getting tripped up by rate limiting on httpbin.org. At least testing locally, that site returns a 503 after 6 successful requests.

$ while true; do curl https://httpbin.org/get; done
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
{"args":{},"headers":{"Accept":"*/*","Connection":"close","Host":"httpbin.org","User-Agent":"curl/7.50.3"},"origin":"199.116.73.216","url":"https://httpbin.org/get"}
The deployment is currently unavailable