Flaky test pkg/config/tlscfg/TestReload (cert_watcher_test.go)

[yurishkuro]

https://travis-ci.com/github/jaegertracing/jaeger/jobs/445878517#L2150

=== RUN   TestReload
    cert_watcher_test.go:110: 
        	Error Trace:	cert_watcher_test.go:110
        	Error:      	Should be true
        	Test:       	TestReload
        	Messages:   	Failed to find wanted logs. All logs: [{{info 2020-11-23 14:13:36.375563204 +0000 UTC m=+0.010957869  Loading modified certificate undefined } [{certificate 15 0 /tmp/cert.crt414800473 <nil>} {event 15 0 WRITE <nil>}]} {{error 2020-11-23 14:13:36.377159282 +0000 UTC m=+0.012553950  Failed to load certificate undefined } [{certificate 15 0 /tmp/cert.crt414800473 <nil>} {event 15 0 WRITE <nil>} {error 26 0  tls: private key does not match public key}]} {{info 2020-11-23 14:13:36.381814355 +0000 UTC m=+0.017208997  Loading modified certificate undefined } [{certificate 15 0 /tmp/cert.crt414800473 <nil>} {event 15 0 WRITE <nil>}]} {{info 2020-11-23 14:13:36.383533045 +0000 UTC m=+0.018927712  Loaded modified certificate undefined } [{certificate 15 0 /tmp/cert.crt414800473 <nil>} {event 15 0 WRITE <nil>}]} {{info 2020-11-23 14:13:36.38356223 +0000 UTC m=+0.018956858  Loading modified certificate undefined } [{certificate 15 0 /tmp/key.crt548122084 <nil>} {event 15 0 WRITE <nil>}]}]
--- FAIL: TestReload (0.03s)

[albertteoh]

This is on me.

[yurishkuro]

@albertteoh it still fails https://github.com/jaegertracing/jaeger/runs/1458096624?check_suite_focus=true#step:5:1881

[albertteoh]

Thanks for the heads up @yurishkuro. It looks like the watcher isn't deterministically picking up every state transition of the cert file. In this case, it didn't detect the cert loading failure due to mismatching private & public keys, but only loaded it after both matching private and public keys were added to the cert, hence both updates were successful; which was not observed locally :(.

I'll put in a PR to remove the first assert since that signal is not reliable... and assert exactly on the condition we're waitUntil for. Please let me know if you can spot any holes in this proposed fix.

[albertteoh]

I think we can close this now; at least I haven't seen this flaky test manifest for some time.

I had incorrectly referenced #2654 in the PR #2655, which should have closed this issue. This is now addressed as well.