Flaky Test - prometheusexporter.TestEndToEndSummarySupport

[dmitryax]

Seen here.

=== RUN   TestEndToEndSummarySupport
    end_to_end_test.go:181: Left-over unmatched Prometheus scrape content: "test_scrape_samples_post_metric_relabeling 0 1646857445910test_scrape_samples_scraped 0 1646857445910test_scrape_series_added 0 1646857445910test_up 0 1646857445910"
--- FAIL: TestEndToEndSummarySupport (5.11s)

[jpkrohling]

And here: https://github.com/open-telemetry/opentelemetry-collector-contrib/runs/5996006951?check_suite_focus=true

[gouthamve]

So, debugging this (with some debug logging config added):

❯ go test . -run=TestEndToEndSummarySupport -count=1 -v
=== RUN   TestEndToEndSummarySupport
{"level":"debug","msg":"Starting provider","provider":"static/0","subs":"map[otel-collector:{}]"}
{"level":"debug","msg":"Discoverer channel closed","provider":"static/0"}
{"level":"debug","msg":"Scrape failed","scrape_pool":"otel-collector","target":"http://127.0.0.1:62688/metrics","error":"Get \"http://127.0.0.1:62688/metrics\": context deadline exceeded"}
--- PASS: TestEndToEndSummarySupport (5.08s)

Basically we scrape a test server 8 times with a scrape-interval of 10ms, but the global scrape interval is 2ms, so the scrape timeout is also set to 2ms. See: https://github.com/prometheus/prometheus/blob/fb2da1f26aec023b1e3c864222aaccbe01969f11/config/config.go#L382-L388 and https://github.com/prometheus/prometheus/blob/fb2da1f26aec023b1e3c864222aaccbe01969f11/config/config.go#L297-L303

If the 8th scrape fails, then we fail the test with the error:

=== RUN   TestEndToEndSummarySupport
    end_to_end_test.go:181: Left-over unmatched Prometheus scrape content: "test_scrape_samples_post_metric_relabeling 0 1649790752438test_scrape_samples_scraped 0 1649790752438test_scrape_series_added 0 1649790752438test_up 0 1649790752438"
--- FAIL: TestEndToEndSummarySupport (5.11s)

Proof (I added a fmt.Println("currentScrapeIndex", currentScrapeIndex)):

=== RUN   TestEndToEndSummarySupport
{"level":"debug","msg":"Starting provider","provider":"static/0","subs":"map[otel-collector:{}]"}
{"level":"debug","msg":"Discoverer channel closed","provider":"static/0"}
{"level":"debug","msg":"Scrape failed","scrape_pool":"otel-collector","target":"http://127.0.0.1:62815/metrics","error":"Get \"http://127.0.0.1:62815/metrics\": context deadline exceeded"}
currentScrapeIndex 1
currentScrapeIndex 2
currentScrapeIndex 3
currentScrapeIndex 4
currentScrapeIndex 5
currentScrapeIndex 6
currentScrapeIndex 7
currentScrapeIndex 8
{"level":"debug","msg":"Scrape failed","scrape_pool":"otel-collector","target":"http://127.0.0.1:62815/metrics","error":"Get \"http://127.0.0.1:62815/metrics\": context deadline exceeded"}
    end_to_end_test.go:182: Left-over unmatched Prometheus scrape content: "test_scrape_samples_post_metric_relabeling 0 1649814848841test_scrape_samples_scraped 0 1649814848841test_scrape_series_added 0 1649814848841test_up 0 1649814848841"
--- FAIL: TestEndToEndSummarySupport (5.46s)

I propose increasing the timeout to 100ms. Sending a PR for the same.