Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][prometheusexporter] Race condition between start and shutdown #36164

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Argannor
Copy link
Contributor

@Argannor Argannor commented Nov 4, 2024

Description

Adjusted the Start and Shutdown sequence of the prometheusexporter to prevent a race condition causing the close tcp 127.0.0.1:8999: use of closed network connection error as observed in #36139.

The behaviour was changed in the following ways:

  • If an error occurs during the creation of the server, close the listener immediately, leaving shutdown a noop
  • Since srv.Shutdown will close all open listeners, the explicit call to ln.Close in the shutdown routine was removed

Link to tracking issue

Fixes #36139

Testing

Unit testing, I temporarily increased the number of iterations on TestPrometheusExporter to 2000. The error did no longer occur. However sporadically another error occured:

=== RUN   TestPrometheusExporter
    prometheus_test.go:103: 
        	Error Trace:	C:/development/code/opentelemetry-collector-contrib/exporter/prometheusexporter/prometheus_test.go:84
        	Error:      	Received unexpected error:
        	            	listen tcp 127.0.0.1:8999: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
        	Test:       	TestPrometheusExporter
--- FAIL: TestPrometheusExporter (1.16s)

I assume that this is because the OS (in my case Windows) won't always close the underlying sockets immediately, blocking it for some time afterwards. I'm not sure there is anything we can do about that.

Documentation

n/a

@Argannor Argannor requested review from dashpole and a team as code owners November 4, 2024 14:28
Copy link

linux-foundation-easycla bot commented Nov 4, 2024

CLA Signed

The committers listed above are authorized under a signed CLA.

@Argannor Argannor force-pushed the fix/prometheusexporter-shutdown-server-2 branch from 3c9ef63 to 83a5851 Compare November 4, 2024 14:30
Signed-off-by: Argannor <arga@argannor.com>
@songy23 songy23 added the Run Windows Enable running windows test on a PR label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
exporter/prometheus Run Windows Enable running windows test on a PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[test][exporter/prometheus] TestPrometheusExporter failures
3 participants