-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Connection refused (os error 111) error.sources=[Connection refused (os error 111)] #12688
Comments
Hey @wamak9, Going to walk through the errors one-by-one to explain what they mean. 1. 2024-06-06T14:03:42.566024937Z [ 61835.932270s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:34180 server.addr=10.109.194.85:4143
2024-06-06T14:03:43.072400780Z [ 61836.438591s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:43040 server.addr=10.109.194.85:4143 This indicates that the client of this service ( 2. 2024-06-06T13:02:43.467367168Z time="2024-06-06T13:02:43Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462 An endpoint is marked as injected (probably has a control plane label) but it is not. Something went wrong where either the pod received the proxy with incomplete configuration, or the pod has been improperly annotated. This is not a typical occurence. Are you using any other features such as native sidecars? 3. 4. I think it's helpful to isolate the logs that are relevant here, and that's probably the first set of logs. From your original description:
Where exactly? The one not being mutually authenticated, or... For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod.
On your self-hosted prometheus? Is it injected or not injected? If it's not, then the annotation won't have anything to configure. Can you do a |
My self-hosted Prometheus is running on the same cluster and Linkerd is not injected on the Prometheus.
For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod. I am not sure what this means, when you say server listening are we talking about proxy and init ?
This is not a typical occurence. Are you using any other features such as native sidecars? |
This indicates that the proxy is trying to connect to port 80 in the local pod. Does the workload have a server that is listening on port 80? If not, this could indicate that there are clients (port scanners?) in your network trying to connect to arbitrary ports across an IP range, or it could indicate a misconfigured client. If you expect your application to serve traffic on port 80, then I would suggest looking at the application server to see if it is healthy. |
What is the issue?
I keep seeing
os error 111
issue on linkerd and have no idea how to fix it.More logs below.
IPs seen in the logs are releated to self-hosted Prometheus.
There are two sets on Prometheus running, one is from Linkerd and one is self hosted by us.
Tried to add bunch of annotations
How can it be reproduced?
Current version running is stable-2.14.9.
Enable linkerd on one of the namespace and then deploy.
Deploy prometheus on different namepace with no linkeerd annotation.
Logs, error output, etc
Linkerd enabled POD logs.
linkerd-destination logs
linkerd Proxy logs
Linkerd Policy Logs
output of
linkerd check -o short
Environment
Possible solution
N/A
Additional context
So, I tried to add linkerd to Monitoring namespace which is running Prometheus.
But that did not help and I am seeing way too many error logs in Prometheus on the same OS error.
I did add an authorization policy, so our self installed Prometheus can scrape metrics.
Would you like to work on fixing this bug?
maybe
The text was updated successfully, but these errors were encountered: