Open
Description
openedon Mar 27, 2024
What's wrong?
I have deployed Static mode Kubernetes operator. And I am using this for pushing the pod logs to loki.
Recently I found out that grafana-agent-logs pod is taking a lot of memory and it is also not able to push the logs.
After checking the logs of the config-reloader container I found some error messages.
level=error ts=2024-03-26T06:33:34.799302016Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:33:34.799311483Z caller=reloader.go:384 msg="Failed to trigger reload. Retrying." err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:34:04.803619827Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
After force deleting this pod the error msg in the config-reloader container is gone and grafana-agent-logs pod’s memory also comes down, it also starts pushing logs to loki.
Can you help me with the reason for the issue and with a better solution, as the current solution is not very optimal
Steps to reproduce
didn't find the reason of this bug
System information
No response
Software version
grafana agent v0.39.0
Configuration
No response
Logs
level=error ts=2024-03-26T06:33:34.799302016Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:33:34.799311483Z caller=reloader.go:384 msg="Failed to trigger reload. Retrying." err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded"
level=error ts=2024-03-26T06:34:04.803619827Z caller=runutil.go:100 msg="function failed. Retrying in next tick" err="trigger reload: reload request failed: Post \"http://127.0.0.1:8080/-/reload\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment