Skip to content

Stop kubernetes log tailer gracefully #1722

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Conversation

ptodev
Copy link
Collaborator

@ptodev ptodev commented Sep 20, 2024

PR Description

As mentioned in #281, users get this warning with loki.source.kubernetes:

level=warn msg="tailer stopped; will retry" target=monitoring/pod-id:grafana component=loki.source.kubernetes.pods err="http2: response body closed"

It looks like reader.ReadString('\n') gets called after stream.Close() has already been called.

Me and @wildum were looking at this today, and we suspect that decoupling the lifetime of the string from the timeout context + moving stream.Close() to the end of tail() could remove the warning message.

The warning message seems to be a cosmetic problem. No logs would be lost, but it's confusing since users don't expect to see a warning for something that works as expected. Making the shutdown more graceful will make the component logs less confusing.

Which issue(s) this PR fixes

Fixes #281

Notes to the Reviewer

I am not yet sure how to test this. When we test it, I'll take the PR out of the draft stage.

PR Checklist

  • CHANGELOG.md updated
  • Documentation added
  • Tests updated
  • Config converters updated

@ptodev ptodev force-pushed the ptodev/fix-kube-warn-log branch from 22f42f2 to 3b08cde Compare October 16, 2024 09:45
// return true, &v1.Pod{}, errors.New("Error creating secret")
// })

// fakeClientset.CoreV1().(*fakecorev1.FakeCoreV1).PrependReactor("logs", "alloy-pod1", func(action k8stesting.Action) (handled bool, ret runtime.Object, err error) {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main problem right now is that I can't get this line to work. I'm not sure how to fake a k8s API which doesn't send any log lines.

@CLAassistant
Copy link

CLAassistant commented Apr 2, 2025

CLA assistant check
All committers have signed the CLA.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

http2: response body closed
2 participants