Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: prevent blocking by running Loki client handle in goroutine #19

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

danteu
Copy link

@danteu danteu commented Jul 27, 2024

  • When calling Client.Handle, the log entry is written to an unbuffered channel. This write blocks when the Loki client is trying to send a log batch to the Loki server. When the Loki server is unreachable, this can block for multiple minutes.
  • This change fixes the issue by running Client.Handle in a separate goroutine.

Fixes #18

- When calling Client.Handle, the log entry is written to an unbuffered
  channel. This write blocks when the Loki client is trying to send a
  log batch to the Loki server. When the Loki server is unreachable,
  this can block for multiple minutes.
- This change fixes the issue by running Client.Handle in a separate
  goroutine.

Fixes samber#18
@samber
Copy link
Owner

samber commented Jul 29, 2024

Hi @danteu and thanks for your bug fix.

IMO, this fix is very dangerous, since it might leak an infinite number of goroutines.

I opened an issue in the client repository for a more reliable fix.

@danteu
Copy link
Author

danteu commented Jul 29, 2024

Hi @samber,

thanks for the review. While I agree that this fix is quite dirty and will lead to a significant number of goroutines in scenarios with many calls to Client.Handle, those goroutines will eventually terminate when sendBatch returns. But if you're logging aggressively, the number of idle goroutines would probably be unreasonable.

Thanks for opening the issue with the client library! It looks abandoned/unmaintained so I'm not very hopeful for a timely fix. :/

Best regards,
Daniel

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Failure to send log message can block the process
2 participants