Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Flush wavefront output sender on error to clean up broken connections #10225

Merged

Conversation

mikolajpe
Copy link
Contributor

@mikolajpe mikolajpe commented Dec 6, 2021

Required for all PRs:

resolves #8812

  1. update wavefront-sdk-go to the most recent version
  2. added flushing of the sender once it encounters an error, without it what's happening is that the connection is never cleaned and the client is constantly trying to send messages via broken connection. This means if the wavefront-proxy goes down and the connection breaks (for example when container has crushed or you are simply deploying new version of it) telegraf will never again acquire new connection and resume sending, with this fix it will

@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Dec 6, 2021

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@telegraf-tiger telegraf-tiger bot added the fix pr to fix corresponding bug label Dec 6, 2021
@mikolajpe
Copy link
Contributor Author

!signed-cla

@telegraf-tiger
Copy link
Contributor

telegraf-tiger bot commented Dec 6, 2021

Thanks so much for the pull request!
🤝 ✒️ Just a reminder that the CLA has not yet been signed, and we'll need it before merging. Please sign the CLA when you get a chance, then post a comment here saying !signed-cla

@mikolajpe mikolajpe changed the title fix(plugins/output/wavefront): Flush wavefront output sender on error to clean up broken connections fix: Flush wavefront output sender on error to clean up broken connections Dec 6, 2021
@mikolajpe mikolajpe closed this Dec 6, 2021
@mikolajpe mikolajpe reopened this Dec 6, 2021
@mikolajpe mikolajpe force-pushed the fix-output-wavefront-flush-sender-on-error branch from 69169c8 to 68f00ef Compare December 7, 2021 07:03
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this issue @mikolajprzybysz! One comment about which error to return. What do you think?

Comment on lines 176 to 177
w.Log.Errorf("wavefront sending error: %v", err)
return w.sender.Flush()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I would prefer

Suggested change
w.Log.Errorf("wavefront sending error: %v", err)
return w.sender.Flush()
if flushErr := w.sender.Flush(); flushErr != nil {
w.Log.Errorf("wavefront flushing error: %v", flushErr)
}
return err

as the sending error is the relevant one to return IMO. What do you think?

plugins/outputs/wavefront/wavefront.go Outdated Show resolved Hide resolved
@srebhan srebhan self-assigned this Dec 7, 2021
Copy link
Member

@srebhan srebhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks @mikolajprzybysz for the quick fix!

@srebhan srebhan added the ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review. label Dec 7, 2021
@powersj powersj merged commit 91cf764 into influxdata:master Dec 14, 2021
sspaink pushed a commit that referenced this pull request Dec 15, 2021
powersj pushed a commit to powersj/telegraf that referenced this pull request Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/wavefront fix pr to fix corresponding bug ready for final review This pull request has been reviewed and/or tested by multiple users and is ready for a final review.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants