Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for gzip compression to amqp input and output #5830

Merged
merged 4 commits into from
May 20, 2019

Conversation

danielnelson
Copy link
Contributor

closes #3557

Required for all PRs:

  • Signed CLA.
  • Associated README.md updated.
  • Has appropriate unit tests.

@danielnelson danielnelson added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label May 10, 2019
@danielnelson danielnelson added this to the 1.11.0 milestone May 10, 2019
@danielnelson danielnelson mentioned this pull request May 10, 2019
3 tasks
@Esity
Copy link

Esity commented May 16, 2019

We should add in the compression message property so we know in RMQ if it is compressed. This should be set based on if compression is enabled or not.

Content encoding, e.g. "gzip". Used by applications, not core RabbitMQ
While we are at it, we should also add one more

Content type, e.g. "application/json". Used by applications, not core RabbitMQ
Grabbed from this page https://www.rabbitmq.com/consumers.html

@Esity
Copy link

Esity commented May 16, 2019

Also tested with and without gzip compression
I added 10k messages to RMQ without 10k metric points per message for a total of 10,000,000 metrics.

Without compression the size on disk was over 4gb. With compression turned on it was 7kB. I would consider that a success @danielnelson

It might be worth testing to see what the additional overhead against Telegraf is for enabling compression but I think it is worth it

@danielnelson
Copy link
Contributor Author

I added the Content-Encoding property, but held off on the Content-Type. I think we should first make this part of the serializer interface separately so that we can pick the right value in a nice way. Another idea that this brings up is doing automatic decoding on the consumer side based on the property, I think it could be nice but I'll have to leave that for later.

@danielnelson danielnelson merged commit b5cd9a9 into master May 20, 2019
@danielnelson danielnelson deleted the amqp-compress branch May 20, 2019 21:36
hwaastad pushed a commit to hwaastad/telegraf that referenced this pull request Jun 13, 2019
bitcharmer pushed a commit to bitcharmer/telegraf that referenced this pull request Oct 18, 2019
athoune pushed a commit to bearstech/telegraf that referenced this pull request Apr 17, 2020
idohalevi pushed a commit to idohalevi/telegraf that referenced this pull request Sep 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/rabbitmq feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AMQP Body Compression
3 participants