Skip to content

Add option to gzip compress request bodies #239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 17, 2024

Conversation

VimCommando
Copy link
Contributor

When sending large volumes of data to the _bulk API it can be significantly more efficient to compress the bodies of HTTP requests. Thanks to JSON's high compression rates, in my testing I can see over 90% reductions in payload size.

Here are the first and last entires, along with two large payloads, from the same 744,057 document ingest run.

Uncompressed request body, total batch processing runtime: 6 minutes 37 seconds.

@timestamp                  action method status request_path                request_length
Nov 28, 2024 @ 01:21:29.072 bulk   POST      200 /metrics-node-default/_bulk       8,992,884
Nov 28, 2024 @ 01:15:00.898 bulk   POST      200 /metrics-index-default/_bulk     37,895,665
Nov 28, 2024 @ 01:15:00.808 bulk   POST      200 /metrics-index-default/_bulk     38,197,057
Nov 28, 2024 @ 01:14:52.887 bulk   POST      200 /settings-cluster-default/_bulk      42,523

Gzip compressed request body, total ingest runtime: 1 minutes 52 seconds.

@timestamp                  action method status request_path                request_length
Nov 28, 2024 @ 01:13:47.982 bulk   POST      200 /settings-index-default/_bulk     1,072,732
Nov 28, 2024 @ 01:12:00.972 bulk   POST      200 /metrics-index-default/_bulk      1,510,647
Nov 28, 2024 @ 01:12:00.963 bulk   POST      200 /metrics-index-default/_bulk      1,510,034
Nov 28, 2024 @ 01:11:55.512 bulk   POST      200 /settings-cluster-default/_bulk      10,103

These payloads compressed exceptionally well, between 70-95% smaller. Because this was measured from a home ISP to the cloud, the total runtime was significantly more impacted by limited upload bandwidth than the target cluster's compute.

I've added a request_body_compression boolean to the TransportBuilder which allows the request_builder to conditionally compresses the body's bytes and inject the Content-Encoding: gzip header.

@VimCommando VimCommando requested a review from swallez November 28, 2024 04:08
@VimCommando VimCommando marked this pull request as ready for review December 11, 2024 19:31
Copy link
Member

@swallez swallez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

@swallez swallez merged commit 5e95345 into elastic:main Dec 17, 2024
2 checks passed
swallez added a commit that referenced this pull request Dec 19, 2024
Co-authored-by: Ryan Eno <ryan.eno@elastic.co>
swallez added a commit that referenced this pull request Dec 19, 2024
Co-authored-by: Ryan Eno <ryan.eno@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants