Enabling compression between Kibana <--> Elasticsearch servers

[mshustov]

Kibana relies on the elasticsearch-js client defaults with compression: false https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/basic-config.html

While enabling compression adds some runtime performance overhead, it might drastically reduce transmission time and increase network bandwidth.
We should conduct load testing for Cloud and on-prem instances to decide whether Kibana will set compression: true by default or we expose it as elasticsearch.compression configuration setting.

A side note: it might be interesting to calculate how much it affects the Data transfer reductions initiative on Cloud.

Subtasks

• allow to configure compression for the ES client - Allow to enable compression for communications with ES #124009
• conduct load testing to understand the bandwidth and cpu usage consequences of enabling compression
• decide if we should enable compression by default

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[delvedor]

The v7 client has two compression options:

• suggestCompression: if true, asks ES for compressed data via accept-encoding header.
• compression: if true, sends compressed data to ES

The v8 client has a single compression option, which does both.

Furthermore, if you enable compression and are using maxResponseSize, remember to configure maxCompressedResponseSize as well.

The client defaults to false unless you are using Elastic Cloud (detected via the cloud id option), in such case, it will enable compression by default as it's recommended with Elastic Cloud.

[mshustov]

The client defaults to false unless you are using Elastic Cloud (detected via the cloud id option), in such case, it will enable compression by default as it's recommended with Elastic Cloud.

Ok, we have to set compression: true explicitly then. Since Kibana doesn't configure the cloud option.

[pgayvallet]

We should conduct load testing for Cloud and on-prem instances to decide whether Kibana will set compression: true by default or we expose it as elasticsearch.compression configuration setting.

We could start by exposing this new elasticsearch.compression config property while preserving the current default value of false, and then, depending on the results of the perf/load testing, decide if we want to switch the default to true later. It would allow customers really wanting/needing this feature to use it asap.

[massimobrignoli]

I agree with this approach. One of our customer with 1000 users in kibana, 3000 dashboards and 4000 queries/minute has tested it and has seen the network traffic dropping by 90% going from 500GB/hour to 50GB/hour.

Of course not all the query will benefit (small queries with a very small result set can be slower) and the CPU usage will increase a bit. But let the users decide.

[pgayvallet]

I opened #124009 to introduce a new elasticsearch.compression configuration property that will default to false (which is the effective current value) for now.

[pgayvallet]

#124009 was merged, I updated the issue accordingly and added a subtasks list.

[pgayvallet]

I performed some testing around the performance and bandwidth impact of enabling compression for the Kibana<->ES communications.

Bandwidth

I couldn't find any proper way to monitor the bandwidth usage when running the load tests, so I felt back to using a homebrew proxy between Kibana and ES to monitor the request and response size of some queries while manually navigating within Kibana.

No big surprise here, the gain is what you would expect from enabling compression on any HTTP-based communication. Depending on the size of the request and response, there is a 20% to 90% compression.

I won't list everything here, but for example the /_search response associated with loading the sample data's dashboard from the dashboard listing page get compressed around 85%, and that's only returning 3 dashboard documents.

// without compression
  request length: 263
  response length: 66893

// with compression
  request length: 181
  response length: 10884

Overall, we should be expecting an overall reduction in bandwidth usage when enabling compression of at least 60%, and probably more like 80%, depending on the specific usages of the instance/customer, which is significant.

Performance impact.

I couldn't perform load testing against the whole stack on Cloud, given the elasticsearch.compression is not allow-listed, so I performed two suites of tests:

1. Local Kib + Local ES
2. Local Kib + Remote ES (in Cloud)

(all suites were ran 3 times, with very similar results so I'm only showing one of the 3 results for each)

Local Kibana on a local ES - 100 users

Without compression

With compression

Analysis

This was surprising. Given that compression/decompression is done outside of the main event loop in node, I was expecting a negligible impact on performance. Tests show that this is far from negligible when Kibana/ES are under heavy load in that scenario.

• Overall mean is +50% when compression is enabled (900 vs 600). When drilling down, it vary between +15% and +100% depending on the requests. Supposedly, the biggest the response, the highest the difference in mean value.

• Min value is not affected much for any endpoint

• Overall 50th pct and 75th pct are doubled (!) . When drilling down, the results vary a lot, from no difference to twice the response time.

• 95th and 99th pcts are less impacted than the previous ones, with approx 25% difference between compression and no compression.

Local Kibana on a Cloud ES - 200 users

Without compression

With compression

Analysis

The results are way more acceptable than during local testing, which tend to indicate that either the bottleneck during local testing was more on the ES side than on the Kibana side, or that adding real latency for the communications (that can't be reproduced when hitting the local loopback in a local-to-local scenario) reduces the compression/decompression overhead significantly.

• Min value is not affected much for any endpoint

• The 50th pct is 25% higher with compression enabled, and mostly consistent for each endpoint

• 75th, 95th and 99th are almost the same with or without compression

Conclusion

1. Should we set elasticsearch.compression to true for all Cloud instances?

Good question, and not sure who should be answering it. We should probably reach out to Cloud to decide whether the bandwidth reduction is worth the performance impact?

@stacey-gammon @lukeelmers wdyt?

2. Should we allow customers to configure elasticsearch.compression on Cloud

If the answer to the previous question is 'no', then ihmo yes. We confirmed that it works correctly and that the performance impact is not significant enough to restrict the usage of this option.