Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support snappy compression in query frontend cache #3207

Merged
merged 4 commits into from
Sep 25, 2020
Merged

Support snappy compression in query frontend cache #3207

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e67dd85
support snappy compression in query frontend cache
yeya24 Sep 23, 2020
a13c445
add changelog
yeya24 Sep 23, 2020
800e2df
rename flag
yeya24 Sep 24, 2020
4a6c803
fix changelog entry
yeya24 Sep 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ We use *breaking :warning:* to mark changes that are not backward compatible (re
- [#3154](https://github.com/thanos-io/thanos/pull/3154) Query Frontend: Add metric `thanos_memcached_getmulti_gate_queries_max`.
- [#3146](https://github.com/thanos-io/thanos/pull/3146) Sidecar: Add `thanos_sidecar_prometheus_store_received_frames` histogram metric.
- [#3147](https://github.com/thanos-io/thanos/pull/3147) Querier: Add `query.metadata.default-time-range` flag to specify the default metadata time range duration for retrieving labels through Labels and Series API when the range parameters are not specified. The zero value means range covers the time since the beginning.
- [#3207](https://github.com/thanos-io/thanos/pull/3207) Query Frontend: Add `query-range.compression` flag to use compression in the query frontend cache.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- [#3207](https://github.com/thanos-io/thanos/pull/3207) Query Frontend: Add `query-range.compression` flag to use compression in the query frontend cache.
- [#3207](https://github.com/thanos-io/thanos/pull/3207) Query Frontend: Add `cache-compression-type` flag to use compression in the query frontend cache.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the catch! Please take a look again.


### Changed

Expand Down
10 changes: 8 additions & 2 deletions cmd/thanos/query_frontend.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,9 @@ func registerQueryFrontend(app *extkingpin.App) {

cfg.CachePathOrContent = *extflag.RegisterPathOrContent(cmd, "query-range.response-cache-config", "YAML file that contains response cache configuration.", false)

cmd.Flag("query-range.compression", "Use compression in results cache. Supported values are: 'snappy' and '' (disable compression).").
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we elaborate when it is used? On response? or when put in Memcached ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, maybe query-range.compression-type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is used before storing the response to Memcached. It doesn't make sense to use it when only fifo-cache is enabled. I should document that.

I am okay with either query-range.compression-type or query-range.compression. The flag is named to compression in Cortex

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is used before storing the response to Memcached. It doesn't make sense to use it when only fifo-cache is enabled. I should document that.

But it will work for other non FIFO backends as well, right? 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it compresses the response and then sends it to the results cache backend. If it is fifo-cache, because snappy cache and fifo cache are in memory, so not needed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technically it could be needed -> It might reduce memory consumed as well (:
But yea, the question is what is available right now (:

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah you are right. Sorry, my mistake. It is a tradeoff between CPU and Memory & network traffic

Copy link
Member

@bwplotka bwplotka Sep 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to be sorry - I think it's extremely useful. The only question is, do we need this for query range or some global cache option? (for future labels and instant)?

I think we need it for all so maybe some generic:

Suggested change
cmd.Flag("query-range.compression", "Use compression in results cache. Supported values are: 'snappy' and '' (disable compression).").
cmd.Flag("cache-compression-type", "Use compression in results cache. Supported values are: 'snappy' and '' (disable compression).").

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the option can be global. But considering other caching configs, I am not sure. This is another problem and maybe we can have a separate issue to talk about it. The main question is, can we use different cache backends and different cache configs for each endpoint we want to cache? @bwplotka @kakkoyun

For example, it is not very good to reuse the same cache config because the response sizes are different for range queries and instant queries. They might need different limits and different max_items configs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Totally agree. To me, something will be the same, some things different (:

Default("").StringVar(&cfg.CacheCompression)

cmd.Flag("query-frontend.downstream-url", "URL of downstream Prometheus Query compatible API.").
Default("http://localhost:9090").StringVar(&cfg.CortexFrontendConfig.DownstreamURL)

Expand Down Expand Up @@ -107,14 +110,17 @@ func runQueryFrontend(
level.Warn(logger).Log("msg", "memcached cache valid time set to 0, so using a default of 24 hours expiration time")
cfg.CortexResultsCacheConfig.CacheConfig.Memcache.Expiration = 24 * time.Hour
}
cfg.CortexResultsCacheConfig = cacheConfig
cfg.CortexResultsCacheConfig = &queryrange.ResultsCacheConfig{
Compression: cfg.CacheCompression,
CacheConfig: *cacheConfig,
}
}

if err := cfg.Validate(); err != nil {
return errors.Wrap(err, "error validating the config")
}

fe, err := cortexfrontend.New(*cfg.CortexFrontendConfig, logger, reg)
fe, err := cortexfrontend.New(*cfg.CortexFrontendConfig, nil, logger, reg)
if err != nil {
return errors.Wrap(err, "setup query frontend")
}
Expand Down
3 changes: 3 additions & 0 deletions docs/components/query-frontend.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,9 @@ Flags:
'query-range.response-cache-config-file' flag
(lower priority). Content of YAML file that
contains response cache configuration.
--query-range.compression=""
Use compression in results cache. Supported values
are: 'snappy' and ” (disable compression).
--query-frontend.downstream-url="http://localhost:9090"
URL of downstream Prometheus Query compatible API.
--query-frontend.compress-responses
Expand Down
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ require (
github.com/cespare/xxhash v1.1.0
github.com/chromedp/cdproto v0.0.0-20200424080200-0de008e41fa0
github.com/chromedp/chromedp v0.5.3
github.com/cortexproject/cortex v1.3.1-0.20200901115931-255ff3306960
github.com/cortexproject/cortex v1.3.1-0.20200923145333-8587ea61fe17
github.com/davecgh/go-spew v1.1.1
github.com/facette/natsort v0.0.0-20181210072756-2cd4dd1e2dcb
github.com/fatih/structtag v1.1.0
github.com/felixge/fgprof v0.9.1
github.com/fsnotify/fsnotify v1.4.7
github.com/fsnotify/fsnotify v1.4.9
github.com/go-kit/kit v0.10.0
github.com/go-openapi/strfmt v0.19.5
github.com/gogo/protobuf v1.3.1
Expand Down Expand Up @@ -51,7 +51,7 @@ require (
github.com/prometheus/prometheus v1.8.2-0.20200819132913-cb830b0a9c78
github.com/uber/jaeger-client-go v2.25.0+incompatible
github.com/uber/jaeger-lib v2.2.0+incompatible
github.com/weaveworks/common v0.0.0-20200820123129-280614068c5e
github.com/weaveworks/common v0.0.0-20200914083218-61ffdd448099
go.elastic.co/apm v1.5.0
go.elastic.co/apm/module/apmot v1.5.0
go.uber.org/atomic v1.6.0
Expand Down
Loading