Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

receive: use async remote writing #7045

Merged
merged 1 commit into from
Jan 26, 2024
Merged

receive: use async remote writing #7045

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cmd/thanos/receive.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -831,6 +831,8 @@ type receiveConfig struct {
writeLimitsConfig *extflag.PathOrContent
storeRateLimits store.SeriesSelectLimits
limitsConfigReloadTimer time.Duration

asyncForwardWorkerCount uint
}

func (rc *receiveConfig) registerFlag(cmd extkingpin.FlagClause) {
Expand Down Expand Up @@ -888,6 +890,7 @@ func (rc *receiveConfig) registerFlag(cmd extkingpin.FlagClause) {

cmd.Flag("receive.replica-header", "HTTP header specifying the replica number of a write request.").Default(receive.DefaultReplicaHeader).StringVar(&rc.replicaHeader)

cmd.Flag("receive.forward.async-workers", "Number of concurrent workers processing forwarding of remote-write requests.").Default("5").UintVar(&rc.asyncForwardWorkerCount)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be breaking change to look out for if 5 is not sufficient for someone, right? I think we should mark it as "breaking" in changelog just so people can be careful in the rollout and know what to look for if 5 should not suffice. ( Even if its most likely enough for most people )

compressionOptions := strings.Join([]string{snappy.Name, compressionNone}, ", ")
cmd.Flag("receive.grpc-compression", "Compression algorithm to use for gRPC requests to other receivers. Must be one of: "+compressionOptions).Default(snappy.Name).EnumVar(&rc.compression, snappy.Name, compressionNone)

Expand Down
11 changes: 11 additions & 0 deletions docs/components/receive.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,14 @@ NOTE:
- Thanos Receive performs best-effort limiting. In case meta-monitoring is down/unreachable, Thanos Receive will not impose limits and only log errors for meta-monitoring being unreachable. Similarly to when one receiver cannot be scraped.
- Support for different limit configuration for different tenants is planned for the future.

## Asynchronous workers

Instead of spawning a new goroutine each time the Receiver forwards a request to another node, it spawns a fixed number of goroutines (workers) that perform the work. This allows avoiding spawning potentially tens or even hundred thousand goroutines if someone starts sending a lot of small requests.

This number of workers is controlled by `--receive.forward.async-workers=`.

Please see the metric `thanos_receive_forward_delay_seconds` to see if you need to increase the number of forwarding workers.

## Flags

```$ mdox-exec="thanos receive --help"
Expand Down Expand Up @@ -308,6 +316,9 @@ Flags:
--receive.default-tenant-id="default-tenant"
Default tenant ID to use when none is provided
via a header.
--receive.forward.async-workers=5
Number of concurrent workers processing
forwarding of remote-write requests.
--receive.grpc-compression=snappy
Compression algorithm to use for gRPC requests
to other receivers. Must be one of: snappy,
Expand Down
Loading
Loading