Add a new benchtool client #164

jtlisi · 2021-04-08T03:36:32Z

Adds a new benchtool tool that can be used to generate load against Prometheus remote-write and PromQL API backends.
Add documentation for the workload configuration file of the benchtool

I have been iterating on this tool for a while now on this branch. However, I feel it is now stable enough to publish in a alpha form.

note: CI is blocked by #165 since I used some functions that are v1.16+

56quarters

LGTM, few nits

56quarters · 2021-04-08T14:06:14Z

pkg/bench/query_runner.go

+
+	// Do DNS client side load balancing if configured
+	dnsProvider *dns.Provider
+	remoteMtx   sync.Mutex


Did this mutex used to guard a particular member variable? It's unclear to below when picking a random client what it's guarding? Everything?

A better name would be addressMtx. It guards the array of addresses. There is a background process updating that slice and we want to be sure all access is mutex protected.

cmd/benchtool/Dockerfile

56quarters · 2021-04-08T14:13:06Z

pkg/bench/query_runner.go

+	for _, queryReq := range q.workload.queries {
+		// every query has a ticker and a Go loop...
+		// not sure if this is a good idea but it should be fine
+		go func(q query) {


Shadowing the receiver here makes it kinda hard to follow this.

Good point. I'll change it to req

56quarters · 2021-04-08T14:15:43Z

pkg/bench/ring_check.go

+		}
+	}
+
+	fmt.Println("ring check:")


Should this use a logger as well?

The reason I printed directly to stdout instead of using logs is because this is the purpose of the ring check module. So we should be able to set the log level to fatal but still get this output.

56quarters · 2021-04-08T14:16:45Z

pkg/bench/write_client.go

+
+const maxErrMsgLen = 512
+
+var UserAgent = fmt.Sprintf("Prometheus/%s", version.Version)


Shouldn't we use our own user agent here?

We can, I'll update this to use benchtool

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

… to interval Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

* feat: initial working version of benchtool Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix(benchtool): register batch-size as default Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix(benchtool): ensure write loop actually loops Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * feat(benchtool): add logging and instrumentation Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix(benchtool): handle case where batchsize is bigger than timeseries Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix(benchtool): instantiate client map Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * chore(benchtool): make noisy log line debug level Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * chore(benchtool): remove ioutil Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * chore(benchtool): use Go 1.16 Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * feat(benchtool): instrument remote-write client Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix(benchtool): fix how metrics are instrumented Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * feat(benchtool): use resp.StatusCode in prom label Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * feat(benchtool): ensure labelset is used correctly Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * feat(benchtool): add logging config options Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add series type map to workload Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add ring-checking module Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix issue with null workload Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * remove early check Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * actual unique metric names Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * start ring Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * actual unique metric names Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * single worker at a time for remote write requests Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * reduce the number of concurrent writers and set default timeout equal to interval Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * increase the number of workers Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * remove work pool and use errgroup Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * use a closure Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * readd a worker channel Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * refactor bench runner into separate files Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * rename write_bench.go --> bench.go Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * start queries Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * stop point for query workload Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add random seed for query workload using adler32 checksum and id Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * improve further Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix no address panic Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix address resolution Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix address resolution Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix address resolution Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add jitter to queries Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * better buckets Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * increase parallel queries Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * move write configs into workload config file Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add support for tenant ID Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix query tenant flag Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * use instance-name flag for tenant name Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix duplicate flag registration bug Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * cleanup a bit Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * increase number of write workers Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * docs(benchtool): add documentation explaining the workload configuration Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * remove unused workload config option Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * remove extra docker tag push steps for local development Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * update README Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * remove from make all step to avoid Go 1.16 compat issue Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * fix linting issues Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * vary the number of writer workers by the number of configured replicas Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * add pprof Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com> * refactor per PR comments Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

@kwangil-ha

…ex (grafana#164) * Fix label and annotation overrides for services (thanks @kwangil-ha) Signed-off-by: ShuzZzle <niclas.schad@gmail.com> * Removed dnssrvnoa resolution from block memcached (probably oversight) and moved back to simple dns resolution Signed-off-by: ShuzZzle <niclas.schad@gmail.com> * add pr number to changelog Signed-off-by: ShuzZzle <niclas.schad@gmail.com>

jtlisi requested a review from a team as a code owner April 8, 2021 03:36

jtlisi added command/benchtool enhancement New feature or request labels Apr 8, 2021

jtlisi force-pushed the benchtool branch from b5251dd to 58ab1c0 Compare April 8, 2021 03:58

56quarters approved these changes Apr 8, 2021

View reviewed changes

jtlisi added 25 commits April 9, 2021 13:44

feat: initial working version of benchtool

4879c76

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix(benchtool): register batch-size as default

8791a85

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix(benchtool): ensure write loop actually loops

783fed7

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

feat(benchtool): add logging and instrumentation

ec3f5d4

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix(benchtool): handle case where batchsize is bigger than timeseries

4098b9a

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix(benchtool): instantiate client map

db4f8b6

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

chore(benchtool): make noisy log line debug level

c406be7

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

chore(benchtool): remove ioutil

f99e803

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

chore(benchtool): use Go 1.16

4715c9a

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

feat(benchtool): instrument remote-write client

65aeab8

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix(benchtool): fix how metrics are instrumented

2c2fc8d

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

feat(benchtool): use resp.StatusCode in prom label

e61e46a

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

feat(benchtool): ensure labelset is used correctly

9939c52

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

feat(benchtool): add logging config options

de23005

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

add series type map to workload

b03c734

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

add ring-checking module

f641b14

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix issue with null workload

6ab183b

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

remove early check

cc05e29

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

actual unique metric names

77a8e17

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

start ring

b340d89

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

actual unique metric names

d4ab692

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

single worker at a time for remote write requests

468e430

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

reduce the number of concurrent writers and set default timeout equal…

e22909e

… to interval Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

increase the number of workers

c01649e

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

remove work pool and use errgroup

52af4a1

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

jtlisi added 21 commits April 9, 2021 13:44

fix no address panic

c1c0cf7

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix address resolution

1c7551b

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix address resolution

eb9cbc2

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix address resolution

1f6a664

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

add jitter to queries

ed2bea7

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

better buckets

a1aa7b6

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

increase parallel queries

4b8c5d7

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

move write configs into workload config file

2a70358

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

add support for tenant ID

0cb48d2

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix query tenant flag

af5180c

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

use instance-name flag for tenant name

c5dae2c

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix duplicate flag registration bug

73be424

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

cleanup a bit

fd46958

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

increase number of write workers

bb6e20c

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

docs(benchtool): add documentation explaining the workload configuration

e726d58

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

remove unused workload config option

db98cc8

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

remove extra docker tag push steps for local development

57ba7cb

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

update README

9e021ae

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

remove from make all step to avoid Go 1.16 compat issue

4077325

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

fix linting issues

28bb454

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

vary the number of writer workers by the number of configured replicas

0f61531

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

jtlisi force-pushed the benchtool branch from ab36ec3 to 0f61531 Compare April 9, 2021 17:44

jtlisi added 2 commits April 9, 2021 15:52

add pprof

3922e1e

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

refactor per PR comments

d49e839

Signed-off-by: Jacob Lisi <jacob.t.lisi@gmail.com>

jtlisi merged commit 7c8b651 into main Apr 12, 2021

jtlisi deleted the benchtool branch April 12, 2021 16:25

simonswine mentioned this pull request Jan 7, 2022

Importing cortextool from separate repo cortex-tools grafana/mimir#708

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a new benchtool client #164

Add a new benchtool client #164

Uh oh!

jtlisi commented Apr 8, 2021 •

edited

Loading

Uh oh!

56quarters left a comment

Uh oh!

56quarters Apr 8, 2021

Uh oh!

jtlisi Apr 9, 2021

Uh oh!

Uh oh!

56quarters Apr 8, 2021

Uh oh!

jtlisi Apr 9, 2021

Uh oh!

56quarters Apr 8, 2021

Uh oh!

jtlisi Apr 9, 2021

Uh oh!

56quarters Apr 8, 2021

Uh oh!

jtlisi Apr 9, 2021

Uh oh!

Uh oh!


		const maxErrMsgLen = 512

		var UserAgent = fmt.Sprintf("Prometheus/%s", version.Version)

Add a new benchtool client #164

Add a new benchtool client #164

Uh oh!

Conversation

jtlisi commented Apr 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

56quarters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jtlisi commented Apr 8, 2021 •

edited

Loading