-max-rate gives much lower throughput than desired

[tgrabiec]

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=100000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 100000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 30.239109ms
Total ops:	 3026
Total rows:	 30250
Operations/s:	 500.349773593381
Rows/s:		 5001.84423370779
Latency:
  max:		 2.850815ms 
  99.9th:	 1.376255ms 
  99th:		 1.015807ms 
  95th:		 753.663µs 
  90th:		 688.127µs 
  median:	 622.591µs

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=80000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 80000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 7.300328ms
Total ops:	 585
Total rows:	 5850
Operations/s:	 400.72400209705376
Rows/s:		 4007.240020970538
Latency:
  max:		 2.719743ms 
  99.9th:	 1.441791ms 
  99th:		 1.081343ms 
  95th:		 819.199µs 
  90th:		 720.895µs 
  median:	 655.359µs

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=70000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 70000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 11.012372ms
Total ops:	 773
Total rows:	 7720
Operations/s:	 350.9807348881303
Rows/s:		 3505.2668477831385
Latency:
  max:		 1.310719ms 
  99.9th:	 1.212415ms 
  99th:		 1.081343ms 
  95th:		 786.431µs 
  90th:		 720.895µs 
  median:	 655.359µs

[dahankzter]

Was the generator machine fully utilized in this cases?

[dahankzter]

Did you update scylla-bench? We switched to using the shard aware driver a few days ago. Perhaps it causes a regression. Can you try to down grade? Should be only the top commit or so.

[fruch]

we need to test this again, to see if s-b throttling behaves as expected or not.

[vponomaryov]

we need to test this again, to see if s-b throttling behaves as expected or not.

This bug is the same as another fixed one:

• Concurrency flag behaves completely wrong when the "concurrency" is larger than the number of partitions in sequential mode #138

The fix was merged here:

• Fix row-thread mapping for sequential workloads #153

Math proof:

 :~/src/scylla-bench $ ./scylla-bench -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=100000
Configuration
Mode:			 write
Workload:		 sequential
Timeout:		 5s
Consistency level:	 quorum
Partition count:	 10
Clustering rows:	 10000000
Clustering row size:	 5120
Rows per request:	 10
Page size:		 1000
Concurrency:		 200
Connections:		 4
Maximum rate:		 100000 op/s
Client compression:	 true

time		operations/s	rows/s		errors	max		99.9th		99th		95th		90th		median
^C
interrupted

Results
Time (avg):	 30.239109ms
Total ops:	 3026
Total rows:	 30250
Operations/s:	 500.349773593381
Rows/s:		 5001.84423370779
Latency:
  max:		 2.850815ms 
  99.9th:	 1.376255ms 
  99th:		 1.015807ms 
  95th:		 753.663µs 
  90th:		 688.127µs 
  median:	 622.591µs

This command was configured to use 200 threads, 100000 op/s and only 10 partitions.
Due to the #138 bug only 1 thread out of 200 was processing anything doing it's equal part ->
100000 / 200 = 500

And the command result if following:
> Operations/s: 500.349773593381

So, this bug must be marked as FIXED like the another newer one: #138

[fruch]

@vponomaryov

I'm not sure I understood the mathematical proof here, so the results summary only show data from one "thread" ?
or that only one thread was working cause of #138

can we show similar command running now, that show correct behavior ?

[vponomaryov]

@vponomaryov

I'm not sure I understood the mathematical proof here, so the results summary only show data from one "thread" ?

Only one thread was doing anything, all other were idle. So, summary data is equal to the "last" thread data.

or that only one thread was working cause of #138

This bug describes completely the same problem as #138 does.

can we show similar command running now, that show correct behavior ?

Specified parameters are pretty tough for my laptop.
Results for the first command from the bug report:

$ $GOPATH/bin/scylla-bench  -workload=sequential -mode=write -replication-factor=1 -partition-count=10 -clustering-row-count=10000000 -clustering-row-size=5120 -concurrency=200 -rows-per-request=10 -max-rate=100000 -nodes 172.17.0.2
Configuration
Mode:                    write
Workload:                sequential
Timeout:                 5s
Max error number at row: unlimited
Max error number:        unlimited
Retries:
  number:                10
  min interval:          80ms
  max interval:          1s
  handler:               sb
Consistency level:       quorum
Partition count:         10
Clustering rows:         10000000
Clustering row size:     Fixed(5120)
Rows per request:        10
Page size:               1000
Concurrency:             200
Connections:             4
Maximum rate:            100000 op/s
Client compression:      true
Hdr memory consumption:  65590400 bytes

 time   ops/s  rows/s errors max    99.9th 99th   95th   90th   median mean   
 1.4s    4254   42540      0 375ms  369ms  151ms  107ms  88ms   43ms   50ms   
 2.9s    3312   33120      0 1s     998ms  940ms  247ms  206ms  28ms   71ms   
 3.9s    2246   22460      0 899ms  897ms  644ms  227ms  159ms  50ms   79ms   ^C
interrupted

 5.6s    1742   17420      0 1.6s   1.6s   1.1s   971ms  313ms  91ms   162ms  
Results
Time (avg):      5.805610906s
Total ops:       15309
Total rows:      153090
Operations/s:    +Inf
Rows/s:          +Inf
raw latency :
  max:           1.590689791s 
  99.9th:        1.078984703s 
  99th:          841.482239ms 
  95th:          242.221055ms 
  90th:          131.989503ms 
  median:        40.534015ms 
  mean:          78.013725ms
c-o fixed latency :
  max:           5.788139519s 
  99.9th:        5.788139519s 
  99th:          5.536481279s 
  95th:          4.928307199s 
  90th:          4.567597055s 
  median:        1.922039807s 
  mean:          2.334058937s

I don't even need numbers here, the fact that this command tried to burn my laptop is enough for me.