[1.0 launch] Scale test RC

[kfswain]

Part of #1454

[kfswain]

Found a CPU hotspot; this is a before run on a simple 16 core machine, 10k QPS, 10k char prompt:

benchmark_http_client.latency_2xx (1802004 samples)
  min: 0s 035ms 358us | mean: 5s 082ms 915us | max: 20s 279ms 459us | pstdev: 2s 714ms 020us

  Percentile  Count       Value          
  0.5         901069      5s 484ms 838us 
  0.75        1351592     6s 955ms 991us 
  0.8         1441665     7s 272ms 660us 
  0.9         1621842     8s 128ms 036us 
  0.95        1711927     8s 987ms 869us 
  0.990625    1785111     11s 299ms 979us
  0.99902344  1800246     13s 366ms 722us

Counter                                 Value       Per second
benchmark.http_2xx                      1802004     6006.69
benchmark.http_4xx                      82          0.27
benchmark.http_5xx                      4370        14.57
benchmark.pool_connection_failure       492         1.64
benchmark.pool_overflow                 1060188     3533.96
benchmark.stream_resets                 86261       287.54
cluster_manager.cluster_added           100         0.33
default.total_match_count               100         0.33
http2.rx_reset                          810         2.70
membership_change                       100         0.33
runtime.load_success                    1           0.00
runtime.override_dir_not_exists         1           0.00
upstream_cx_close_notify                2626        8.75
upstream_cx_connect_fail                493         1.64
upstream_cx_destroy                     3466        11.55
upstream_cx_destroy_local               2625        8.75
upstream_cx_destroy_remote              841         2.80
upstream_cx_destroy_remote_with_active_rq348         1.16
upstream_cx_destroy_with_active_rq      348         1.16
upstream_cx_http2_total                 4126        13.75
upstream_cx_protocol_error              2405        8.02
upstream_cx_rx_bytes_total              835472208   2784910.61
upstream_cx_total                       4126        13.75
upstream_cx_tx_bytes_total              38450511435 128168521.19
upstream_rq_pending_failure_eject       6959        23.20
upstream_rq_pending_overflow            1060188     3533.96
upstream_rq_pending_total               5028        16.76
upstream_rq_total                       1939320     6464.41

Attached is the CPU profile during a run, note the CPU time spent on sprintf

[kfswain]

This (and some lock contention fixes) were pushed in #1523

[kfswain]

Giving a final run (*without the prefix-cache-scorer) of:

the .5, .75, .95, .99 percentiles are all comaprable to the 0.5.1 image, the .999 latency is an outlier but within acceptable bounds given the .99 latency

benchmark_http_client.latency_2xx (2997852 samples)
  min: 0s 035ms 493us | mean: 0s 048ms 435us | max: 2s 629ms 828us | pstdev: 0s 042ms 602us

  Percentile  Count       Value          
  0.5         1498944     0s 044ms 883us 
  0.75        2248560     0s 047ms 669us 
  0.8         2398329     0s 048ms 574us 
  0.9         2698095     0s 052ms 035us 
  0.95        2847989     0s 057ms 430us 
  0.990625    2969748     0s 099ms 266us 
  0.99902344  2994925     0s 623ms 673us 


Counter                                 Value       Per second
benchmark.http_2xx                      2997852     9992.84
benchmark.pool_overflow                 1135        3.78
benchmark.stream_resets                 382         1.27
cluster_manager.cluster_added           100         0.33
default.total_match_count               100         0.33
membership_change                       100         0.33
runtime.load_success                    1           0.00
runtime.override_dir_not_exists         1           0.00
upstream_cx_destroy                     48          0.16
upstream_cx_destroy_remote              48          0.16
upstream_cx_destroy_remote_with_active_rq48          0.16
upstream_cx_destroy_with_active_rq      48          0.16
upstream_cx_http2_total                 150         0.50
upstream_cx_rx_bytes_total              1385202990  4617341.11
upstream_cx_total                       150         0.50
upstream_cx_tx_bytes_total              60756780080 202522504.20
upstream_rq_pending_failure_eject       382         1.27
upstream_rq_pending_overflow            1135        3.78
upstream_rq_pending_total               150         0.50
upstream_rq_total                       2998865     9996.21

[kfswain]

The final scale test with prefix scoring gives:

benchmark_http_client.latency_2xx (2994784 samples)
  min: 0s 035ms 483us | mean: 0s 286ms 553us | max: 4s 541ms 906us | pstdev: 0s 425ms 462us

  Percentile  Count       Value          
  0.5         1497396     0s 060ms 203us 
  0.75        2246109     0s 351ms 584us 
  0.8         2395836     0s 473ms 366us 
  0.9         2695309     0s 889ms 716us 
  0.95        2845048     1s 256ms 259us 
  0.990625    2966709     1s 936ms 457us 
  0.99902344  2991860     2s 579ms 890us 

Counter                                 Value       Per second
benchmark.http_2xx                      2994784     9982.61
benchmark.pool_overflow                 1802        6.01
benchmark.stream_resets                 57          0.19
cluster_manager.cluster_added           100         0.33
default.total_match_count               100         0.33
membership_change                       100         0.33
runtime.load_success                    1           0.00
runtime.override_dir_not_exists         1           0.00
upstream_cx_destroy                     4           0.01
upstream_cx_destroy_remote              4           0.01
upstream_cx_destroy_remote_with_active_rq4           0.01
upstream_cx_destroy_with_active_rq      4           0.01
upstream_cx_http2_total                 292         0.97
upstream_cx_rx_bytes_total              1383967876  4613226.07
upstream_cx_total                       292         0.97
upstream_cx_tx_bytes_total              60743525010 202478408.68
upstream_rq_pending_failure_eject       57          0.19
upstream_rq_pending_overflow            1802        6.01
upstream_rq_pending_total               315         1.05
upstream_rq_total                       2998198     9993.99

The lowered performance is expected given the more computationally expensive nature of prefix cache aware routing.

There are improvements we can make long term to the data sharing model to remove these hotspots for all plugins, but those would be in future updates.

[kfswain]

Considering this resolved

[kfswain]

Usage peaks on default v1 algo:

• 10k QPS
• 10k prompt length (all random chars)