v1.32.0: Metrics CPU usage / response time heavily increased after upgrading to 1.32.0 from 1.32.0-rc1

[Half-Shot]

👉 Did you upgrade to 1.32.0?
👉 Instructions for resolving associated Prometheus issues are at #9854 (comment)

Original issue report continues below:

---

Are the master metrics response times and CPU usage since upgrading. Performance doesn't feel degraded but this is clearly Not Good.

I'm was on 82fcd3cce8a5969f575c4811b524d5224dde8a751 and I am now on f98d0985d3816f23a410e2f741b2b180739e5ae7

[Half-Shot]

Suspect it could be #9829, which didn't land in the RC.

[Half-Shot]

Looks like the /metrics response is ballooning over time.

will@halfy-matrix:~$ date && curl 'http://127.0.0.1:9092/_synapse/metrics' --silent | wc -l
Tue 20 Apr 15:36:48 UTC 2021
204492
will@halfy-matrix:~$ date && curl 'http://127.0.0.1:9092/_synapse/metrics' --silent | wc -l
Tue 20 Apr 15:37:03 UTC 2021
206127
will@halfy-matrix:~$ date && curl 'http://127.0.0.1:9092/_synapse/metrics' --silent | wc -l
Tue 20 Apr 15:37:10 UTC 2021
206577
will@halfy-matrix:~$ date && curl 'http://127.0.0.1:9092/_synapse/metrics' --silent | wc -l
Tue 20 Apr 15:40:13 UTC 2021
227322

[jdreichmann]

I can confirm this issue, see the following rough tests:

[jdreichmann@iodine git]$ while true; do date && printf "master %d\t" $(curl 'http://127.0.0.1:9100/_synapse/metrics' --silent | wc -l) && printf "appservice %d\t" $(curl 'http://127.0.0.1:19300/_synapse/metrics' --silent | wc -l) && printf "fed-send %d\n" $(curl 'http:/
/127.0.0.1:19400/_synapse/metrics' --silent | wc -l); sleep 2; done                                                                     
Tue 20 Apr 2021 05:55:50 PM CEST
master 10796    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:55:53 PM CEST
master 10796    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:55:55 PM CEST
master 10797    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:55:57 PM CEST
master 10796    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:55:59 PM CEST
master 10796    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:56:02 PM CEST
master 10797    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:56:04 PM CEST
master 10797    appservice 2389 fed-send 2672         
Tue 20 Apr 2021 05:56:06 PM CEST
master 10796    appservice 2389 fed-send 2675         
Tue 20 Apr 2021 05:56:08 PM CEST
master 10796    appservice 2389 fed-send 2675         
Tue 20 Apr 2021 05:56:11 PM CEST
master 10796    appservice 2389 fed-send 2675         
Tue 20 Apr 2021 05:56:13 PM CEST
master 10797    appservice 2389 fed-send 2675
Tue 20 Apr 2021 05:56:15 PM CEST
master 10798    appservice 2389 fed-send 2675
Tue 20 Apr 2021 05:56:17 PM CEST
master 10797    appservice 2389 fed-send 2675
^C

on 1.32.0rc1

[jdreichmann@iodine git]$ while true; do date && printf "master %d\t" $(curl 'http://127.0.0.1:9100/_synapse/metrics' --silent | wc -l) && printf "appservice %d\t" $(curl 'http://127.0.0.1:19300/_synapse/metrics' --silent | wc -l) && printf "fed-send %d\n" $(curl 'http:/
/127.0.0.1:19400/_synapse/metrics' --silent | wc -l); sleep 2; done 
Tue 20 Apr 2021 05:59:03 PM CEST
master 21121    appservice 4732 fed-send 6065
Tue 20 Apr 2021 05:59:05 PM CEST
master 22454    appservice 5122 fed-send 6393
Tue 20 Apr 2021 05:59:08 PM CEST
master 24113    appservice 5537 fed-send 6693
Tue 20 Apr 2021 05:59:11 PM CEST
master 25340    appservice 5792 fed-send 6846
Tue 20 Apr 2021 05:59:14 PM CEST
master 26510    appservice 6232 fed-send 7147
Tue 20 Apr 2021 05:59:17 PM CEST
master 28667    appservice 6682 fed-send 7438
Tue 20 Apr 2021 05:59:20 PM CEST
master 30196    appservice 7072 fed-send 7597
Tue 20 Apr 2021 05:59:23 PM CEST
master 31436    appservice 7442 fed-send 7817
Tue 20 Apr 2021 05:59:25 PM CEST
master 33262    appservice 8010 fed-send 9188
Tue 20 Apr 2021 05:59:28 PM CEST
master 34374    appservice 8295 fed-send 9576
Tue 20 Apr 2021 05:59:31 PM CEST
master 36216    appservice 8885 fed-send 55662
Tue 20 Apr 2021 05:59:35 PM CEST
master 40693    appservice 10670        fed-send 77592
Tue 20 Apr 2021 05:59:38 PM CEST
master 42597    appservice 11279        fed-send 78720


Tue 20 Apr 2021 05:59:42 PM CEST
master 44737    appservice 11921        fed-send 78798
Tue 20 Apr 2021 05:59:46 PM CEST
master 46387    appservice 12326        fed-send 79341
Tue 20 Apr 2021 05:59:50 PM CEST
master 47956    appservice 12866        fed-send 80181
Tue 20 Apr 2021 05:59:54 PM CEST
master 49379    appservice 13259        fed-send 81066
Tue 20 Apr 2021 05:59:57 PM CEST
master 50797    appservice 13772        fed-send 81276
Tue 20 Apr 2021 06:00:01 PM CEST
master 52801    appservice 14408        fed-send 81444
Tue 20 Apr 2021 06:00:04 PM CEST
master 54298    appservice 14996        fed-send 82497
Tue 20 Apr 2021 06:00:08 PM CEST
master 56251    appservice 15524        fed-send 82665
Tue 20 Apr 2021 06:00:12 PM CEST
master 57086    appservice 15839        fed-send 84501
Tue 20 Apr 2021 06:00:16 PM CEST
master 59586    appservice 16646        fed-send 86526
Tue 20 Apr 2021 06:00:20 PM CEST
master 62222    appservice 17405        fed-send 87129
^C

on 1.32.0

Before the update, i had ~40K total series in prometheus, after the update, this has increased to ~2.2M total series, hogging all RAM and making prometheus unusable and crashed it:

[richvdh]

this is fixed by #9854