Introducing monitoring

[BentiGorlich]

• add monitoring capabilities for troubleshooting purposes. This cannot be enabled or disabled from the UI, you have to go into the .env files for this
• Add monitoring capabilities for the biggest factors: curl requests (AP), twig rendering (frontend) and of course database querying
• 2 sources can start an execution context: an incoming request and a started message
• added an admin UI for inspecting requests and their response times, with an overview grouped by route name

The main goal of this PR is to add monitoring capabilities, so we can see what our messengers are doing in the background. Additionally I think the graph in the overview (even though it is from a dev instance) shows pretty well that we have to tackle our twig rendering. My guess is that that is a big problem on larger instances, like fedia.io.

Some screenshots:

[BentiGorlich]

Just a warning: In about 2 days the gathered stats take up about 16GB on my relatively small server that mainly gets a lot of AP requests, but not that much user requests

table_name	table_size	indexes_size	total_size
"public"."monitoring_query"	14 GB	599 MB	14 GB
"public"."monitoring_twig_render"	1338 MB	294 MB	1633 MB
"public"."monitoring_execution_context"	345 MB	52 MB	396 MB
"public"."monitoring_curl_request"	14 MB	4792 kB	18 MB

[blued-gear]

Maybe the stats then should get their own DB connection configuration. This way admins can store the data somewhere where the fill up is not much of a problem or even choose a DB which allows compression.

[melroy89]

This is really exciting !

But yeah, you definitely want this is a separate DB.. Or maybe even in a timeseries optimized DB. Especially due to the sizes.

[BentiGorlich]

I think I can bring down the size a lot if I separate the query string out to another table and only save a reference to that in the "instances" of the query. Also I want to make some things optional. At the moment it does record the parameters of each query as well, which might be overkill or just unnecessary, so I want it to be optional

[BentiGorlich]

And it is quite a lot of data, so you might not want to monitor it for that long anyway. Otherwise you just don't get through it :D

[melroy89]

And it is quite a lot of data, so you might not want to monitor it for that long anyway. Otherwise you just don't get through it :D

We need monitoring of the monitoring, that is monitoring. 😅

[BentiGorlich]

Ok, so I pushed a few changes:

• to save space in the DB:
  • make parameter storing optional via env var
  • similar queries/statements will not take up space for each call, but will be saved to a separate table and referenced by hash
• Make the overview chart also display aggregated stats based on the current filter
• Add a dropdown to switch between the total time and the mean time in the overview chart
• Add a gradient to the twig renders either based on the percentage of the total duration or the parent duration

I have it live on gehirneimer, without saving the parameters to the DB. Lets see how much space it is taking up in 2 days :)