Memory consumption - OOM

[MichaelZhasil]

Describe the bug
With any amount of assigned memory there is always OOM and restart after a while.
We have pgwatch running on several environments number of monitored databases vary from 10 to 100.
Currently we are running pgwatch on dedicated node with 54Gi of RAM assigned.
pgwatch start's working on about 1% of assigned capacity but within 10 minutes it hops up several times each hop consuming around 20% more of RAM until it fails on OOM and is restarted by K8S.
We switched to latest version 3.3.0 but the problem prevails

To Reproduce
Steps to reproduce the behavior:

1. Have pgwatch running
2. Monitor tens of databases
3. Wait for a while - depends on assigned memory
4. OOM Killed
5. No visible error in pgwatch logs

Expected behavior
Memory consumption stops somewhere. / Some memory is freed

Desktop (please complete the following information):

• Image version 3.3.0

Additional context
We are running full metrics preset, with only database sink.

[pashagolub]

Hello @MichaelZhasil! Thanks for the report! I believe this issues is fixed by #767.

Do you have an opportunity to build and run patched version?

[michael-zhasil-pfx]

Unfortunately at the moment we are using image cybertec image from docker hub. But I could try cobble something together

[ondar]

It was just released as a image v3.4.0, no? So no need to build.

[michael-zhasil-pfx]

Sorry, didn't notice that. Unfortunately the issue is still there.
I am not sure if it's a clue or another issue but I've also noticed, that we have gaps in measurements. There is like 10 minute gap for all databases in cpu_load table (should run every minute) and it's quite regular. Also I've got "measurements written" log every minute, but with the load there I think it should definitively be on the preconfigured 250ms

[pashagolub]

Also I've got "measurements written" log every minute, but with the load there I think it should definitively be on the preconfigured 250ms

Would you please explain what do you mean?

[michael-zhasil-pfx]

I might misinterpreted things. I mean this option
https://github.com/cybertec-postgresql/pgwatch/blob/master/internal/sinks/cmdopts.go#L8
I at least thought it's delay between flush functions
https://github.com/cybertec-postgresql/pgwatch/blob/master/internal/sinks/postgres.go#L226

[pashagolub]

Yes, you're correct. This options control the cache of the sink

pgwatch/internal/sinks/postgres.go

Line 198 in 861c882

cacheTimeout := pgw.opts.BatchingDelay

But every measurement should be there anyway, losses are not acceptable.

So if I will run 10 monitored sources with full preset I should notice this, right? Both OOM and cpu measurements lost.

[pashagolub]

Would you please run for 10-30mins this script and provide me with results.

while true; do
  ts=$(date +%F_%T)
  rss=$(docker exec pgwatch sh -c "grep VmRSS /proc/1/status | awk '{print \$2}'") 
  echo "$ts VmRSS: ${rss} KB"
  sleep 30
done | tee -a mem_log.txt

I want to see how things are really bad and if your environment produces some additional data for investigations.

Thanks in advance!

[pashagolub]

Also would you please show me how you run pgwatch? Which sinks are enabled? At what value OOM-killer restarts pgwatch?

[michael-zhasil-pfx]

There are arguments for pgwatch, we used previously also prometheus sink, but it generated errors with duplicitous labels, which thinking about it might be still atached to the same issue as prometheus worked fine with lower loads.

  - "/pgwatch/pgwatch"
  - "--sources=/pgwatch/sources/sources.yaml"
  - "--metrics=/pgwatch/metrics/metrics.yaml"
  - '--sink=postgres://$DB_USER:$DB_PASSWORD@$DB_HOST:5432/$DB_NAME'

We have currently request and limit set to 54Gi of memory, so it's killed when it reaches over this value, or generally while it jumps above request value I never saw it to go to limit.

mem_log.txt

[pashagolub]

Thanks a lot! I see the trend. I will get back to you. I need some amount of time to refactor a good part of the architecture for POC.

[pashagolub]

Would you please try the patched image cybertecpostgresql/pgwatch:fix764. It behaves stable in my tests.