Memory leak in long-lived connection pool

[seanlinsley]

In the image below, memory usage is stable with prune_db_connections as described in deadpool-rs/deadpool#273, plus malloc_trim(0). Then disabling prune_db_connections shows a clear memory leak.

Heaptrack shows two leaks, both around BytesMut usage:

Related discussions online:

• https://stackoverflow.com/questions/73725299
• docs: unclear how to free memory from a bytesmut tokio-rs/bytes#634
• Memory Leak on Actix 3.2.0 actix/actix-web#1780
• Memory leak on high number of concurrent connections hyperium/hyper#1790
• Proxy memory leak over HTTP 1 linkerd/linkerd2#11012

Possible debugging steps:

• add code to track every message sent/received, where the buffer size has grown since the last message (might require a fork of the bytes crate)
• write a reproduction script that's similar to the production workload (RDS, binary COPY FROM, queries that cause unique constraint error)

Possible fixes:

• revert ffd7245
  • this would likely fix the leak if it's actually a bug in BytesMut usage and not something like closed connections never being freed. and it'd be a good opportunity to confirm/deny the performance difference that motivated the change originally

[sfackler]

Do you have a self contained reproduction?

[seanlinsley]

I've tried to write a reproduction script, but it doesn't leak memory. If there is a bug in BytesMut usage it might be around the exact handling of messages sent/received to a remote database, using both regular queries (with heavy use of array binds) and binary COPY FROM, and with some queries resulting in unique constraint errors.

[inzanez]

@seanlinsley is there any update on this? I am seeing similar things (queries with many parameters) increasing memory usage quite heavily.
Is it already clear where that is coming from?

[seanlinsley]

We haven't been able to reproduce this outside of our production environment, even after building a load tester with similar queries. For now it's been sufficient to prune old connections as described in deadpool-rs/deadpool#273

[inzanez]

Ok,...well, I am trying to reproduce this right now in a minimal example. Hopefully I will manage,...

[inzanez]

I assume you are using the deadpool / pgsql combination in a webserver, correct? If so, what is the underlying web framework?

[inzanez]

@seanlinsley can you give some details about your production environment? Are you using a web framework? Could it be that the executable is running in a container environment like Kubernetes, Openshift etc.? Does that container environment enforce memory limits?...

[inzanez]

@sfackler my tests suggest that frequently used connections have ~100MByte of memory reserved at rest. So they pretty much never free that memory. Having a connection pool of 25 clients could take up ~2.5GByte of memory which seems to be quite a lot.
I have not yet managed to locate the exact region of where that memory allocation comes from though. I am still trying but maybe you already have a clue where I might have to look?...

[inzanez]

It seems that the memory is allocated mostly in the read- and write-buffer of the

pub struct Connection<S, T> {
    stream: Framed<MaybeTlsStream<S, T>, PostgresCode
    ...
```,

[Sagebati]

I think I also have this problem

The memory is never released and can go as hight until OOM my server is ~8Go