Description
- asyncpg version: 0.18.3
- PostgreSQL version: 9.6
- Do you use a PostgreSQL SaaS?: no
- Python version: 3.7
- Platform: Ubuntu 18.04.2 LTS
- Do you use pgbouncer?: no
- Did you install asyncpg with pip?: yes
- Can the issue be reproduced under both asyncio and
uvloop?: not related
_StatementCache on_remove callback is called for expired entries (1) and for least recently used entries when the cache is full (2), but not when clear() method is called (3).
Connection sets own _maybe_gc_stmt method (4) as a callback for _StatementCache. The method is used to mark non-referenced PreparedStatementState objects as closed and put them in _stmts_to_close set to schedule to close them later (they will be closed on the server side when _cleanup_stmts is called (5)).
As noted above, the callback is not called when the cache is cleared by _StatementCache.clear method, therefore the statements will never be closed on the postgres side (“closed” in the meaning of the Close frontend protocol message) and can no longer be used by asyncpg. This leads to memory leak on the server side until the connection is closed.
There are two Connection methods that used to drop the cache:_drop_local_statement_cache and _drop_global_statement_cache (6). Both of them (directly or indirectly) call _StatementCache.clear. Some Connection methods (set_type_codec, reset_type_codec, set_builtin_type_codec, reload_schema_state) drop the cache unconditionally, some methods do so if the database schema has been changed (_do_execute → (execute, executemany, fetch, fetchval, fetchrow)). In any case, each cache drop may increase the number of unreachable named prepared statements that eat postgres memory.
Is this a bug or an expected behavior?