A memory leak that involves SQLiteRows but isn't visible by goref

[gavraz]

Hi,
We're facing a significant memory leak in our product (v1.14, running on Linux) and have been actively investigating its cause.
Our pprof heap profile points to the SQLiteRows.Columns function and related calls on SQLiteStmt and SQLiteConn as the source. Based on the code:

1. SQLiteRows holds a reference to SQLiteStmt, which in turn references SQLiteConn.
2. Our analysis of the database/sql package suggests that the SQLiteRows instance is the only Go object maintaining a reference to the memory allocated by Columns() in our specific use case.
   Therefore, while pprof indicates a leak of SQLiteRows references, our code inspection and usage patterns don't immediately reveal any obvious leaks of these objects. We've confirmed that the memory usage increases steadily over time, not in sudden spikes, and we've ruled out goroutine leaks as a contributing factor.

Interestingly, a core dump analyzed with goref shows only around 130MB of in-use data, which doesn't seem related to sqlite3. This contrasts sharply with the heap profile, which indicates several gigabytes of Go-allocated memory. This discrepancy leads us to suspect that C allocations might be contributing to the observed Go heap growth (potentially making the heap profile appear inaccurate in this specific scenario).

We've considered potential issues with Go runtime or finalizer changes, but your current implementation appears correct. We also tested with the latest Go 1.23, but the problem persists.

We are currently investigating a potentially relevant issue discussed in this PR on a fork, which might offer an explanation for this behavior.

Unfortunately, we haven't been able to reproduce the issue in a controlled environment yet.

Any guidance or insights you could provide on this investigation would be greatly appreciated.

Thank you.

[mattn]

Thank you! So rqlite#23 would be help to fix this?

[gavraz]

Hi @mattn

We finally figured it out, and it turns out it wasn't related to the PR we previously discussed.
I'm writing this here as I believe it could be the underlying cause of similar leak reports related to this package.
The long story short is Go's runtime uses a single goroutine to execute finalizers. If this goroutine becomes blocked, finalizers queue up, leading to a potential leak.
Given the recent addition of finalizers for rows, statements, and connections in your package, it's now susceptible to this issue (as are other packages).
Specifically, SqliteRows.cls being true for queries means the finalizer is not manually removed, relying on the runtime to execute it. Therefore, references to rows (and their associated statements and connections) are leaked.

While many write about the risks involving finalizers, we haven't found explicit mention of this blocking behavior implications. Notably, the runtime doesn't provide any indication when a finalizer goroutine is blocked for extended periods (potentially indefinitely). We intend to engage the community in this issue to come up with a proposal.

I leave it open in case you have more questions but feel free to close it.

Thank you.

[itizir]

Hmm, that's interesting... Curious as to what's special in your usecase for it to behave this way: is the load particularly high? At what kind of rates are those Rows created? Is the program very busy doing other things?
You are careful about always calling Rows.Close in a timely way, right?

[gavraz]

(For those interested, here is the discussion with the go team: https://groups.google.com/g/golang-nuts/c/uL68-fxg2K4.)

@itizir
There is high load for DB reads. That for itself is not a problem.
Yes, we are careful to close the rows but notice that rows.cls=true so the call to rows.Close will not remove the finalizer. This implies the finalizer will be queued. But, since the finalizer routine is blocked by another package's finalizer, the finalizer queue will grow indefinitely and causing a leak of these objects.

[itizir]

@gavraz, great read! Thanks a lot for the link.

Oh, and for people who just want the tl;dr without reading the full discussion: the offending package with a blocking finalizer is https://github.com/hirochachacha/go-smb2

[gavraz]

A full explanation of the investigation and result is available on our recent blog post Leak and Seek: A Go Runtime Mystery.
Note you may use our open-source tool to detect blocked finalizers: https://github.com/cyolosecurity/finq.

[mattn]

@gavraz thank your great working.