debug memory leak in wave

[matthiasdiener]

⚠️ Not meant for merging ⚠️

cc #839

[matthiasdiener]

Impact of different workarounds on memory consumption of wave:

Workaround	Mempool total	Overhead/step	Stable after
None (baseline)	14.6 MByte	---	Step 1,500
gc.collect() every 1 step (most aggressive)	2.0 MByte	0.02 s	Step 1
gc.collect() every 10 steps (current production)	4.3 MByte	2 ms	Step 10
gc.set_threshold(1,1,1)	6.2 MByte	0.02 s	Step 32
gc.collect(1) every 1 step	2.1 MByte	0.07 ms	Step 3
gc.collect(1) every 10 steps	9.2 MByte	0.006 ms	Step 15,000
gc.collect(0) every 1 step	10.5 MByte	0.07 ms	Step 800
gc.collect(0) every 10 steps	14.0 MByte	0.006 ms	Step 7,500
gc.collect(2) every 1 step	2.0 MByte	0.02 s	Step 1
gc.collect(2) every 10 steps	4.3 MByte	2 ms	Step 10
mempool.free_held() every 1 step	2 - 12 MByte	0.01 ms	Step 1
gc.disable()	20,000 MByte	---	---

[matthiasdiener]

Every time step, approx. 650 objects are garbage collected with the code in this PR. A typedump of those objects is in https://gist.github.com/matthiasdiener/33956f42bbd60321c75ad2ed62ea2fc6 . A few data structures contain references to PooledSVM allocations (see e.g. line 25 in that file). Are these the objects we should focus on?

[inducer]

Possibly? They look like somewhat plausible suspects for keeping an array alive.

[matthiasdiener]

Here is a reference graph with the objects that are garbage collected on a timestep (made with inducer/pytools#177, slightly modified to limit the number of characters shown per string).

[inducer]

Do the arrows in the graph point from referred to referent? (i.e. the opposite of the 'refers-to') graph?

[matthiasdiener]

Do the arrows in the graph point from referred to referent?

Yes, this is true. I'll switch inducer/pytools#177 to point the edges in the other direction.

[matthiasdiener]

Closing, workarounds (including gc.freeze) have been integrated in the drivers.