Memory management in TASO

[lmondada]

TASO is quite RAM-heavy with a couple of data structures that can grow quite large + multithreading might multiply this.

In the first beta launch RAM will be limited at 16GB. It would probably be wise to manage our hashtables etc to aim for a constant memory footprint, somewhere close to the total amount available. This would avoid running out of memory and as a bonus would reduce the memory allocation/de-allocation overheads.

This discussion is meant as a place to coordinate all memory-related tasks and improvement ideas for TASO.

[aborgna-q]

We may be able to use peak_alloc to dynamically manage the priority queue size. Although I'd prefer to avoid overwriting the global allocator for this.

[aborgna-q]

A nice observation is that, at least for circuit splitting, the memory usage remains similar independently of the number of nodes (we have more queues, but the circuit sizes are smaller) so tweaking the RAM usage via queue resizing can be done independently from growing the parallelism.

[aborgna-q]

There's been multiple memory optimisations since this discussion;

• Limiting the amount of in-flight circuits in the process channels.
• Making sure we're not keeping stale data on the queues.
• For the badger-optimiser bin, using the jemalloc allocator.

Currently we can run small-ish circuits with <1GB of RAM, and the memory consumed only scales with the size of the input and not that much with the amount of threads.