debug.traceTransaction is a memory hog

[kumarde]

System information

Geth version: 1.7.3 stable
OS & Version: Linux

I'm trying to run a moderate number of transaction traces (10K+) using debug.traceTransaction on a fairly beefy machine (128GB of RAM, 32 cores), but I can only get through about ~1K before it uses all the RAM AND all the swap and gets killed by the OS.

Is there any intuition as to why this is happening? Is this "bug" worthy?

[karalabe]

The entire tracing is rewritten on master already. We're also introducing much more capable JavaScript tracing functionality, which allows users to write their own tracers as well as use a few ones built in (#15516).

That being said, tracing a transaction and creating a dump of all opcodes will always be humongous since some transactions simply run that many VM instructions. That's why our suggestion is to either use the built in custom tracers, or to write your own that only collects what you need, and doesn't export everything and the kitchen sink. These custom tracers run in the node, so there's no communication overhead and the memory overhead is only what your tracer personally needs.

[kumarde]

Interesting.

For my purpose, I need the entirety of the opcodes and all state from the VM, so on that matter I don't think I can budge.

However, it's nice that things have been rewritten, I can see how this improves memory performance. Thanks!

[karalabe]

Tracing each opcode and everything along with it won't get any less resource intensive, since the node needs to assemble everything into one gigantic trace in memory, which is enormous.

Why do you need everything? If you write your own .js tracer, that has access to everything whilst running inside the node, so you can gather and export whatever you need without pulling all the useless data too that you don't need.