Very high memory usage on the Rendering Level stage

[ateijelo]

I'm rendering a world that's less than 200Mb. With 4 threads, Papyrus is rendering at 6~7 cps. It takes 2 hours to render ~50000 chunks. That could be better, but I don't mind waiting. During this phase memory usage remains controlled (less than 1GB)

However, when Papyrus gets to the point where it says:

Rendering Level 19 with source coordinates X -80 to 122, Z -196 to 58
202 of 256 lines render @ zoom level 19

memory usage skyrockets to fill all my 16Gb of RAM and then starts swapping.

On htop:

VIRT   RES   SHR  S CPU% MEM%   TIME+  Command
24.5G 13.9G  5956 D 36.6 89.7  8h31:49 ./PapyrusCs ...

that's ~25Gb of virtual memory, 13.9 of which are resident (because my desktop is loaded)
My RAM and Swap are to the brim:

Mem[|||||||||||||||||||||||||||14.9G/15.5G]   Tasks: 170, 525 thr; 2 running
Swp[|||||||||||||||||||||||||||12.4G/13.7G]   Load average: 12.15 11.91 9.37 

It won't be long until it gets oom-killed.

Is it keeping every rendered image in RAM somehow?

[ateijelo]

And... it's gone:

Rendering Level 19 with source coordinates X -80 to 122, Z -196 to 58
202 of 256 lines render @ zoom level 19
Rendering Level 18 with source coordinates X -40 to 62, Z -98 to 30
46 of 128 lines render @ zoom level 18
Killed

[ateijelo]

If you want to troubleshoot on your end, I can upload my world to Google Drive or some other place.

[ateijelo]

I've made some progress with this issue:

• I found that 0.3.6 and 0.3.7 generated corrupted images, they had a brownish sheen with a weird whitish gradient towards the bottom
• Going back to 0.3.5 fixed that and increased the generation speed back to 60~70 cps, which means my 50000 chunk world takes about 8 minutes to generate the first tile layer.
• I downloaded the source code and commented out the render zoom levels line in Program.cs:

-            strat.RenderZoomLevels();
+            //strat.RenderZoomLevels();

• Compiled with dotnet build, it solved the high memory consumption
• Since that change means there are no zoom levels below 20, and I don't know a lot of C#, I wrote this Python3 script to generate the missing levels.
• This takes a couple of minutes for my world and doesn't consume any significant amount of memory.
  So, this unblocks me for the moment to keep generating my map.
  Thanks a lot for the work put in PapyrusCS. If you want my world to test, let me know.