Memory usage

[daliborfilus]

First of all, thank you for this project. It really helps in day to day crystalling :-)

I however experience very high RAM usage of the crystalline binary.
Using vs code + crystal extension, using crystalline instead of scry.
I have two projects opened, each one very small (< 5k lines of my code, around 34k lines of code including everything in lib/).
Every day, the memory usage is above 3 GB. Currently it's 6.9 GB RES.

Isn't it a little too much? Can be something done about it? Can I do something to help?

I experience this currently with downloaded binary from release v0.3.0 for linux, even though I've used custom builds in the past with the same issue.

[elbywan]

Hey @daliborfilus,

Every day, the memory usage is above 3 GB. Currently it's 6.9 GB RES.
Isn't it a little too much? Can be something done about it? Can I do something to help?

This issue bothers me a lot, but I'm afraid there is nothing we can do about it.

Crystalline instantiate and use the crystal compiler in a separate thread, which allocates a bunch of memory every time it performs a static analysis of the project (depending on the code size - more than 1GB).

The high memory usage is caused by the fact that the garbage collector used by crystal struggles to cope with large allocations like this.

[daliborfilus]

That's unfortunate. Btw I noticed now the vscode still highlights some typo as undefined method (fixed for more than 30 min) and crystalline process still had 6.9 GB resident usage and process time was still the same - maybe the extension got stuck on something. The memory seems to never go down, even for hours, so the GC maybe seems to think the memory is still referenced from somewhere...? When I killed the process, vscode re-launched it and the error went away, so I don't know if it was vscode who was stuck, or crystalline who got overwhelmed.

Anyway, I'll keep watching the process and "refresh it" from some time to time :-)

I'll close the issue because you cannot directly control/fix this.

[elbywan]

For the record the GC behaviour can be checked by running the following code that compiles itself. (no crystalline involved)

Heap memory should grow very high (bdwgc allocates large amounts of virtual memory and does not return it to the os - depending on the compilation flags), and real memory should stabilize at some point around 6GB to 8GB (at least it does on my computer).

require "benchmark"
require "compiler/crystal/**"

def code_analysis(sources : Array(Crystal::Compiler::Source))
  dev_null = File.open(File::NULL, "w")
  compiler = Crystal::Compiler.new
  compiler.no_codegen = true
  compiler.color = false
  compiler.no_cleanup = true
  compiler.wants_doc = true
  compiler.stdout = dev_null
  compiler.stderr = dev_null
  result = compiler.compile(sources, "")

  raise result if result.is_a? Exception

  result
ensure
  dev_null.try &.close
end

file = File.new(__FILE__)
sources = [
  Crystal::Compiler::Source.new(file.path, file.gets_to_end),
]
file.close

# Compiles itself over and over
i = 0
loop do
  puts "iteration: #{i += 1}"
  puts Benchmark.measure {
    result = code_analysis(sources)
  }
  puts GC.prof_stats
end

And to check whether the memory is freed when the program is not doing anything: (hint: it isn't 😉)

# Update the program above with these loops instead of the infinite compilation:
i = 0
result = nil

# Compiles itself 10 times
loop do
  break if i >= 10
  puts "iteration: #{i += 1}"
  puts Benchmark.measure {
    result = code_analysis(sources)
  }
  puts GC.prof_stats
end

# Force GC every 5 seconds while doing nothing
loop do
  sleep 5.seconds
  GC.collect
  puts GC.prof_stats
end

[petr-fischer]

Is this GC behavior reported to the Crystal devs?

[elbywan]

@petr-fischer I don't think that they care about this, this is standard GC behaviour and they know that the compiler can take multiple GB of memory when compiling programs. In general the crystal compiler is killed after completion since it is a one-shot process, and so this does not involve the GC.

[daliborfilus]

Recently crystalline started to use more than 8 GB of RAM for me, just after few minutes. I couldn't do anything and had to wait for OOM to kick in several times after all of my 32 gigs of RAM were consumed. I tried to endure it and kill crystalline in the background every ten minutes... but I gave up. I had to disable crystalline for my projects because of this. It's very unfortunate.
I don't know if it started to be such a problem for me because of any specific reason (crystal update, app libs update, heavy use of macros or something similar), or if it started just because I started to use vscode+crystalline far more often in that period of time.
The memory usage was always in gigabytes for me before, so it isn't anything new however.

[petr-fischer]

@elbywan What about to run crystaline "compilations" in a temporary (one-time) external processes (outside of the main crystaline process)? Is it possible?

[elbywan]

@petr-fischer Not possible, the compiler output (ast/types…) is not serializable.

[carlhoerberg]

have you tried starting crystalline with the env var GC_UNMAP_THRESHOLD=1 set? that will tell boehm to aggressively release memory. https://github.com/ivmai/bdwgc/blob/master/doc/README.environment#L150-L153

[petr-fischer]

@carlhoerberg It definitely seems better. Within 5 minutes of quick testing I get to just under 4GB anyway, but it looks like RAM usage doesn't go up as aggressively as before. Thanks so much for this tip. Where is the best place to put the variable? Make a custom script for crystaline to set it up? Some better place?

[petr-fischer]

Maybe @elbywan can incorporate some GC_* ENV vars tuning into the solution? This would certainly be good for all users...

[elbywan]

@petr-fischer @carlhoerberg I already extensively tried a while ago different combinations of GC related environment variables and flags including GC_UNMAP_THRESHOLD and did not observe any notable improvements unfortunately.