java.lang.OutOfMemoryError parsing `./src/cluster.c` from Redis

[jjhenkel]

I was running the following command (where ./src/cluster.c is coming from a fresh clone of Redis):

/target # bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/cluster.c

After waiting for a minute or so I get the following output (at this point memory utilization is about 12G on a 256G workstation machine---so I'm not actually out of memory; I'm also running with a very high maximum heap memory limit -Xmx20000m):

Exception in thread "main" java.lang.OutOfMemoryError
    at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
    at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
    at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2033)
    at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:454)
    at com.fasterxml.jackson.core.json.JsonGeneratorImpl.writeStringField(JsonGeneratorImpl.java:203)
    at tech.sourced.babelfish.JsonASTVisitor.lambda$visit$3(JsonASTVisitor.java:679)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:153)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:662)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIdExpression.accept(CPPASTIdExpression.java:87)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:662)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTBinaryExpression.accept(CPPASTBinaryExpression.java:163)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTExpressionStatement.accept(CPPASTExpressionStatement.java:64)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:251)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTCompoundStatement.accept(CPPASTCompoundStatement.java:84)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:251)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTCompoundStatement.accept(CPPASTCompoundStatement.java:84)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:251)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTCompoundStatement.accept(CPPASTCompoundStatement.java:84)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTIfStatement.accept(CPPASTIfStatement.java:133)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:251)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1207)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTCompoundStatement.accept(CPPASTCompoundStatement.java:84)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:268)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:294)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:851)
    at org.eclipse.cdt.internal.core.dom.parser.cpp.CPPASTFunctionDefinition.accept(CPPASTFunctionDefinition.java:188)
    at tech.sourced.babelfish.JsonASTVisitor.writeChildProperty(JsonASTVisitor.java:251)
    at tech.sourced.babelfish.JsonASTVisitor.visitChildren(JsonASTVisitor.java:327)
    at tech.sourced.babelfish.JsonASTVisitor.visitWrapper(JsonASTVisitor.java:155)
    at tech.sourced.babelfish.JsonASTVisitor.visit(JsonASTVisitor.java:1348)
    at org.eclipse.cdt.internal.core.dom.parser.ASTTranslationUnit.accept(ASTTranslationUnit.java:267)
    at tech.sourced.babelfish.TranslationUnitSerializer.serialize(TranslationUnitSerializer.java:69)
    at tech.sourced.babelfish.TranslationUnitSerializer.serialize(TranslationUnitSerializer.java:44)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:2509)
    at tech.sourced.babelfish.TranslationUnitJSONMapper.writeValue(TranslationUnitJSONMapper.java:40)
    at tech.sourced.babelfish.DriverResponse.send(DriverResponse.java:44)
    at tech.sourced.babelfish.Main.process(Main.java:53)
    at tech.sourced.babelfish.Main.main(Main.java:13)

I am able to parse other files in this repository successfully (including other C files); here are some parse attempts that all succeed:

bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/redis_init_script
bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/releasetools/01_create_tarball.sh
bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/releasetools/02_upload_tarball.sh
bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/releasetools/03_test_release.sh
bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/releasetools/04_release_hash.sh
bblfsh-cli -a localhost:9432 -l bash -m native --v2 ./utils/whatisdoing.sh
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./src/modules/gendoc.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./src/redis-trib.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/generate-command-help.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/hyperloglog/hll-err.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/hyperloglog/hll-gnuplot-graph.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/lru/test-lru.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/redis-copy.rb
bblfsh-cli -a localhost:9438 -l ruby -m native --v2 ./utils/redis-sha1.rb
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/adlist.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/adlist.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae_epoll.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae_evport.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae_kqueue.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/ae_select.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/anet.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/anet.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/aof.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/asciilogo.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/atomicvar.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/bio.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/bio.h
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/bitops.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/blocked.c
bblfsh-cli -a localhost:9440 -l c -m native --v2 ./src/childinfo.c  

It's just ./src/cluster.c that seems to cause issues.

Also, in general, I'm observing steadily increasing memory usage as I parse more and more files---I'm not entirely sure why this is? Perhaps there's a memory leak somewhere?

[juanjux]

Investigating this. The increasing memory usage could just be the JVM "warming up" and catching compiled and other stuff, could you check if it stabilises at some point or does it always increase?

[juanjux]

Hi again! Are you running some workaround for the timeouts? I'm trying to reproduce this but every time, either with bblfsh-cli, running the driver directly or using the source as a fixture for the driver integration tests, it gives me a gRPC timeout (context deadline exceeded) which, while not the best result, at least doesn't crash the driver.

[jjhenkel]

Hi! I'll try and get this in a reproducible docker image for you. My setup is non-standard as I'm trying to run bblfsh parsers in a high-throughput computing environment with very limited permissions.

[juanjux]

Sounds like computing nirvana :) Thanks for the docker image.

[jjhenkel]

Okay, here's an image: docker pull jjhenkel/ht-parser:repro.

In case this helps, here's the code I'm using. The .repro variants of the files are what that image is running.

You can run it against an updated cpp driver using the following:

docker run -it -v `pwd`/temp/cpp:/opt/drivers/cpp --rm jjhenkel/ht-parser:repro

Where $PWD/temp/cpp was just the contents of the /opt/driver directory from the cpp-driver image (built using the bblfsh-sdk).

[juanjux]

Thanks! I'll try it.

[juanjux]

I confirm that I can reproduce the OOM with your image, I'll can can continue trying to see what is eating so much memory from here.

[dennwc]

Have the same issue with another random .cpp file.

The problem for me also is that the driver doesn't return an error to the client and it's just hanging until the deadline is reached.

[juanjux]

Update:

This PR created thanks to a finding of @dennwc that the output was being held in a buffer entirely instead of streaming fixes the OOM in Java, but doesn't really fix the problem with that file, just moved the memory usage to the SDK/native part (which at least with my 16GB of RAM will eventually awake the kernel OOM).

The memory usage (and time to parse/crash) still looks unproporcionate to the size of the file (because 5000 lines is big but not ridiculously so). So I won't close this because we still need to investigate what's happening (maybe the driver enters into some recursive trap while trying to parse the file).

[dennwc]

@juanjux Since you mentioned that OOM is now is caused by the driver server (Go), it seems like we are hitting an infinite recursion somewhere.

This will explain why driver server crashes now - both Java and Go stream output, and Go is building an in-memory AST directly from an input. So Java spins in an infinite loop sending the same data over and over again, while Go tries to keep up and build more and more AST layers in memory.

[juanjux]

Yeah, that's the only explanation I see, there is not way that file could generate a 16GB+ UAST, I'll debug further once I finish polishing some things in the C# driver.