shellcheck v0.9.0 will keep eating ram until a segfault or a system lockup

[sonic2kk]

For bugs

• Rule Id (if any, e.g. SC1000): N/A
• My shellcheck version (shellcheck --version or "online"): v.0.9.0
• The rule's wiki page does not already cover this (e.g. https://shellcheck.net/wiki/SC2086)
• I tried on https://www.shellcheck.net/ and verified that this is still a problem on the latest commit
  • On this subject, I believe the latest commit at the time of writing is the v0.9.0 release commit. Apologies if not and I will gladly test a more bleeding-edge build
  • I cannot try online as the script I am using is too large to work online

Here's a snippet or screenshot that shows the problem:

This issue is specific to one Bash project that I maintain called SteamTinkerLaunch. With shellcheck 0.8.0, there were no issues. When I updated to shellcheck 0.9.0, I started encountering a problem.

Running shellcheck v.0.9.0 on SteamTinkerLaunch (a 22,000+ lines Bash script), shellcheck will keep using ram until a segfault or a system crash. I am running a PC with 32GB RAM and before a system lockup, KSysGuard said 30GB RAM was being used.

I have tested shelllcheck from the official Arch repos (0.9.0-3 at time of writing), and the GitHub 0.9.0 release.

I have confirmed that with shellcheck 0.8.0, this issue is not present. I am currently using 0.8.0 to check SteamTinkerLaunch.

I have a couple of very small test scripts that I used to prototype ideas, there are only about 20 lines long or so. I tested shellcheck 0.8.0 and 0.9.0 on these scripts and they both worked without issue (in fact, 0.9.0 seemed marginally faster). These scripts don't have any fancy code, I just tested on them to see if they would also reproduce the issue and they did not.

System Information:

• Distribution: Arch Linux
• Desktop: KDE Plasma Wayland 5.26.4
• Kernel: 6.1.1 (Linux Zen Kernel)
• CPU: Ryzen 7 3700X
• RAM: 32GB

---

I realise this is a less-than-helpful issue to open with the information I have at present. I am happy to help diagnose the problem further however I can, and in the meantime I am equally happy to keep using 0.8.0 for SteamTinkerLaunch. I don't have any other information to provide at the moment but I will provide any required information to try and fix this issue, if it is even a Shellcheck issue.

Thanks for creating Shellcheck :-)

[Cobalt-Inferno]

Tried on your project with a similar desktop configuration, and it also began consuming large amounts of RAM.

[austin987]

If you can build shellcheck from source, you can use 'git bisect' (https://git-scm.com/docs/git-bisect) to identify what change caused the regression.

You could also try using delta (https://github.com/mpflanzer/delta) to reduce the problematic script down to a minimal testcase.

[sonic2kk]

Ah! I have never used bisect before but I am vaguely aware of it. I'll give that a try now!

[sonic2kk]

Unfortunately I'm having many difficulties building shellcheck from source, there are lots of errors from Cabal about files missing from base. The Arch wiki page for using Haskell on Arch seems very involved, and to an outsider with no experience using Haskell I will admit I am a bit lost. The suggest commands did not work and even trying to manually install various Haskell packages did not seem to work unfortunately.

I am sorry to say I may not be able to bisect the issue, but it seems like it can be reproduced, so I am happy to stay involved and try to help out in any other ways that I can.

[koalaman]

This is undoubtedly due to ShellCheck's new data flow analysis engine. It takes great care to be acceptably fast even for larger scripts, but 22,000 lines is something else. Given the amount of optimization work that went into it in the first place, I'm guessing the most realistic solution is a flag that disables DFA and either skips those checks or reverts to the old, naive algorithm.

[austin987]

FWIW, when using shellcheck-0.9.0 with winetricks (23506 lines), I see that it takes longer than it did with 0.8.0 (15-18s vs 6s), but it does complete without exhausting memory.

git bisect for my testcase shows:

f77a545282f02010a0c52b66bd6ee5e860c2b314 is the first bad commit
commit f77a545282f02010a0c52b66bd6ee5e860c2b314
Author: Vidar Holen <vidar@vidarholen.net>
Date:   Tue Jul 19 14:23:27 2022 -0700

    Control Flow Graph / Data Flow Analysis support

[ale5000-git]

Just an idea, but maybe shellcheck can forcefully call the garbage collector from time to time.

[douzzer]

This is a hard upgrade blocker for us. We use shellcheck to check configure scripts. In 0.9.0, this consumes over 100 GB of RAM and takes over 5 minutes. In 0.8.0 it uses 625 MB and takes 20 seconds on the same configure script. And 0.9.0 didn't identify any new defects relative to 0.8.0.

So for our use case, we will need to be able to select the old strategy (disabling the new data flow analysis engine).

[akwala]

This is undoubtedly due to ShellCheck's new data flow analysis engine. It takes great care to be acceptably fast even for larger scripts, but 22,000 lines is something else. Given the amount of optimization work that went into it in the first place, I'm guessing the most realistic solution is a flag that disables DFA and either skips those checks or reverts to the old, naive algorithm.

[shellcheck 0.9.0]
Happens to me with a script of 1218 lines, in VSCodium, on a machine w/ 16GB of RAM. Eats up the memory within a couple of minutes.

Operating System: Kubuntu 22.04
KDE Plasma Version: 5.24.7
KDE Frameworks Version: 5.98.0
Qt Version: 5.15.3
Kernel Version: 5.15.0-76-generic (64-bit)

[magicus]

@akwala 1218 lines sounds like the shortest reproducer for this issue so far. I know this is much later, but do you happen to have a copy of that script that you can share here?

[koalaman]

With d80fdfa you can specify # shellcheck extended-analysis=false to skip DFA in particularly large files (also works in rc files, and via --extended-analysis=false.