memory-usage-tests fail with GHC 9.2

[robx]

With GHC 9.2, CI runs of the memory-usage-tests suite fail by exhausting the heap:

>>> **/memory-usage-tests -j1 --hide-successes 
memory-usage-tests: Heap exhausted;
memory-usage-tests: Current maximum heap size is 8388608 bytes (8 MB).
memory-usage-tests: Use `+RTS -M<size>' to increase it.
<<< **/memory-usage-tests -j1 --hide-successes  (33/212 sec, 251)
<<< /home/runner/work/cabal/cabal/dist-newstyle-validate-ghc-9.2.1/build/x86_64-linux/ghc-9.2.1/cabal-install-3.7.0.0/t/memory-usage-tests/build/memory-usage-tests/memory-usage-tests -j1 --hide-successes  (33/212 sec, 251)

Example: https://github.com/haskell/cabal/runs/5411331428?check_suite_focus=true
(master CI runs don't get to the cli-tests; that log is from #8022 which skips failing tests in lib-suite)

The test suite deliberately runs with a heap limited to 8MB:

-- Tests to run with a limited stack and heap size
--
Test-Suite memory-usage-tests
[...]
  ghc-options: -threaded -rtsopts "-with-rtsopts=-M8M -K1K"

Running locally on macOS with GHC 9.2.1:

Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    basic space leak test:          OK (21.46s)
    package with many flags:        OK (6.28s)
    issue #2899:                    OK (4.31s)
    duplicate dependencies:         OK (0.02s)
    duplicate flagged dependencies: memory-usage-tests: Heap exhausted;
memory-usage-tests: Current maximum heap size is 8388608 bytes (8 MB).
memory-usage-tests: Use `+RTS -M<size>' to increase it.

[Mikolaj]

It seems that heap is artificially restricted to 8MB in the run from the log you provide so, yes, running out of heap would be the signal of test failure.

[robx]

Yeah. Would be nice if

• the 8MB limit was more explicit -- guess we could pass it in validate.sh? Seems the 8MB is just GHC default, so if that ever changes the tests might accidentally start failing or passing erroneously
• we could see if it's a particular test case that fails (should run this locally, but it'll take a while to build all the dependencies on a new GHC version)

[Mikolaj]

I've found this:

~/r/cabal$ git grep -7 "\-M8"
cabal-install/cabal-install.cabal-Test-Suite memory-usage-tests
cabal-install/cabal-install.cabal-  import: warnings, base-dep, cabal-dep, cabal-syntax-dep, cabal-install-solver-dep
cabal-install/cabal-install.cabal-  type: exitcode-stdio-1.0
cabal-install/cabal-install.cabal-  main-is: MemoryUsageTests.hs
cabal-install/cabal-install.cabal-  hs-source-dirs: tests
cabal-install/cabal-install.cabal-  default-language: Haskell2010
cabal-install/cabal-install.cabal-
cabal-install/cabal-install.cabal:  ghc-options: -threaded -rtsopts "-with-rtsopts=-M8M -K1K"

[robx]

Ah great, thanks! Update the issue to include that info.

[robx]

By comparison, with GHC 8.10.7 I see:

Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    basic space leak test:          OK (26.68s)
    package with many flags:        OK (5.21s)
    issue #2899:                    OK (5.54s)
    duplicate dependencies:         FAIL
      Exception: stack overflow
      Use -p '/duplicate dependencies/' to rerun this test only.
    duplicate flagged dependencies: OK (0.13s)

1 out of 5 tests failed (37.58s)

With GHC 9.2.1:

### reproduce above, with command line RTS arguments
$ cabal run memory-usage-tests -- +RTS -M8M -K1K -RTS
Up to date
Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    basic space leak test:          OK (16.90s)
    package with many flags:        OK (4.12s)
    issue #2899:                    OK (3.84s)
    duplicate dependencies:         OK (0.01s)
    duplicate flagged dependencies: memory-usage-tests: Heap exhausted;
memory-usage-tests: Current maximum heap size is 8388608 bytes (8 MB).
memory-usage-tests: Use `+RTS -M<size>' to increase it.

### but rasing to 9M passes
$ cabal run memory-usage-tests -- +RTS -M9M -K1K -RTS
Up to date
Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    basic space leak test:          OK (16.45s)
    package with many flags:        OK (3.84s)
    issue #2899:                    OK (3.68s)
    duplicate dependencies:         OK (0.01s)
    duplicate flagged dependencies: OK (0.07s)

All 5 tests passed (24.07s)

### test cases interact, the failing case passes alone at 8M
$ cabal run memory-usage-tests -- -p duplicate\ flagged +RTS -M8M -K1K -RTS
Up to date
Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    duplicate flagged dependencies: OK (0.07s)

All 1 tests passed (0.08s)

### but 7M is too little
$ cabal run memory-usage-tests -- -p duplicate\ flagged +RTS -M7M -K1K -RTS
Up to date
Memory Usage
  UnitTests.Distribution.Solver.Modular.MemoryUsage
    duplicate flagged dependencies: memory-usage-tests: Heap exhausted;
memory-usage-tests: Current maximum heap size is 7340032 bytes (7 MB).
memory-usage-tests: Use `+RTS -M<size>' to increase it.

[robx]

Judging by this, I'd say there's a modest increase of memory usage with 9.2, but it doesn't appear that things scale worse. I.e., the things the tests test against (exponential space usage) are still fine, and we can raise the limit a bit.

(Also ideally we'd be running the test cases independently, but that seems too complicated to be worth it.)

[Mikolaj]

Good call. Let's stick 16MB globally in the cabal file?