Hydra sometimes runs out of memory while building `dhall`

[sjakobi]

Recent examples:

• Support for headers.dhall configuration #2236 (comment)
• Adjust megaparsec upper bounds for SemVer #2312, with error:

  [11 of 47] Compiling Dhall.Syntax     ( src/Dhall/Syntax.hs, dist/build/Dhall/Syntax.o, dist/build/Dhall/Syntax.dyn_o )
  /nix/store/qdf49mvm79r83n9c9s7pkmmjqwhrw8jv-stdenv-linux/setup: line 1316: 3376277 Killed                  ./Setup build
  builder for '/nix/store/80snyp1ffix7ls7ayjjwxs0sqw1d3fqi-dhall-1.40.1.drv' failed with exit code 137
  

If Dhall.Syntax turns out to trigger this problem frequently, we could try to split that module in order to reduce the burden on GHC.

[Gabriella439]

Yeah, we should probably split it into one module per type at this point since it's getting large

[sjakobi]

It just happened again in #2314. Dhall.Syntax indeed seems to be the problem.

[sjakobi]

Here are GHC's current stats when I build dhall with GHC 8.10.7 and this patch:

--- a/cabal.project
+++ b/cabal.project
@@ -1 +1,4 @@
 packages: ./dhall ./dhall-bash ./dhall-json ./dhall-yaml ./dhall-lsp-server ./dhall-nix ./dhall-docs ./dhall-openapi ./dhall-nixpkgs ./dhall-csv ./dhall-toml
+
+package dhall
+  ghc-options: -Rghc-timing

<<ghc: 122860900680 bytes, 4470 GCs, 170946838/468747064 avg/max bytes residency (28 samples), 1128M in use, 0.000 INIT (0.000 elapsed), 65.290 MUT (68.303 elapsed), 20.892 GC (20.931 elapsed) :ghc>>

The number that we want to reduce is the 1128M in use.

Here are some ideas for reducing the size of Dhall.Syntax:

• Move denote, renote and shallowDenote to a new module Dhall.Syntax.Notes denote is used in Dhall.Pretty.Internal
• Move Import and its components to a new module Dhall.Syntax.Import
• Move desugarWith to Dhall.Core
• Move shift to a separate module Dhall.Shift
• Move subExpressions etc. to a separate module Dhall.Syntax.Optics (unsafeSE is used in Dhall.Syntax though, and should probably stay)
• Move Chunks and related functions to Dhall.Syntax.Chunks Chunks depends on Expr and its instances.
• Extract Const to own module
• Extract Var to own module
• Extract DhallDouble to own module
• Extract PreferAnnotation to own module
• Move reservedKeywords and reservedIdentifiers to Dhall.Syntax.Reserved

[sjakobi]

I've implemented some relatively simple refactorings in #2317. Unfortunately the effect is rather unsatisfactory so far. I think the lowest maximum residency I saw was 1099MB (down from the initial 1128MB), but one of the last changes must have somehow negated these (tiny) improvements, so I'm now measuring 1138MB:

<<ghc: 123205423920 bytes, 4286 GCs, 174184726/464036480 avg/max bytes residency (27 samples), 1138M in use, 0.000 INIT (0.000 elapsed), 68.056 MUT (71.126 elapsed), 21.036 GC (21.061 elapsed) :ghc>>

There are a few more things that can be extracted without causing dependency cycles. However my suspicion is that the main GHC perf challenges are the derived instances of Expr and its components like RecordField. The relevant GHC tickets are https://gitlab.haskell.org/ghc/ghc/-/issues/5642 and https://gitlab.haskell.org/ghc/ghc/-/issues/9557. The main workaround I'm aware of is to write instances by hand. Are there any other tricks?

[Gabriella439]

This is why I think there needs to be one module per type (e.g. Dhall.Syntax.{Expr,Binding,RecordField}), because that would split up the derived instances

[sjakobi]

This is why I think there needs to be one module per type (e.g. Dhall.Syntax.{Expr,Binding,RecordField}), because that would split up the derived instances

This will involve adding .hs-boot files, but I'll give it a try.