Too many open files with cabal new-sdist and many data-files

[nomeata]

A project that I was trying to build contains one package (ghc-toolkit) that has 1054 files in data-files defined
(https://github.com/tweag/asterius/tree/caf15690c54aef3ae4eecbc7224c6163de040f7a/ghc-toolkit).

It also contains another package genapply that does not even depend on ghc-toolkit.

I can run cabal new-build genapply just fine. But if I run cabal new-install genapply, I get

$ cabal new-install genapply -w ~/build/haskell/ghc/inplace/bin/ghc-stage2 --allow-newer=base
Wrote tarball sdist to
/home/jojo/build/haskell/asterius/dist-newstyle/sdist/asterius-0.0.1.tar.gz
boot-libs/template-haskell/Language/Haskell/TH/LanguageExtensions.hs: openBinaryFile: resource exhausted (Too many open files)

Interestingly,

$ cabal new-sdist genapply 
Downloading
https://github.com/haskell/primitive/archive/ea9f0426646ecf8a5a0a1659b29a8dc014ceda05.tar.gz
Wrote tarball sdist to
/home/jojo/build/haskell/asterius/dist-newstyle/sdist/genapply-0.1.tar.gz

works, but

$ cabal new-sdist all
Downloading
https://github.com/haskell/primitive/archive/ea9f0426646ecf8a5a0a1659b29a8dc014ceda05.tar.gz
Wrote tarball sdist to
/home/jojo/build/haskell/asterius/dist-newstyle/sdist/asterius-0.0.1.tar.gz
boot-libs/template-haskell/Language/Haskell/TH/Ppr.hs: openBinaryFile: resource exhausted (Too many open files)

does not.

So maybe there are two issues here:

• cabal new-install creates more sdists than it has to.
• cabal new-sdist does not work well with large number of data-files.

[hvr]

It's quite possible new-sdist might have uncovered a long-standing bug in the sdist code...

[typedrat]

This means files aren't getting closed correctly, because if new-sdist works for one it should work for all.

[typedrat]

I've been looking at the code and thinking about this for the last bit, and I just do not see how this bug is possible.

I can see two options:

1. listPackageSources or something in its call tree is leaking file handles
2. Somehow, BSL.readFile is leaking file handles, because that's the only thing that actually deals with the file system for each file in the sdist in CmdSdist.hs

Neither of these seems particularly plausible, but there has to be something going on.

[23Skidoo]

Lazy I/O? A missing bracket leading to a resource leak?

[typedrat]

I don't see any obvious resource acquisitions, so I'm pretty close to saying (through the process of elimination) it's (curses!) lazy IO.

[typedrat]

Read an entire file lazily into a ByteString. The Handle will be held open until EOF is encountered.

So I need to slurp the whole file into memory and then convert that strict BS into a lazy BS, I guess.

[hvr]

NB: this was already cherry-picked to 2.4

[nomeata]

Thanks!