GHC's `-package <pkg>` can expose installed package not listed by `ghc-pkg list <pkg>`

[juhp]

EDIT (by @mpilgrem): The anomaly described below is likely due to the bug (from Stack's perspective) in GHC versions described at:

• https://gitlab.haskell.org/ghc/ghc/-/issues/25025
• https://gitlab.haskell.org/ghc/ghc/-/issues/25035

In short:

• if an installed package has a name that is in the format that Cabal (the library) uses to 'mung' sub-libraries, some versions of GHC treat that name as if it were the name of the Cabal package for the purpose's of GHC's -package option; and
• as a consequence, GHC's -package <pkg> option can, in practice, cause an installed package to be exposed that is not one listed by ghc-pkg list <pkg>.

Stack, on the other hand, relies on an installed package having a unique name specified by its name field and that GHC's -package <pkg> option will expose only a installed package listed by ghc-pkg list <pkg>.

---

So far noone else has seen this: so likely it is an anomaly only in my system.

General summary/comments

After updating my projects to latest lts-22 (lts >= 22.42), builds using attoparsec started to fail for me.

Steps to reproduce

(Not reproducible so far except on my laptop)

stack --resolver lts-22.43 build aeson

Expected

To build normally.

Actual

EDIT (by @mpilgrem): Some output has been reformatted for clarity

$ stack --resolver lts-22.43 build aeson
:
aeson                        > configure
aeson                        > Configuring aeson-2.1.2.1...
aeson                        > Error: Cabal-simple_LRiJ_wrZ_3.10.3.0_ghc-9.6.6: Encountered missing or
aeson                        > private dependencies:
aeson                        > attoparsec ==0.14.4
aeson                        > 
Completed 13 action(s).

Error: [S-7282]
       Stack failed to execute the build plan.
       
       While executing the build plan, Stack encountered the error:
       
       [S-7011]
       While building package aeson-2.1.2.1 (scroll up to its section to see the error) using:
       /var/home/petersen/.stack/setup-exe-cache/x86_64-linux-tinfo6/Cabal-simple_LRiJ_wrZ_3.10.3.0_ghc-9.6.6 
--verbose=1 
--builddir=.stack-work/dist/x86_64-linux-tinfo6/ghc-9.6.6 
configure 
--with-ghc=/var/home/petersen/.stack/programs/x86_64-linux/ghc-tinfo6-9.6.6/bin/ghc 
--with-ghc-pkg=/usr/bin/ghc-pkg-9.6.6 
--user 
--package-db=clear 
--package-db=global 
--package-db=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/pkgdb 
--libdir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/lib 
--bindir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/bin 
--datadir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/share 
--libexecdir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/libexec 
--sysconfdir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/etc 
--docdir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/doc/aeson-2.1.2.1 
--htmldir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/doc/aeson-2.1.2.1 
--haddockdir=/var/home/petersen/.stack/snapshots/x86_64-linux-tinfo6/18cf10429d6b1427d08f2a7a920a4c784cc1f0eaa6921458e36e04319528d1da/9.6.6/doc/aeson-2.1.2.1 
--dependency=OneTuple=OneTuple-0.4.2-7od06fXUUXRBNk1V3VY0g5 
--dependency=QuickCheck=QuickCheck-2.14.3-E9VttT8lnId93rGQPNsiU9 
--dependency=attoparsec=attoparsec-0.14.4-GAKm7J8Gleq2k06HFjeIeY-attoparsec-internal 
--dependency=base=base-4.18.2.1 
--dependency=base-compat-batteries=base-compat-batteries-0.13.1-5fbdeENWe9mF7TcjAFq4dk 
--dependency=bytestring=bytestring-0.11.5.3 
--dependency=containers=containers-0.6.7 
--dependency=data-fix=data-fix-0.3.4-2n4PElhIt1Q7w0n82lPfpl 
--dependency=deepseq=deepseq-1.4.8.1 
--dependency=dlist=dlist-1.0-7vDlnn0Hdvg35SyXLwMaWr 
--dependency=exceptions=exceptions-0.10.7 
--dependency=generically=generically-0.1.1-I9byc5Nil798plofO827gA 
--dependency=ghc-prim=ghc-prim-0.10.0 
--dependency=hashable=hashable-1.4.4.0-GGHvrv8RGdcAwi4fOW9L5d 
--dependency=indexed-traversable=indexed-traversable-0.1.4-8j5HZpShpE5BqFup9Ojenr 
--dependency=primitive=primitive-0.8.0.0-G7z1XrhwN0bFkYsIqIr1QU 
--dependency=scientific=scientific-0.3.7.0-BAvZdJlkHDUEf0LOVEPz37 
--dependency=semialign=semialign-1.3.1-7lhN9UdvYxAGbqZZCyRcNC 
--dependency=strict=strict-0.5-GMywuJToqFcduYfo019sj 
--dependency=tagged=tagged-0.8.8-Kzng2lnKElzJiyKd9g735c 
--dependency=template-haskell=template-haskell-2.20.0.0 
--dependency=text=text-2.0.2 
--dependency=text-short=text-short-0.1.6-2ZxoUm7jRrSBd8GMEi1Fas 
--dependency=th-abstraction=th-abstraction-0.5.0.0-HAFjiAO2nGN58SdxVZCnLH 
--dependency=these=these-1.2.1-KST5KzzcktxI0GEHGKxqQO 
--dependency=time=time-1.12.2 
--dependency=time-compat=time-compat-1.9.6.1-DIvC8GbQSzcCziVynZc4vt 
--dependency=unordered-containers=unordered-containers-0.2.20-5m928IsagguARIfUwwmGLp 
--dependency=uuid-types=uuid-types-1.0.5.1-3QSWhjKdWiE4JA9TKvvlMc 
--dependency=vector=vector-0.13.1.0-Jdel1KiNlSEIXGg2MpN3IL 
--dependency=witherable=witherable-0.4.2-D4tJt8ldIw17tHtCEGjK6w 
--dependency=attoparsec:attoparsec-internal=attoparsec-0.14.4-GAKm7J8Gleq2k06HFjeIeY-attoparsec-internal 
-f-cffi 
-fordered-keymap 
--exact-configuration 
--ghc-option=-fhide-source-paths
       Process exited with code: ExitFailure 1

See https://gist.github.com/juhp/8cf70c3e00509347ff5e5f4bbfad74c5 for the full --verbose output

Highlighting a few points:

• - attoparsec-0.14.4 appears twice (probably once for attoparsec and once for attoparsec-internal?)
• for attoparsec there is only --dependency=attoparsec=attoparsec-0.14.4-GAKm7J8Gleq2k06HFjeIeY-attoparsec-internal (attoparsec:attoparsec-internal gets mapped to the same pkgid)

Stack version

2.15.7 and 3.1.1

Method of installation

Fedora Linux and local build

Platform

Fedora Linux

Misc

I am using a workaround: adding os-string-2.0.6 to my project stack.yaml files (the version of os-string in lts < 22.42)

[mpilgrem]

Looking at the gist, one thing that is odd is that the unofficial version of Stack 2.15.7 that you are using itself lists attoparsec-0.14.4 twice as a dependency that Stack has been compiled against. It seems to me possible that something may have corrupted the local write-only (snapshot) database of immutable packages.

EDIT: Actually, that is a red-herring. I see that where a package has a sub-library (for example, pantry or tar), its version gets listed twice by stack --version.

[juhp]

I tried with both --system-ghc and also removing (renaming) ~/.stack/snapshots/ and can still reproduce here.
I guess I can try to rename "harder" (like all ~/.stack)... (though looks to me that these caused everything to get rebuilt?)

Also just noting here too (as I mentioned in matrix) that I couldn't reproduce in a fresh Fedora container.

[juhp]

I uploaded the stack-3.1.1 output too for completeness: https://gist.github.com/juhp/e4f92e6df99524da0d7df72afe2470d4

[juhp]

Also I also tried in a container first building with lts-22.41 and then lts-22.43, but this also worked fine.

[mpilgrem]

I am wondering if this GHC bug is relevant:

• https://gitlab.haskell.org/ghc/ghc/-/issues/25025
• https://gitlab.haskell.org/ghc/ghc/-/issues/25035

[juhp]

Ah yes indeed - I had quite forgotten about that problem 😢
Bullseye I guess!

Is there a related stack issue?

[mpilgrem]

@juhp, I have edited this issue to make it Stack's related issue.

[juhp]

BTW I finally "took the (small) plunge" and tried moving (aka "deleting") ~/.stack/ out of the way and the problem still occurs for me, so I don't know any solution other than avoiding ghc-9.6.6 (other than my workaround ;o) 😢

So wonder what triggers this on my machine, since it appears not reproducible, or is it just bad luck?

---

Guess I forgot to write down my workaround, just in case anyway one should hit this somehow, in stack.yaml:

extra-deps:
  - os-string-2.0.6

[juhp]

Hmm, I am wondering if this interference is actually caused by the system library packages?

Because now I started to get the same problem locally with Stackage LTS 23 after upgrading system (distro) Haskell to ghc-9.8.4: it wasn't happening before with lts-23 when the system ghc and libraries were based on ghc-9.6.

Further after the upgrade, I no longer need my workaround hack for lts-22, which suggests my hypothesis may be correct.

I am not sure how to differentiate ghc boot/core libs from other Haskell system libraries though for stack.

I am now building with stack-3.1.1 btw.

[juhp]

But the biggest problem now for me is that the previous workaround no longer works with lts-23: or perhaps i need different package workaround this time - which is quite annoying.

[mpilgrem]

@juhp, the problem is packages that have a sub-libary. e.g attoparsec-0.14.4 introduced attoparsec:attoparsec-internal. EDIT: I've added another GHC issue above in the first post above that explains how the experience of the problem can feel arbitrary as between different OS and GHC versions.