Package fails to build only as source-repository-package dependency, only in certain contexts

[wenkokke]

Describe the bug
A simple Haskell package with helper functions builds just fine, both as a standalone package and as a source-repository-package dependency in one context, but fails to build as a source repository package in a different context.

To Reproduce
Steps to reproduce the behavior:

1. Download this code
2. Unzip it
3. From within the directory, run cabal v2-build

The command fails with the following error message:

[11 of 16] Compiling Shoggoth.Metadata ( shoggoth/Shoggoth/Metadata.hs, dist/build/Shoggoth/Metadata.o, dist/build/Shoggoth/Metadata.dyn_o )

shoggoth/Shoggoth/Metadata.hs:25:1: error:
    Could not load module ‘Data.Aeson.Key’
    It is a member of the hidden package ‘aeson-2.0.2.0’.
    Perhaps you need to add ‘aeson’ to the build-depends in your .cabal file.
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
25 | import Data.Aeson.Key qualified as Key
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

shoggoth/Shoggoth/Metadata.hs:26:1: error:
    Could not load module ‘Data.Aeson.KeyMap’
    It is a member of the hidden package ‘aeson-2.0.2.0’.
    Perhaps you need to add ‘aeson’ to the build-depends in your .cabal file.
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.
   |
26 | import Data.Aeson.KeyMap qualified as KeyMap
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cabal: Failed to build shoggoth-0.1 (which is required by exe:builder from
plfa-20.10). See the build log above for details.

However, aeson is verifiably in build-depends!

Expected behavior
The package builds just fine, or at least fails with a different error message.

System information

• macOS 11.6.5

┌─────────────────────────────────────GHCup────────────────────────────────────┐
│    Tool  Version         Tags                          Notes                 │
│──────────────────────────────────────────────────────────────────────────────│
│✔✔  GHCup 0.1.17.8        latest,recommended                                  │
│──────────────────────────────────────────────────────────────────────────────│
│✔✔  Stack 2.7.5           latest,recommended                                  │
│✗   Stack 2.7.3                                                               │
│──────────────────────────────────────────────────────────────────────────────│
│✔✔  HLS   1.7.0.0         latest,recommended                                  │
│✗   HLS   1.6.1.0                                                             │
│✗   HLS   1.5.1                                                               │
│✗   HLS   1.5.0                                                               │
│✗   HLS   1.4.0                                                               │
│──────────────────────────────────────────────────────────────────────────────│
│✔✔  cabal 3.6.2.0         latest,recommended                                  │
│✗   cabal 3.4.1.0                                                             │
│──────────────────────────────────────────────────────────────────────────────│
│✗   GHC   9.2.3           latest,base-4.16.2.0                                │
│✓   GHC   9.2.2           base-4.16.1.0                 hls-powered           │
│✗   GHC   9.0.2           base-4.15.1.0                 hls-powered           │
│✔✔  GHC   8.10.7          recommended,base-4.14.3.0     hls-powered           │
└──────────────────────────────────────────────────────────────────────────────┘

Additional context
...

[Mikolaj]

Could you paste somewhere a whole log of the failure with cabal build -v2? Thank you.

[wenkokke]

Could you paste somewhere a whole log of the failure with cabal build -v2? Thank you.

Where do I find this? (I have access to the contents of the dist-newstyle directory, but am currently testing if, nuking ~/.cabal fixes the issue, so I cannot rerun the offending build command at the moment.)

[Mikolaj]

I guess, run cabal build -v2 and redirect all outputs to a file?

[wenkokke]

I have access to the contents of the dist-newstyle directory, but since I am currently testing whether or not removing ~/.cabal solves the problem, I currently can't rerun the offending build command.

[Mikolaj]

Sure, no rush. Actually, that reminds me, that should be cabal clean; cabal build -v2 to make sure we focus on a single reproducible failure.

[wenkokke]

From plan.json:

    {
      "type": "configured",
      "id": "shggth-0.1-36793c2f",
      "pkg-name": "shoggoth",
      "pkg-version": "0.1",
      "flags": { "install-agda": false },
      "style": "global",
      "pkg-src": {
        "type": "source-repo",
        "source-repo": {
          "type": "git",
          "location": "https://github.com/wenkokke/shoggoth",
          "tag": "949a43cab17aff6d49653d24e1a021898a7d4aa4"
        }
      },
      "pkg-src-sha256": "00d0a4828642f84c5cf3ab8b3fd0099eb5f3be5286ae4072c8c1a7b20f25db5f",
      "depends": [
        "sn-1.5.6.0-02421f7e",
        "base-4.14.3.0",
        "bmp-0.4.0-c817ea12",
        "bytestring-0.10.12.0",
        "chrstdtct-1.1.0.2-5d452a9b",
        "ctprc-0.7-afbd1683",
        "containers-0.6.5.1",
        "dt-dflt-clss-0.1.2.0-5baedbb4",
        "directory-1.3.6.0",
        "dclyt-0.4-b5c764c2",
        "flmnp-0.3.6.3-e0a1956a",
        "filepath-1.4.2.1",
        "frntmttr-0.1.0.2-35388d2a",
        "hsmn-1.0.3-60c416c2",
        "hsss-0.8.0-29b5c336",
        "mtl-2.2.2",
        "pndc-2.18-b040173e",
        "pndc-typs-1.22.2-6734c9bf",
        "shk-0.19.6-9ed18ffb",
        "tgsp-0.14.8-b536f052",
        "text-1.2.4.1",
        "txt-c-0.7.1.0-21406b30",
        "txt-rgx-rplc-0.1.1.4-7200ae66",
        "time-1.9.3",
        "transformers-0.5.6.2",
        "nrdrd-cntnrs-0.2.19.1-6baae5d4",
        "vctr-0.12.3.1-1402bc13",
        "yml-0.11.8.0-c27ff54c"
      ],
      "exe-depends": [],
      "component-name": "lib"
    },

This might indicate that the issue is a consequence of the fact that the package is installed globally, and the fact that the version number—which I have been leaving the same—takes precedence over the commit hash?

[Mikolaj]

Globally? Which package? That may break in so many ways, perhaps including the one you hypothesise. I've also heard removing stray .ghc.environment files sometimes helps. BTW, no "aeson" in the build plan?

[wenkokke]

Aeson is in the build plan with an older version:

    {
      "type": "configured",
      "id": "sn-1.5.6.0-02421f7e",
      "pkg-name": "aeson",
      "pkg-version": "1.5.6.0",
      "flags": {
        "bytestring-builder": false,
        "cffi": false,
        "developer": false,
        "fast": false
      },
      "style": "global",
      "pkg-src": {
        "type": "repo-tar",
        "repo": { "type": "secure-repo", "uri": "http://hackage.haskell.org/" }
      },
      "pkg-cabal-sha256": "5003bb6fd260d2e2d5a51dee2bf5c8e8f29e4e0e0288fef805c22dcc80ecab06",
      "pkg-src-sha256": "0361c34be3d2ec945201f02501693436fbda10dcc549469481a084b2de22bfe8",
      "depends": [
        "ttprsc-0.13.2.5-0effffb7",
        "base-4.14.3.0",
        "bs-cmpt-bttrs-0.11.2-168a6838",
        "bytestring-0.10.12.0",
        "containers-0.6.5.1",
        "dt-fx-0.3.2-dbb0ea8c",
        "deepseq-1.4.4.0",
        "dlst-1.0-34d69780",
        "ghc-prim-0.6.1",
        "hshbl-1.3.5.0-111dd363",
        "prmtv-0.7.3.0-41733dbd",
        "scntfc-0.3.7.0-50f80d3a",
        "strct-0.4.0.1-0df4efd2",
        "tggd-0.8.6.1-0ed977a4",
        "template-haskell-2.16.0.0",
        "text-1.2.4.1",
        "th-bstrctn-0.4.3.0-51001f19",
        "ths-1.1.1.1-a7262edc",
        "time-1.9.3",
        "tm-cmpt-1.9.6.1-58a8243f",
        "nrdrd-cntnrs-0.2.19.1-6baae5d4",
        "d-typs-1.0.5-ef0bbb42",
        "vctr-0.12.3.1-1402bc13"
      ],
      "exe-depends": [],
      "component-name": "lib"
    },

[wenkokke]

I suppose this could mean that, because I do not specify a lower bound, cabal picks an older version, and then reports the compile failure saying that the module that I am asking for only exists in the hidden package aeson-2.0.2.0, without reporting that it is currently using an older version of the same package.

That would also explain why this error only occurs in certain contexts; some contexts might force the package to use a lower or higher version.

It might be a good idea to have a specific error message for this case:
When emitting an error message such as "It is a member of the hidden package ‘aeson-2.0.2.0’.", it may be good to check whether or not an earlier version of that package is part of the build plan, and if so tailor the error message to include that fact, e.g.,

Could not load module ‘Data.Aeson.Key’
    It is a member of the hidden package ‘aeson-2.0.2.0’.
    Perhaps you need to add the version constraint ‘>= 2.0.2.0’ to ‘aeson’ the build-depends in your .cabal file.

[Mikolaj]

Yes, it's possible that's what is happening. Does providing a lower bound fix the original failures?

I like your idea for improving the error message. Would you mind opening it as a separate feature request ticket?