Conditionally built module are not loaded if Cabal disables them and stack.yaml reenables them with stack (revisit after commercialhaskell/stack#5480)

[konn]

Summary

If some modules are conditionally built depending on flags, and the flag is off by default, then even if one explicitly enable that flag in corresponding stack.yaml used by HLS, those modules are not loaded automatically.

Your environment

Output of haskell-language-server --probe-tools or haskell-language-server-wrapper --probe-tools:

$ ~/Library/Application\ Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-wrapper-0.8.0-darwin --probe-tools
haskell-language-server version: 0.8.0.0 (GHC: 8.10.1) (PATH: /Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-wrapper-0.8.0-darwin) (GIT hash: eb58f13f7b8e4f9bc771af30ff9fd82dc4309ff5)
Tool versions found on the $PATH
cabal:          3.2.0.0
stack:          2.5.1
ghc:            8.6.5

Which lsp-client do you use: VSCode

Describe your project (alternative: link to the project): I have the repro repo for this.

The project has the following module hierarchy:

src
├── Lib
│   └── Internal.hs
├── Lib.hs
├── Lib2
│   └── Internal.hs
└── Lib2.hs

Here, Lib imports Lib.Internal and Lib2 imports Lib2.Internal.
Lib2 and Lib2.Internal are conditionally built based on the flag dsiable-lib2, which is on by default in cabal file.
In stack.yaml, which is used for both building and HLS, disable-lib2 flag is explicitly disabled as follows, hence Lib2* must be built with stack:

flags:
  hls-conditional-modules:
    disable-lib2: false

Contents of hie.yaml:
Not needed; but I have one with generated by gen-hie:

cradle:
  stack:
    - path: "src"
      component: "hls-conditional-modules:lib"

Steps to reproduce

1. Open project directory, without any file open.
2. Open src/Lib2.hs
3. Wait for modules to be loaded

Expected behaviour

Lib2 will be successfully loaded without any error.

Actual behaviour

HLS could not load Lib2.Internal:

{
	"resource": "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/src/Lib2.hs",
	"owner": "Haskell (hls-conditional-modules)",
	"severity": 8,
	"message": "Could not load module ‘Lib2.Internal’\nIt is a member of the hidden package ‘hls-conditional-modules-0.1.0.0’.\nYou can run ‘:set -package hls-conditional-modules’ to expose it.\n(Note: this unloads all the modules in the current scope.)",
	"source": "not found",
	"startLineNumber": 3,
	"startColumn": 8,
	"endLineNumber": 3,
	"endColumn": 21
}

If one opens src/Lib2/Internal.hs, the above error dissapears.
Additionally, If one disables disable-lib2 flag in Cabal file, Lib2.Internal will be loaded just by opening Lib2.hs alone.

Include debug information

Execute in the root of your project the command haskell-language-server --debug . and paste the logs here:

Debug output:

$ ~/Library/Application\ Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-wrapper-0.8.0-darwin --debug     
Found "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/hie.yaml" for "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/a"
Module "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/a" is loaded by Cradle: Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
Run entered for haskell-language-server-wrapper(haskell-language-server-wrapper-0.8.0-darwin) Version 0.8.0.0, Git revision eb58f13f7b8e4f9bc771af30ff9fd82dc4309ff5 (dirty) x86_64 ghc-8.10.1
Current directory: /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules
Operating system: darwin
Arguments: ["--debug"]
Cradle directory: /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules
Cradle type: Stack

Tool versions found on the $PATH
cabal:          3.2.0.0
stack:          2.5.1
ghc:            8.6.5


Consulting the cradle to get project GHC version...
Project GHC version: 8.10.3
haskell-language-server exe candidates: ["haskell-language-server-8.10.3","haskell-language-server-8.10","haskell-language-server"]
Cannot find any haskell-language-server exe, looked for: haskell-language-server-8.10.3, haskell-language-server-8.10, haskell-language-server
金 01 29 hiromi@Swing ~/Documents/Programming/Haskell/git/hls-conditional-modules
$ ~/Library/Application\ Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.8.0-darwin-8.10.3 --debug 
haskell-language-server version: 0.8.0.0 (GHC: 8.10.3) (PATH: /Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.8.0-darwin-8.10.3) (GIT hash: eb58f13f7b8e4f9bc771af30ff9fd82dc4309ff5)
(haskell-language-server)Ghcide setup tester in /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:          3.2.0.0
stack:          2.5.1
ghc:            8.6.5


Step 1/4: Finding files to test in /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules
Found 5 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/Lib/Internal.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
> hls-conditional-modules> configure (lib)
> Configuring hls-conditional-modules-0.1.0.0...
> hls-conditional-modules> initial-build-steps (lib)
> Configuring GHCi with the following packages: hls-conditional-modules
> /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/.stack-work/install/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/snapshots/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/programs/x86_64-osx/ghc-8.10.3/lib/ghc-8.10.3/package.conf.d
[INFO] Using interface files cache dir: ghcide
[INFO] Making new HscEnv[main]
[INFO] Consulting the cradle for "src/Lib2.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
> hls-conditional-modules> configure (lib)
> Configuring hls-conditional-modules-0.1.0.0...
> hls-conditional-modules> initial-build-steps (lib)
> Configuring GHCi with the following packages: hls-conditional-modules
> /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/.stack-work/install/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/snapshots/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/programs/x86_64-osx/ghc-8.10.3/lib/ghc-8.10.3/package.conf.d
[INFO] Using interface files cache dir: ghcide
[INFO] Using interface files cache dir: ghcide
[INFO] Making new HscEnv[main,main]
[INFO] Consulting the cradle for "src/Lib2/Internal.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
> hls-conditional-modules> configure (lib)
> Configuring hls-conditional-modules-0.1.0.0...
> hls-conditional-modules> initial-build-steps (lib)
> Configuring GHCi with the following packages: hls-conditional-modules
> /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/.stack-work/install/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/snapshots/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/programs/x86_64-osx/ghc-8.10.3/lib/ghc-8.10.3/package.conf.d
[INFO] Using interface files cache dir: ghcide
[INFO] Using interface files cache dir: ghcide
[INFO] Using interface files cache dir: ghcide
[INFO] Making new HscEnv[main,main,main]
[INFO] Consulting the cradle for "Setup.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
File:     /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/Setup.hs
Hidden:   no
Range:    1:1-2:1
Source:   cradle
Severity: DsError
Message: 
  Multi Cradle: No prefixes matched
  pwd: /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules
  filepath: /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/Setup.hs
  prefixes:
  ("src",Stack {component = Just "hls-conditional-modules:lib", stackYaml = Nothing})
Files that failed:
[INFO] finish: User TypeCheck (took 0.06s)
 * /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/Setup.hs

Paste the logs from the lsp-client, e.g. for VS Code

LSP logs:

[client] run command: "/Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.8.0-darwin-8.10.3 --lsp"
[client] debug command: "/Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.8.0-darwin-8.10.3 --lsp"
[client] server cwd: undefined
haskell-language-server version: 0.8.0.0 (GHC: 8.10.3) (PATH: /Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.8.0-darwin-8.10.3) (GIT hash: eb58f13f7b8e4f9bc771af30ff9fd82dc4309ff5)
Starting (haskell-language-server)LSP server...
  with arguments: LspArguments {argLSP = True, argsCwd = Nothing, argFiles = [], argsShakeProfiling = Nothing, argsTesting = False, argsExamplePlugin = False, argsDebugOn = False, argsLogFile = Nothing, argsThreads = 0, argsProjectGhcVersion = False}
  with plugins: [PluginId "brittany",PluginId "class",PluginId "eval",PluginId "floskell",PluginId "fourmolu",PluginId "ghcide",PluginId "hlint",PluginId "importLens",PluginId "moduleName",PluginId "ormolu",PluginId "pragmas",PluginId "retrie",PluginId "stylish-haskell",PluginId "tactic"]
  in directory: /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules
If you are seeing this in a terminal, you probably should have run ghcide WITHOUT the --lsp option!
 Started LSP server in 0.00s
2021-01-29 11:52:18.469968 [ThreadId 25] - Registering ide configuration: IdeConfiguration {workspaceFolders = fromList [NormalizedUri 326243741438682258 "file:///Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules"], clientSettings = hashed Nothing}
2021-01-29 11:52:18.471264 [ThreadId 25] - Configuration changed: Object (fromList [("haskell",Object (fromList [("hlint",Object (fromList [("logLevel",String "info"),("executablePath",String "/Users/hiromi/.local/bin/hlint")])),("logFile",String ""),("updateBehavior",String "keep-up-to-date"),("hlintOn",Bool True),("formatOnImportOn",Bool True),("indentationRules",Object (fromList [("enabled",Bool True)])),("liquidOn",Bool False),("languageServerVariant",String "haskell-language-server"),("serverExecutablePath",String ""),("diagnosticsOnChange",Bool True),("completionSnippetsOn",Bool True),("maxNumberOfProblems",Number 100.0),("formattingProvider",String "fourmolu"),("trace",Object (fromList [("server",String "off")]))]))])
2021-01-29 11:52:18.471905 [ThreadId 25] - Opened text document: file:///Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/src/Lib2.hs
2021-01-29 11:52:18.472613 [ThreadId 105] - hlint:getIdeas:file:NormalizedFilePath "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/src/Lib2.hs"
2021-01-29 11:52:18.47305 [ThreadId 109] - Consulting the cradle for "src/Lib2.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules", cradleOptsProg = CradleAction: Stack}
2021-01-29 11:52:18.730881 [ThreadId 127] - Plugin.makeCodeLens (ideLogger)
> hls-conditional-modules> configure (lib)
> Configuring hls-conditional-modules-0.1.0.0...
> hls-conditional-modules> initial-build-steps (lib)
> Configuring GHCi with the following packages: hls-conditional-modules
> /Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/.stack-work/install/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/snapshots/x86_64-osx/9a76837884c2ac46e9c194c478c3b6e3b88485dcc802a09c5f1784038d24c283/8.10.3/pkgdb:/Users/hiromi/.stack/programs/x86_64-osx/ghc-8.10.3/lib/ghc-8.10.3/package.conf.d
2021-01-29 11:52:22.260249 [ThreadId 109] - Using interface files cache dir: ghcide
2021-01-29 11:52:22.260319 [ThreadId 109] - Making new HscEnv[main]
2021-01-29 11:52:22.273968 [ThreadId 199] - hlint:getIdeas:file:NormalizedFilePath "/Users/hiromi/Documents/Programming/Haskell/git/hls-conditional-modules/src/Lib2.hs"
2021-01-29 11:52:22.27579 [ThreadId 254] - finish: getSession (took 0.00s)
2021-01-29 11:52:22.278784 [ThreadId 350] - finish: codeLens (took 0.00s)
2021-01-29 11:52:22.27927 [ThreadId 364] - finish:  (took 0.00s)
2021-01-29 11:52:22.279577 [ThreadId 367] - finish: ModuleName.ghcSession (took 0.00s)
2021-01-29 11:52:22.280069 [ThreadId 372] - finish: ModuleName.GetParsedModule (took 0.00s)
2021-01-29 11:52:57.777813 [ThreadId 433] - finish: CodeAction (took 0.00s)
2021-01-29 11:52:58.002377 [ThreadId 439] - finish: CodeAction:PackageExports (took 0.22s)
2021-01-29 11:52:58.002629 [ThreadId 440] - finish: importLens (took 0.00s)
2021-01-29 11:52:58.002753 [ThreadId 442] - finish: MinimalImports (took 0.00s)
2021-01-29 11:52:58.003021 [ThreadId 444] - finish: addPragma (took 0.00s)
2021-01-29 11:52:58.003235 [ThreadId 446] - finish: retrie (took 0.00s)
2021-01-29 11:52:58.003459 [ThreadId 448] - finish: tactic (took 0.00s)
2021-01-29 11:52:58.564398 [ThreadId 453] - Plugin.makeCodeLens (ideLogger)
2021-01-29 11:52:58.564829 [ThreadId 454] - finish: getSession (took 0.00s)
2021-01-29 11:52:58.565176 [ThreadId 456] - finish: codeLens (took 0.00s)
2021-01-29 11:52:58.565468 [ThreadId 458] - finish:  (took 0.00s)
2021-01-29 11:52:58.565849 [ThreadId 460] - finish: ModuleName.ghcSession (took 0.00s)
2021-01-29 11:52:58.566617 [ThreadId 462] - finish: ModuleName.GetParsedModule (took 0.00s)
2021-01-29 11:56:42.555448 [ThreadId 468] - finish: CodeAction (took 0.00s)
2021-01-29 11:56:42.555638 [ThreadId 470] - finish: CodeAction:PackageExports (took 0.00s)
2021-01-29 11:56:42.555941 [ThreadId 472] - finish: importLens (took 0.00s)
2021-01-29 11:56:42.556063 [ThreadId 474] - finish: MinimalImports (took 0.00s)
2021-01-29 11:56:42.55655 [ThreadId 476] - finish: addPragma (took 0.00s)
2021-01-29 11:56:42.556789 [ThreadId 478] - finish: retrie (took 0.00s)
2021-01-29 11:56:42.557005 [ThreadId 480] - finish: tactic (took 0.00s)

[konn]

It seems that if we use cabal with cabal.project instead of stack, the problem disappears.
This can be checked with cabal branch. And if we disable flag in cabal.project, it reports the hidden-module error as expected.

[fendor]

Looks like a stack bug to me.

I checked locally, stack does not list the targets Lib2 and Lib2.Internal as targets. (As can be checked by invoking hie-bios debug src/Lib2.hs, then looking at the file pointed to by the flag -ghci-script )

Example:

> cat /run/user/1000/haskell-stack-ghci/4a4516ab/ghci-script
:add Lib Lib.Internal
:module + Lib Lib.Internal

You can see that when you execute stack repl, only two modules are imported.

[konn]

So it must be reported to commercialhaskell/stack as an upstream bug, right?

[fendor]

Yeah, ideally

[konn]

OK, I've just reported it as commercialhaskell/stack#5480.