`file-embed`: Additions to `extra-source-files` don't trigger rebuild of module

[andreasabel]

Report against Stack-3.3.1 (latest).
I noticed that Stack does not seem to honor the extra-source-files when deciding whether to rebuild a project, in contrast to Cabal which does so.

This is relevant when using file-embed to embed files into the executable. Changes to the embedded files should trigger a recompilation to keep the executable up-to-date.

E.g. I want that these files are always embedded in their latest version:

extra-source-files:
    data/README.md
    data/sub/Foo.md

I added the second one in a second step. I see the following behavioral difference between Stack and Cabal:

$ stack run                                                                              ✔ │ ghc-9.4.8 hs │ 10:06:41 
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"

$ cabal run                                                                              ✔ │ ghc-9.4.8 hs │ 10:14:10 
File: README.md
Content: "This is a test of the `file-embed` package,\napplied to the `data-dir` of a Haskell project.\n"
File: sub/Foo.md
Content: "This is a file in a subdirectory.\nIt contains 2 lines.\n"

WANT: Changes to the amount and content of the extra-source-files should trigger a rebuild.

I attach a reproducer to play with.

embed-datadir-0.1.tar.gz

[andreasabel]

This issue might be releated:

• qAddDependentFile doesn't cause re-compilation without extra-source-files #4360

[mpilgrem]

Possibly relevant history:

• qAddDependentFile doesn't cause re-compilation without extra-source-files #4360 (as above)
• extra-source-files outside the package directory don't trigger a rebuild #1982
• Custom Preprocessed Files Don't Rebuild #1891

The Cabal user guide currently documents that both extra-files (added cabal-verison: 3.14) and extra-source-files are to add files to an sdist archive, but the latter are also tracked by cabal build (first documented with Cabal 3.10, introduced in Cabal 3.4:

• Rebuild on changes in all extra-source-files haskell/cabal#6889

).

[mpilgrem]

On Windows 11, I can't reproduce stack build (or stack run) not respecting changes to extra-source-files, per se:

❯ stack run
testESF-0.1.0.0: unregistering (local file changes: README.md)
testESF> build (lib + exe) with ghc-9.8.4

Also, I can't reproduce§ with package file-embed and $(embedDir "data") with lts-23.9 (GHC 9.8.4, template-haskell-2.21.0.0, file-embed-0.0.16.0) or lts-21.25 (GHC 9.4.8, template-haskell-2.19.0.0, file-embed-0.0.15.0). In each case, stack build recognises a change to README.md and embeds the changed file into the built executable.

§ EDIT: OK, I now see there is a problem when a new file is added to extra-source-files and Main.hs has already been compiled. I see that the Haddock documentation for embedFileIfExists has:

Warning: When a build is compiled with the file missing, a recompile when the file exists might not trigger an embed of the file. You might try to fix this by doing a clean build.

I am wondering if that is a more general problem with file-embed.

EDIT2: If I understand correctly, file-embed-0.0.16.0 has:

pairToExp :: FilePath -> (FilePath, B.ByteString) -> Q Exp
pairToExp _root (path, bs) = do
#if MIN_VERSION_template_haskell(2,7,0)
    qAddDependentFile $ _root ++ '/' : path -- <<< This tells GHC to track changes to file contents
#endif
    exp' <- bsToExp bs
    return $! TupE
#if MIN_VERSION_template_haskell(2,16,0)
      $ map Just
#endif
      [LitE $ StringL path, exp']

So, if a file is added subsequently: [1] GHC does not know to recompile Main.hs; and [2] Stack has no reason to recompile Main.hs either (as it has not changed).

[andreasabel]

@mpilgrem Thanks for the analysis!

I see that you flagged this as resolution: upstream issue, does this mean that you think that the problem is solely in file-embed?
Then one would wonder why Cabal can do this correctly, but Stack not.

It seems that changes to extra-source-files are more aggressively triggering rebuilds in Cabal compared to Stack.
This was maybe annoying in the time when extra-doc-files (and now extra-files) did not exist so folks needed to abuse extra-source-files for the purpose of just adding files to the sdist tarball.

Just from a theoretical perspective, file-embed by itself has no means to get the case right when files are added to a directory embedded by embedDir. It can, when executed, mark all the files in the directory as involved in the build (qAddDependentFile), so that changes to these files will trigger a rebuild subsequently. But as it cannot know the future, it cannot prepare for files that are added to the directory. So in this case, the trigger for a rebuild has to come externally, namely from the build orchestrator (Stack or Cabal) that sees a new entry in the file list denoted by extra-source-files (so, after expansion of wildcards) and will force a total rebuild.

I see that you found the respective PR for cabal:

• Rebuild on changes in all extra-source-files haskell/cabal#6889

So you are likely aware of all of this already; I just wanted to write it down for the sake of clarity.