Addition of generic pre build and post build hooks.

[erikd]

Describe the feature request
The ability to call shell scripts just before and just after each package is built.

At IOG, our use case for these is for build caching, particularly in CI. Since these pre and post hooks are just shell scripts, there are probably other uses for them.

Additional context
At IOG we have a number of very large Haskell projects with deep dependency trees, that can take a long time to build in CI.

The obvious answer to long build times is caching of build products. A previous attempt at this caching was made, but that solution was not really very satisfactory, because the cache was keyed on ${CPU}-${OS}-${GHC_VERSION)-${hash-of-dependencies}. The first three are obvious. The problem is hash-of-dependencies. If a single high level dependency changes, there will be no cache hit and everything will be built from scratch. As it turns out, this is actually the most common scenario.

A better caching solution is one where the caching is done on individual dependencies rather than on all the dependencies as a huge blob. Caching individual dependencies means that when a high level dependency changes, there is a very high likelihood that all the lower level dependencies will still be found in the cache.

My initial implementation of this package level caching was done as a simple wrapper around cabal that used rsync to fetch and save the cache over ssh to another machine. This proved highly effective and I was able to populate the cache from one machine and use if from another (both machines running Debian Linux).

However, @angerman came up with an even better solution that required adding the ability to run shell scripts before and after the build of each individual package. Using this feature (we have rough patches against cabal HEAD and version 3.10.3.0) we are able to use our own Amazon S3 storage for our cache. We do not propose to make this S3 storage public (obvious potential security problems) but any organization like ours or any individual could use their own S3 storage. I also have a working pair of pre and post build hooks that use ssh to a different machine as the storage backend.

The naive patch against HEAD (error handling could be improved, maybe the hook names could be made configurable) is:

diff --git a/cabal-install/src/Distribution/Client/ProjectBuilding/UnpackedPackage.hs b/cabal-install/src/Distribution/Client/ProjectBuilding/UnpackedPackage.hs
index 065334d5c..570c9b18c 100644
--- a/cabal-install/src/Distribution/Client/ProjectBuilding/UnpackedPackage.hs
+++ b/cabal-install/src/Distribution/Client/ProjectBuilding/UnpackedPackage.hs
@@ -678,7 +678,22 @@ buildAndInstallUnpackedPackage
           runConfigure
         PBBuildPhase{runBuild} -> do
           noticeProgress ProgressBuilding
-          runBuild
+          -- run preBuildHook. If it returns with 0, we assume the build was
+          -- successful. If not, run the build.
+          code <- rawSystemExitCode verbosity (Just srcdir) "preBuildHook" [
+              (unUnitId $ installedUnitId rpkg)
+            , (getSymbolicPath srcdir)
+            , (getSymbolicPath builddir)
+            ] `catchIO` (\_ -> return (ExitFailure 10))
+          when (code /= ExitSuccess) $ do
+            runBuild
+            -- not sure, if we want to care about a failed postBuildHook?
+            void $ rawSystemExitCode verbosity (Just srcdir) "postBuildHook" [
+                (unUnitId $ installedUnitId rpkg)
+              , (getSymbolicPath srcdir)
+              , (getSymbolicPath builddir)
+              ] `catchIO` (\_ -> return (ExitFailure 10))
+
         PBHaddockPhase{runHaddock} -> do
           noticeProgress ProgressHaddock
           runHaddock

An example of our preBuildHook script (kept in ~/.cabal/iog-hooks, S3 credentials pulled from s3-credentials.bash, the aws executable is from the awscli package) is as follows:

#!/usr/bin/env bash

# shellcheck disable=SC1091,SC2046
. $(dirname "${BASH_SOURCE[0]}")/s3-credentials.bash

CACHE_KEY="$1.tar.gz"

# Check if artifact exists in S3 (with endpoint and credentials)
aws s3 ls s3://"$CACHE_BUCKET"/"$CACHE_KEY" --endpoint-url "$AWS_ENDPOINT" > /dev/null 2>&1

# shellcheck disable=SC2181
if [ $? -eq 0 ]; then
  echo "S3 hit       $1"
  aws s3 cp s3://"$CACHE_BUCKET"/"$CACHE_KEY" - --endpoint-url "$AWS_ENDPOINT" | tar -xz
else
  echo "S3 miss      $1"
  exit 1
fi

To use the cache I run cabal as:

PATH=$PATH:$HOME/.cabal/iog-hooks cabal 

[erikd]

Example output using the ssh storage backend:

...
Starting     criterion-1.6.3.0 (lib)
Building     criterion-1.6.3.0 (lib)
SSH miss     criterion-1.6.3.0-4239da68ccfe796a2fcfc4950e53b50c0b0b508391999faf9fb38d92e10aae86
SSH upload   criterion-1.6.3.0-4239da68ccfe796a2fcfc4950e53b50c0b0b508391999faf9fb38d92e10aae86
Installing   criterion-1.6.3.0 (lib)
Completed    criterion-1.6.3.0 (lib)

...

[fgaz]

Related: #7394

I think this would be a welcome feature, especially if it's written with the above issue in mind so it can be extended to other actions.

[erikd]

I just did a test using ghc-9.6 on a machine with 12 cores and 48 of RAM (ie significantly more powerful than expected CI build machines). The version of cabal being used is a patched 3.10.3.0.

With an empty S3 cache:

cabal clean
rm -rf ~/.cabal/store/ghc-9.6.4/
time PATH=$PATH:$HOME/.cabal/iog-hooks/ cabal build all

real    29m53.885s
user    105m12.177s
sys     11m5.029s

With a warm S3 cache (ie immediately after the above):

cabal clean
rm -rf ~/.cabal/store/ghc-9.6.4/
time PATH=$PATH:$HOME/.cabal/iog-hooks/ cabal build all


real    13m20.817s
user    37m24.021s
sys     6m58.173s

Without the S3 cache, but just using ~/.cabal/store/ghc-9.6.4/:

cabal clean
time cabal build all

real	4m0.510s
user	8m6.877s
sys	0m39.885s

[mpickering]

Is there an advantage to doing this using a hook rather than populating the caches based on plan.json before starting to do any building?

It seems that using an approach like cabal-cache, using plan.json would be easier and wouldn't require modifying cabal-install.

[newhoggy]

Does this issue propose to fix this?

Note that because builds are not reproducible in general (nor even necessarily ABI compatible) then it is essential that the loser abandon their build and use the one installed by the winner, so that subsequent packages are built against the exact package from the store rather than some morally equivalent package that may not be ABI compatible.

References:

• https://github.com/haskell-works/cabal-cache/issues/166
• https://hackage.haskell.org/package/cabal-install-3.8.1.0/docs/Distribution-Client-Store.html

[newhoggy]

There is another I've issue encountered, such as packages having absolute paths built into them which mean they are non-transferrable to other machines where the build location, store location or some other environment differs.

I have some examples of such packages:

➜  tmp aws s3 ls s3://cache.haskellworks.io/archive/v2/f58b96473c/ghc-9.8.2/
2024-03-01 21:17:12     104869 ansi-terminal-1.0.2-a01f3219a7a4a7a4eb77ff621a0ed6b6e450136b05407e626307f299190a71f4.tar.gz
2024-03-01 21:17:12     180356 ansi-terminal-types-0.11.5-6827843e420e0d028f9e48e791f67001efa3cb47d381e9b165cc8f607a821d78.tar.gz
2024-03-01 21:17:13     391194 colour-2.3.6-c937c1fae7c2f789054e5eb5c3a1da6e4af1f236fc621e02a544dc78739e0a37.tar.gz
2024-03-01 21:17:13     260291 concurrent-output-1.10.20-e5726e733cf6304131aaf02376ead2ea69cba2bd2247c84253034ffb886d3da1.tar.gz
2024-03-01 21:17:14     763100 happy-1.20.1.1-e-happy-7b1696a89248290667aee02ca6d97d8d7bc03d6bc936a5b79e3d972cf6149914.tar.gz
2024-03-01 21:17:15    1842811 hedgehog-1.4-f3dd734526516a61afde47d069e4ff9e6fb5b254943565727396b95e10a9d87a.tar.gz
2024-03-01 21:17:14     490535 hedgehog-extras-0.6.1.0-5b53412e9dce31c1bf0c3c1946a5f3a121c09d3f879ee270bbeb28933031de3d.tar.gz
2024-03-04 20:56:49     490469 hedgehog-extras-0.6.1.0-a717c8aeedaa81cf162a2936a8af86a26f5e25dfcf03b8c4768137b7dffbcce2.tar.gz
2024-03-01 21:17:15     704096 hsc2hs-0.68.10-e-hsc2hs-1022064a9b8047005f5a964e5ffc29aa050879f17b2d8b0353edd710087bd83b.tar.gz
2024-03-01 21:17:16     672138 optparse-applicative-0.18.1.0-29c379e4e141e0fc10805ddbd48a772bb236d5ed446942f8f177b5fa43bec33b.tar.gz
2024-03-01 21:17:16     291745 pretty-show-1.10-767af4c07eab91ce69eba890ed7ef3435f397b0b0db4843a5ff7993e73e3af84.tar.gz
2024-03-01 21:17:16     127976 prettyprinter-ansi-terminal-1.1.3-87e22b7729218bd826726f9ba761abe4444cfdcc37607e1594238643531220cd.tar.gz
2024-03-01 21:17:17     752879 tasty-1.5-f5a2ece4f2fd1a09fbad0832fdd84bf0497cee2ac510cd1c50565175637cf596.tar.gz
2024-03-01 21:17:17     141137 tasty-discover-5.0.0-5fcc9e3560b4cb8470b7c13d067c5390dc0a7993801da102d8092fdc5dc2b61c.tar.gz
2024-03-01 21:17:17     612782 tasty-discover-5.0.0-e-tasty-discover-bd4ef76688b231ecc5ef6f0d99866cf056e55532494977897dc649f679ed4368.tar.gz
2024-03-01 21:17:17      84577 tasty-hedgehog-1.4.0.2-343fcb9e5a2dae3fc4bb0ea138d1ed3e745ab9fbc7dba38e113eaceb9b922636.tar.gz

cabal-cache currently attempts to detect such cases and attempts to mitigate the problem by not transferring such packages if it would break for this reason.

[newhoggy]

Is there an advantage to doing this using a hook rather than populating the caches based on plan.json before starting to do any building?

Brainstorming a bit:

I can see this potentially evolving to the point where in CI or even on development machines we don't build any of the dependencies because we can point to a caching build server. If we reach this point the advantage is that if we have a large projects, components that depend on already cached dependencies can start building earlier whilst the remote build server is busy populating the cache for yet unbuilt dependencies.

It could be a paid for cloud service as well, maybe something the Haskell Foundation can run?

The advantage of this would be minimal idle time by build client.

[angerman]

There are a few advantages to hooks that you do not get from reconstructing the store outside of cabal. As @erikd showed the patch is currently also ~20lines of extra code in cabals codebase and it permits all kinds of use for the hooks. You could run some time profiling with them, or anything else you'd like to do around builds.

• By being at the location where it is, and effectively substituting GHC's build, you can omit _Paths' modules from the reconstruction, and have GHC build paths modules for you.
• It's not trying to reconstruct the store, and leaves all the store logic (layout, ...) up to cabal. Inlcuding .conf generation, copying and installing. (Most of the logic of cabal is retained in cabal, and the hooks do not have to concern themselves with any of that.

ABI is a major issue. Luckily for us we are getting close to making it irrelevant for us.

[angerman]

One last note: these are generic hooks. Caching may be one use for them. But hooks into cabal have been requested (and discussed outside of issue trackers) multiple times.

[newhoggy]

Can you elaborate on the _Paths modules? Are they still going to be included in the packages but ghc patches after the fact? Or will they be gone from the packages?

[newhoggy]

I wanted to understand if pre/post build hooks are the only ones we need. For example do we need a no build hook because the package was already built?

[angerman]

@newhoggy caching is clearly only one instance of what could be done with these hooks. (And we likely want cabal to have more hooks!).

Can you elaborate on the _Paths modules? Are they still going to be included in the packages but ghc patches after the fact? Or will they be gone from the packages?

You can just not backup any _Paths modules in the tarball you build --exclude=*_Paths.* or similar.

When you restore the cached objects into the build folder ghc will just build the ones that are missing. After that cabal will continue with copy/reg/install.

I wanted to understand if pre/post build hooks are the only ones we need. For example do we need a no build hook because the package was already built?

For this PoC, the pre/post hooks seem to be enough. However, you'd likely want to have many more hooks in cabal to hook different phases for experimentation, and extensions.

[hasufell]

Related: #7394

I think this would be a welcome feature, especially if it's written with the above issue in mind so it can be extended to other actions.

Exactly. This has been requested/discussed since the dawn of time.

Shell hooks are as old as unix and the other major design pattern after "pipes". I'm not sure it even warrants a discussion... and we surely shouldn't be discussing specific use cases. There are infinite.

The question is more about:

• where exactly do we want to insert hooks
• what information should be exposed to the hooks

I think last time it was discussed, it wasn't really clear where to insert hooks, because cabal had no clean architecture that resembles strictly the configure/build/install phases that traditional package managers have.

So this would be the only concern I see: which parts are stable enough architecturally that they won't break hooks in the future. Or do we need a refactor first?

[geekosaur]

Another place I'd consider a cabal hook is SCM support (cf. #9883) — ideally we'd just drop a script somewhere rather than needing to add it into cabal-install and make a release. (Also goes for the discussion currently going on on Matrix about possibly having cabal sdist generate source-repository stanzas, although that one would go in the opposite direction.)