Add separate cache for getPkgConfigDb by andreabedini · Pull Request #9422 · haskell/cabal

andreabedini · 2023-11-08T03:16:19Z

Querying pkg-config for the version of every module can be a very expensive operation on some systems. This change adds a separate, per-project, cache for pkgConfigDB; reducing the cost from "every plan change" to "every pkg-config-db change per project".

No input key is required since getPkgConfigDb already specifies the directories to monitor for changes. These are obtained from pkg-config itself as pkg-config --variable pc_path pkg-config as documented in pkg-config(1).

A notice is presented to the user when refreshing the packagedb.

Closes #8930 and subsumes #9360.

changelog entry to follow

geekosaur

Would it be worth adding a test that ran cabal twice and verified that the cached pkg-config output is used the second time?

andreabedini · 2023-11-08T03:57:21Z

Would it be worth adding a test that ran cabal twice and verified that the cached pkg-config output is used the second time?

I think it would be, I am not sure whether we have anything similar already. I'll check.

geekosaur · 2023-11-08T04:12:50Z

Not for caching, but there's a test that verifies that a package reinstall causes a warning IIRC.

gbaz · 2023-11-08T04:51:08Z

this looks excellent, thanks!

mpickering · 2023-11-08T13:56:15Z

Can you describe the cache invalidation strategy?

Does the cache get invalidated when PKG_CONFIG_PATH environment variable gets changed?

It would be good to add some tests testing the different invalidation mechanisms.

andreabedini · 2023-11-09T04:54:57Z

Can you describe the cache invalidation strategy?

Yes, I am relying on what getPkgConfigDb does currently:

getPkgConfigDb :: Verbosity -> ProgramDb -> Rebuild PkgConfigDb
getPkgConfigDb verbosity progdb = do
  dirs <- liftIO $ getPkgConfigDbDirs verbosity progdb
  -- Just monitor the dirs so we'll notice new .pc files.
  -- Alternatively we could monitor all the .pc files too.
  traverse_ monitorDirectoryStatus dirs
  liftIO $ readPkgConfigDb verbosity progdb

It obtains PKG_CONFIG_PATH and for monitors each directory for changes. As the comment says, it does not monitor each individual files.

Perhaps it could be worth monitoring the result of getPkgConfigDbDirs itself?

It would be good to add some tests testing the different invalidation mechanisms.

I agree, I am planning to write some.

andreabedini · 2023-11-21T07:31:23Z

I did some rework and added tests but one of the tests (ExtraProgPath) seems to have picked up a small difference in behaviour.

andreabedini · 2023-12-05T10:00:00Z

@mpickering do you think the tests are sufficient? I am checking the search path (for the pkg-config executable) and PKG_CONFIG_PATH. I don't think it is a perfect solution but it seems to be the approach taken elsewhere.

andreabedini · 2024-01-24T14:30:13Z

@jasagredo does 3ae0488 look alright to you? I tried to follow what you did in #9527

andreabedini · 2024-01-29T04:11:22Z

Actual output differs from expected:

stderr:
--- /home/runner/work/cabal/cabal/cabal-testsuite/PackageTests/ExtraProgPath/setup.dist/setup.out.normalized	2024-01-24 14:53:40.341551022 +0000
+++ /home/runner/work/cabal/cabal/cabal-testsuite/PackageTests/ExtraProgPath/setup.dist/setup.comp.out.normalized	2024-01-24 14:53:40.341551022 +0000
@@ -1,8 +1,6 @@
 # cabal v2-build
 Warning: cannot determine version of <ROOT>/./pkg-config :
 ""
-Warning: cannot determine version of <ROOT>/./pkg-config :
-""
 Resolving dependencies...
 Error: [Cabal-7107]
 Could not resolve dependencies:
*** Exception: ExitFailure 1

*** unexpected failure for PackageTests/ExtraProgPath/setup.test.hs

🧐

fgaz · 2024-02-02T10:36:23Z

@michaelpj

So from a Nix perspective, the thing that should force cache invalidation is that we have a different pkg-config executable in the two cases

the pkg-config executable is the same in all my shells. What changes is the non-standard variable $PKG_CONFIG_PATH_FOR_TARGET, confirmed by https://github.com/NixOS/nixpkgs/blob/55ca7939944136731b34316a4c6fc2d4a11da122/pkgs/build-support/pkg-config-wrapper/pkg-config-wrapper.sh

I think there's nothing we can do here unfortunately.

@andreabedini I tried again and the result is identical

andreabedini · 2024-02-05T04:27:35Z

the pkg-config executable is the same in all my shells. What changes is the non-standard variable $PKG_CONFIG_PATH_FOR_TARGET, confirmed

oh, in my tests the path to the wrapper was changing :-/

michaelpj · 2024-02-05T09:13:56Z

the pkg-config executable is the same in all my shells. What changes is the non-standard variable $PKG_CONFIG_PATH_FOR_TARGET

Right, and in the nix world this is all detectable because the derivation that produces the pkg-config executable is changing, but it's hard for us to perform as fine-grained a check in cabal. It wouldn't be enough to hash the executable, since we also need to know about stuff it uses transitively...

Some ideas:

Add an environment variable to turn off the caching, tell Nix folks to set it
Make the cache per-project rather than global. Then somewhat by coincidence this will work out, since the granularity at which Nix folks tend to have different pkg-config executables is per-project. Also I guess a bit more consistent with cabal's other caches, and easily cleanable with cabal clean It already is per-project
Try and at least advertise that we are getting cached results in solver output, so people know something is up. Especially if thee is a failure...

EDIT: since it's per-project the damage seems less bad, but it maybe forces Nix users to cabal clean when they update their pkg-config dependencies...

andreabedini · 2024-02-06T13:09:15Z

I don't have much time to dedicate to this so I would appreciate if anybody could help pushing this over the line. It's complete but I had issues with the tests.

jasagredo · 2024-06-10T15:44:41Z

So I tried to run this on Windows again. Some remarks:

There are symlinks here. I would advise against that, they do not work on windows, one has to do weird magic tricks to support them.
~~A test is actually failing: PackageTests\CCompilerOverride\setup.test.hs. I will look into it eventually. It says it cannot find the C compiler.~~ It failed because I didn't have clang in my UCRT environment.
My ./validate.sh process dies abruptly with this same error I reported time ago: Running the test-suite terminates abruptly with fd:5: hGetLine: end of file in MSYS2 Windows #9571

jasagredo · 2024-06-14T12:54:57Z

I got this error on my machine:

fatal: 'C:\msys64\tmp\repos-67360\src' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Unit Tests
  UnitTests.Distribution.Client.Get
    forkPackages, network tests
      git clone:                                                                                                                       FAIL (2.06s)
        tests\UnitTests\Distribution\Client\Get.hs:219:
        expected a file to exist: C:\msys64\tmp\repos-67360\zlib1/zlib.cabal
        Use -p '/git clone/' to rerun this test only.

1 out of 543 tests failed (77.03s)
<<< C:\Users\Javier\code\cabal\dist-newstyle-validate-ghc-9.8.2\build\x86_64-windows\ghc-9.8.2\cabal-install-3.11.0.0\t\unit-tests\build\unit-tests\unit-tests.exe -j1 --hide-successes (87/104 sec, 1)
<<< C:\Users\Javier\code\cabal\dist-newstyle-validate-ghc-9.8.2\build\x86_64-windows\ghc-9.8.2\cabal-install-3.11.0.0\t\unit-tests\build\unit-tests\unit-tests.exe -j1 --hide-successes (87/104 sec, 1)

jasagredo · 2024-06-14T13:07:46Z

^ This was fixed in #9915. Needs a rebase

jasagredo · 2024-06-17T11:14:24Z

All tests pass in my machine if using:

Rebased on top of update zlib.cabal location #9915
No symlinks
CLANG64 environment
Installed pkgconf from mingw-w64-clang64-x86_64-pkgconf
Used --io-manager=native in a couple of test-suites

jasagredo · 2024-06-17T11:23:26Z

FTR: I'm using this to undo-redo the symlinks on Windows

andreabedini · 2024-06-17T15:14:16Z

I did the rebase but I some tests failed locally. Perhaps the code has bit-rotted a bit.

jasagredo · 2024-06-17T19:12:36Z

You will have to add a step somewhere to install pkg-config on Windows:

Cannot find pkg-config program. Cabal will continue without solving for

Copilot

Pull request overview

This PR introduces a dedicated per-project cache for the pkg-config database to avoid re-querying pkg-config on every plan change, reducing repeated solver overhead (especially on systems where pkg-config queries are expensive).

Changes:

Add a new getPkgConfigDb caching layer stored in the project cache directory, with a user-facing notice when the pkg-config DB is queried/refreshed.
Refactor project planning to compute and pass pkgConfigDB into the solver/elaboration phases.
Add a testsuite case to validate pkg-config DB caching and invalidation behavior (and adjust an existing test’s golden-output update step).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
`cabal-install/src/Distribution/Client/ProjectPlanning.hs`	Adds per-project file monitor for pkg-config DB and threads cached `pkgConfigDB` into planning phases.
`cabal-install/src/Distribution/Client/ProjectConfig.hs`	Exports and adds `resolveProgramDb` helper for program path overrides.
`cabal-testsuite/PackageTests/MonitorPkgConfig/*`	New test validating pkg-config DB cache reuse + invalidation signals.
`cabal-testsuite/PackageTests/NewUpdate/RejectFutureIndexStates/cabal.test.hs`	Simplifies golden-output rewriting to avoid platform-specific `sed -i` handling.
`changelog.d/pr-9422`	Changelog entry describing the new pkg-config DB caching behavior.

Comments suppressed due to low confidence (2)

cabal-install/src/Distribution/Client/ProjectPlanning.hs:675

The improved/elaborated plan caches do not depend on the pkg-config DB cache. Since pkgConfigDB is computed outside rerunIfChanged and is not included in the cache keys, changes in pkg-config dirs/PKG_CONFIG_PATH can leave fileMonitorImprovedPlan/fileMonitorElaboratedPlan returning a stale plan without re-running the solver/elaboration. Include a pkg-config signature (or a monitored file) in these cache keys so pkg-config changes invalidate the plan caches.

      -- The overall improved plan is cached
      rerunIfChanged
        verbosity
        fileMonitorImprovedPlan
        -- react to changes in the project config,
        -- the package .cabal files and the path
        (projectConfigMonitored, localPackages, progsearchpath)
        $ do

cabal-install/src/Distribution/Client/ProjectPlanning.hs:793

fileMonitorSolverPlan’s cache key does not include pkgConfigDB, even though planPackages depends on it. If the pkg-config DB changes, the solver plan can be reused incorrectly. Add pkgConfigDB (or a stable signature derived from it) to the rerunIfChanged key, or otherwise ensure this monitor is invalidated when the pkg-config DB cache refreshes.

          rerunIfChanged
            verbosity
            fileMonitorSolverPlan
            ( solverSettings
            , localPackages
            , localPackagesEnabledStanzas
            , compiler
            , platform
            , programDbSignature progdb
            )

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-04-13T07:50:59Z


+      progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)


resolveProgramDb is built only from projectConfigLocalPackages, so it ignores the shared/global extra-prog-path (projectConfigProgPathExtra) that configureCompiler includes. This can regress locating pkg-config when it is only available via the shared config; consider constructing this ProgramDb from both shared + local program path settings (or reuse the same logic as configureCompiler).

Suggested change

progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)

projectConfigPrograms =

projectConfigShared projectConfig

<> projectConfigLocalPackages projectConfig

progdb <- liftIO $ resolveProgramDb verbosity projectConfigPrograms

Copilot · 2026-04-13T07:51:00Z

+      progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)
+      monitorFiles (programsMonitorFiles progdb)
+


monitorFiles (programsMonitorFiles progdb) is executed outside any rerunIfChanged action, so these monitored files are not recorded in any file monitor cache and won’t trigger rebuilds. If you intended program changes (e.g. pkg-config executable) to invalidate caches, move the monitoring into the relevant rerunIfChanged body (or add the signature to the key) instead of doing it here.

Copilot · 2026-04-13T07:51:00Z

+-- | ProgramDb with user specified paths
+resolveProgramDb :: Verbosity -> PackageConfig -> IO ProgramDb
+resolveProgramDb verbosity packageConfig = do
+  let extraPath = fromNubList (packageConfigProgramPathExtra packageConfig)
+  programDb <- prependProgramSearchPath verbosity extraPath [] defaultProgramDb
+  let paths = Map.toList $ getMapLast (packageConfigProgramPaths packageConfig)
+  return $ userSpecifyPaths paths programDb


resolveProgramDb only applies packageConfigProgramPathExtra / packageConfigProgramPaths. Callers that need the full project program search path (including ProjectConfigShared.projectConfigProgPathExtra) will miss it. Either extend this helper to incorporate the shared/global extra-prog-path, or rename/document it as “package config only” to avoid accidental misuse (as in ProjectPlanning’s pkg-config lookup).

Copilot · 2026-04-13T07:51:00Z

@@ -11,8 +11,7 @@ main = skipIfCIAndWindows 10230 >> cabalTest (flakyIfCI 9530 $ withProjectFile "
          . resultOutput
        <$> recordMode DoNotRecord (cabal' "update" [])
  -- update golden output with actual timestamp


This sed invocation uses the w cabal.out command, which appends if cabal.out already exists. That can make the test output accumulate across reruns in the same directory. Prefer writing via stdout redirection (e.g. sed ... cabal.out.in > cabal.out) or explicitly removing/truncating cabal.out first.

Suggested change

-- update golden output with actual timestamp

-- update golden output with actual timestamp

shell "rm" ["-f", "cabal.out"]

Copilot · 2026-04-13T07:51:01Z

+  liftIO $ do
+    createDirectoryIfMissing True pkgConfigPath
+    setEnv "PKG_CONFIG_PATH" pkgConfigPath
+
+  cabal' "v2-build" ["--dry-run", "p", "-v2"]


This test mutates the process environment via setEnv, which can leak into subsequent tests in the same process. Prefer using the testsuite’s withEnv helper so the env override is scoped to the cabal invocations (and is automatically restored afterwards).

Copilot · 2026-04-13T07:51:01Z

@@ -0,0 +1,35 @@
+import Distribution.Compat.Environment (setEnv)
+import System.Directory (copyFile, createDirectoryIfMissing, removeDirectoryRecursive)


removeDirectoryRecursive is imported but never used in this test, which adds noise and can produce warnings under -Wall. Remove the unused import (or use it if cleanup is intended).

Suggested change

import System.Directory (copyFile, createDirectoryIfMissing, removeDirectoryRecursive)

import System.Directory (copyFile, createDirectoryIfMissing)

Querying pkg-config for the version of every module can be a very expensive operation on some systems. This change adds a separate, per-project, cache for PkgConfigDB; reducing the cost from "every plan change" to "every pkg-config-db change per project". The cache key is composed by the pkg-config configured program and the list of directories reported by pkg-config's pc_path variable. Co-authored-by: andreabedini <69135+andreabedini@users.noreply.github.com> fix: skip MonitorPkgConfig test on Windows (no pkg-config available) fix(MonitorPkgConfig): provide fake pkg-config for Windows CI fix: address review comments

andreabedini requested review from andreasabel and gbaz November 8, 2023 03:16

geekosaur reviewed Nov 8, 2023

View reviewed changes

gbaz approved these changes Nov 8, 2023

View reviewed changes

chreekat reviewed Nov 8, 2023

View reviewed changes

Comment thread changelog.d/pr-9422 Outdated

andreabedini force-pushed the andrea/pkgconfig-cache branch 2 times, most recently from 6170fe3 to eaf490a Compare November 8, 2023 13:03

elldritch mentioned this pull request Nov 16, 2023

cabal invokes pkg-config on every installed package, so cabal run or cabal install takes 20 minutes on my system #8930

Open

georgefst mentioned this pull request Nov 17, 2023

Allow disabling pkg-config version checks #9458

Open

andreabedini self-assigned this Nov 18, 2023

andreabedini force-pushed the andrea/pkgconfig-cache branch from eaf490a to 20acde7 Compare November 21, 2023 07:30

andreabedini force-pushed the andrea/pkgconfig-cache branch 2 times, most recently from 03e8346 to 89d3ee1 Compare November 27, 2023 07:55

andreabedini force-pushed the andrea/pkgconfig-cache branch from 89d3ee1 to 0f81808 Compare December 4, 2023 10:01

andreabedini commented Dec 4, 2023

View reviewed changes

Comment thread cabal-testsuite/PackageTests/ExtraProgPath/setup.out

andreasabel reviewed Dec 4, 2023

View reviewed changes

Comment thread cabal-testsuite/PackageTests/Backpack/Includes2/cabal-external-target.out Outdated

andreabedini force-pushed the andrea/pkgconfig-cache branch 4 times, most recently from 74e96f0 to 9cf9a17 Compare December 5, 2023 03:48

andreabedini force-pushed the andrea/pkgconfig-cache branch from 9cf9a17 to f97fb81 Compare December 6, 2023 07:12

sebright added the re: pkg-config Concerning pkg-config and pkgconfig-depends constraints label Dec 7, 2023

andreabedini added the attention: needs-review label Dec 8, 2023

andreabedini added attention: needs-help Help wanted with this issue/PR and removed attention: needs-review labels Feb 6, 2024

andreabedini force-pushed the andrea/pkgconfig-cache branch 3 times, most recently from 547f70c to 5b71c97 Compare February 18, 2024 13:53

andreabedini force-pushed the andrea/pkgconfig-cache branch from 5b71c97 to 331f3cd Compare June 17, 2024 15:13

andreabedini mentioned this pull request Jun 21, 2024

Make clear when pkg-config is not present #10122

Merged

5 tasks

andreabedini force-pushed the andrea/pkgconfig-cache branch 2 times, most recently from ac9fab2 to 5749297 Compare June 25, 2024 09:18

Copilot AI mentioned this pull request Apr 10, 2026

Rebase PR #9422 (Add separate cache for getPkgConfigDb) on haskell/cabal master andreabedini/cabal#3

Closed

Copilot AI review requested due to automatic review settings April 13, 2026 07:44

andreabedini force-pushed the andrea/pkgconfig-cache branch from 5749297 to 3f54ae6 Compare April 13, 2026 07:44

Copilot started reviewing on behalf of andreabedini April 13, 2026 07:45 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

andreabedini force-pushed the andrea/pkgconfig-cache branch from 3f54ae6 to 0101ae9 Compare May 28, 2026 08:19

andreabedini force-pushed the andrea/pkgconfig-cache branch from 0101ae9 to b42d236 Compare May 28, 2026 13:43


		progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)

-      progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)
+          projectConfigPrograms =
+            projectConfigShared projectConfig
+            <> projectConfigLocalPackages projectConfig
+      progdb <- liftIO $ resolveProgramDb verbosity projectConfigPrograms

		progdb <- liftIO $ resolveProgramDb verbosity (projectConfigLocalPackages projectConfig)
		monitorFiles (programsMonitorFiles progdb)

	-- update golden output with actual timestamp
	-- update golden output with actual timestamp
	shell "rm" ["-f", "cabal.out"]

		@@ -0,0 +1,35 @@
		import Distribution.Compat.Environment (setEnv)
		import System.Directory (copyFile, createDirectoryIfMissing, removeDirectoryRecursive)

Uh oh!

Conversation

andreabedini commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geekosaur left a comment

Choose a reason for hiding this comment

Uh oh!

andreabedini commented Nov 8, 2023

Uh oh!

geekosaur commented Nov 8, 2023

Uh oh!

gbaz commented Nov 8, 2023

Uh oh!

Uh oh!

mpickering commented Nov 8, 2023

Uh oh!

andreabedini commented Nov 9, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreabedini commented Nov 21, 2023

Uh oh!

Uh oh!

Uh oh!

andreabedini commented Dec 5, 2023

Uh oh!

andreabedini commented Jan 24, 2024

Uh oh!

andreabedini commented Jan 29, 2024

Uh oh!

fgaz commented Feb 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreabedini commented Feb 5, 2024

Uh oh!

michaelpj commented Feb 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreabedini commented Feb 6, 2024

Uh oh!

jasagredo commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasagredo commented Jun 14, 2024

Uh oh!

jasagredo commented Jun 14, 2024

Uh oh!

jasagredo commented Jun 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jasagredo commented Jun 17, 2024

Uh oh!

andreabedini commented Jun 17, 2024

Uh oh!

jasagredo commented Jun 17, 2024

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

andreabedini commented Nov 8, 2023 •

edited

Loading

andreabedini commented Nov 9, 2023 •

edited

Loading

fgaz commented Feb 2, 2024 •

edited

Loading

michaelpj commented Feb 5, 2024 •

edited

Loading

jasagredo commented Jun 10, 2024 •

edited

Loading

jasagredo commented Jun 17, 2024 •

edited

Loading