Update solver Hackage benchmark for v2-install.

[grayjay]

Fixes #6417.

Changes in this commit:

• Set the --lib flag so that the benchmark works for packages with libraries or
  executables.
• Set --ignore-project to prevent cabal projects from affecting the results.
• Handle four types of cabal errors that were introduced by v2-install.
• Use +nowrap to simplify parsing of cabal errors.

---

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.
• If the change is docs-only, [ci skip] is used to avoid triggering the build bots.

Please also shortly describe how you tested your change. Bonus points for added tests!

[phadej]

Nice, --ignore-project immediately in use!

[grayjay]

I noticed your PR when I first pushed my branch. Thanks for adding the flag!

[phadej]

When I try

cabal run hackage-benchmark -- --cabal1 /opt/cabal/3.0/bin/cabal --cabal2 $(cabal-plan list-bin cabal) --packages tree-diff --print-trials

I get

trial/summary    package   result1        result2              mean1       mean2     stddev1     stddev2     speedup
trial            tree-diff Unknown        Solution            0.014s      5.325s

I think that Unknown is a failure, how can I check why it's unknown?

[phadej]

Oh, I know, because cabal-install-3.0 doens't yet know about --ignore-project; that makes sense. (EDIT: I backported that patch, so hopefully cabal-install-3.0.1.0 will know).

[phadej]

Another thing:

On my system, the ~/.cabal/store is huge, which affects the solver "significantly", in the sense that the relative speedups are considerably less.

Is there some mechanism to prevent store from affecting the benchmarks?

Could we create some temporary directory, and use --storedir=tempstore, then cabal won't spend much time reading it.

[phadej]

Something like

diff --git a/solver-benchmarks/HackageBenchmark.hs b/solver-benchmarks/HackageBenchmark.hs
index 145b49f93..df79d361e 100644
--- a/solver-benchmarks/HackageBenchmark.hs
+++ b/solver-benchmarks/HackageBenchmark.hs
@@ -176,7 +176,7 @@ runCabal timeoutSeconds cabal flags pkg = do
     let timeout = "timeout --foreground -sINT " ++ show timeoutSeconds
         cabalCmd =
             unwords $
-            [cabal, "install", "--ignore-project", "--lib", unPackageName pkg, "--dry-run", "-vsilent+nowrap"] ++ flags
+            [cabal, "--store-dir=/tmp/store", "v2-install", "--ignore-project", "--lib", unPackageName pkg, "--dry-run", "-vsilent+nowrap"] ++ flags
         cmd = (shell (timeout ++ " " ++ cabalCmd)) { std_err = CreatePipe }
 
     -- TODO: Read stdout and compare the install plans.

• We should create temporary directory using temporary, rather then hard coding one
• I'd also prefer v2-install.

EDIT: on my machine, from

trial/summary    package   result1        result2              mean1       mean2     stddev1     stddev2     speedup
trial            tree-diff Solution       Solution            4.683s      5.037s

to

trial/summary    package   result1        result2              mean1       mean2     stddev1     stddev2     speedup
trial            tree-diff Solution       Solution            2.561s      2.392s

so this is significant.

[phadej]

In fact, we probably should specify some non-existing environment, as --lib reads the default one.

diff --git a/solver-benchmarks/HackageBenchmark.hs b/solver-benchmarks/HackageBenchmark.hs
index 145b49f93..ed2bdd5f8 100644
--- a/solver-benchmarks/HackageBenchmark.hs
+++ b/solver-benchmarks/HackageBenchmark.hs
@@ -176,7 +176,7 @@ runCabal timeoutSeconds cabal flags pkg = do
     let timeout = "timeout --foreground -sINT " ++ show timeoutSeconds
         cabalCmd =
             unwords $
-            [cabal, "install", "--ignore-project", "--lib", unPackageName pkg, "--dry-run", "-vsilent+nowrap"] ++ flags
+            [cabal, "--store-dir=/tmp/store", "v2-install", "--package-env=hackage-benchmarks-dont-use", "--ignore-project", "--lib", unPackageName pkg, "--dry-run", "-vsilent+nowrap"] ++ flags
         cmd = (shell (timeout ++ " " ++ cabalCmd)) { std_err = CreatePipe }
 
     -- TODO: Read stdout and compare the install plans.

note: --package-env; I think it's ok to hardcode one which is unlikely to exist, as far as it's not the default.

I'm not sure whether it affects significantly the run-times, but it definitely affects the solver results.

There probably should be some comments, why we have so much different flags for seemingly so simple task.

Sorry for the comment spam.

[grayjay]

@phadej Thanks for the suggestions! I didn't realize that so much time was spent reading the store, so I definitely want to skip that. I just specified a directory that shouldn't exist, because cabal shouldn't need to create the store with --dry-run. I also added v2- and --package-env and documented the flags.

[phadej]

Merged in #6447, thanks!