Race condition updating pantry database?

[chreekat]

General summary/comments

I'm just investigating some failures in GHC CI and I don't know if stack is responsible or not. I've been given two examples, but I'm told there are others from today.

Example 1: https://gitlab.haskell.org/ghc/ghc/-/jobs/1736793
Example 2: https://gitlab.haskell.org/ghc/ghc/-/jobs/1736433

Reasons for thinking it's a stack bug:

1. in the two examples, there is similar behavior. stack notices it is missing data in its pantry database and goes to update it by downloading and processing the Hackage tarball. That seems to succeed, but then stack immediately gives up and complains about missing data for a different package. Did the second package think that no update occurred (since the previous one already had succeeded), and then threw an error?

Reasons why I'm not sure:

1. Not only is it a different pair of packages in both cases (which could be explained by nondeterministic concurrent processing of packages), but in one case js-dgtable gets processed just fine, and in the other case it's one of the packages whose data is missing. How is it that GHC CI has different Pantry databases between runs? I need to investigate this further.

Steps to reproduce

😬 I haven't tried, but something like this might work:

docker run -it --rm registry.gitlab.haskell.org/ghc/ci-images/x86_64-linux-deb10:572353e0644044fe3a5465bba4342a9a0b0eb60e

docker$ git clone https://gitlab.haskell.org/ghc/ghc
docker$ cd ghc
docker$ .gitlab/ci.sh setup
docker$ .gitlab/ci.sh configure
docker$ hadrian/build-stack --version

Stack version

ghc@8659c4aa4286:~$ stack --version
Version 2.9.3, Git revision 6cf638947a863f49857f9cfbf72a38a48b183e7e x86_64 hpack-0.35.1

Method of installation

• Unholy magicks

Platform

• x86_64-linux

[chreekat]

Relevant lines from https://gitlab.haskell.org/ghc/ghc/-/jobs/1736793:

825: No cryptographic hash found for Hackage package extra-1.7.10, updating
<snip>
1397: extra > Registering library for extra-1.7.10..
1398: Error: [S-7282]
1399: Stack failed to execute the build plan.
1400:
1401: While executing the build plan, Stack encountered the following errors:
1402:
1403: Error: [S-922]
1404: No cryptographic hash found for Hackage package random-1.2.1.1

[mpilgrem]

I've just encountered the same sort of thing with the CI at the pantry repository. It is unlikely to be the executable - something adverse must have happened elsewhere.

[mpilgrem]

Pantry error S-922 corresponds to NoHackageCryptographicHash. It gets thrown here:

getHackageTarball ::
     (HasPantryConfig env, HasLogFunc env, HasProcessContext env)
  => PackageIdentifierRevision
  -> Maybe TreeKey
  -> RIO env HackageTarballResult
getHackageTarball pir mtreeKey = do
  let PackageIdentifierRevision name ver _cfi = pir
  cabalFile <- resolveCabalFileInfo pir
  let rpli = RPLIHackage pir mtreeKey
  withCachedTree rpli name ver cabalFile $ do
    cabalFileKey <- withStorage $ getBlobKey cabalFile
    mpair <- withStorage $ loadHackageTarballInfo name ver
    (sha, size) <-
      case mpair of
        Just pair -> pure pair
        Nothing -> do
          let exc = NoHackageCryptographicHash $ PackageIdentifier name ver
          updated <- updateHackageIndex $ Just $ display exc <> ", updating"
          mpair2 <-
            case updated of
              UpdateOccurred -> withStorage $ loadHackageTarballInfo name ver
              NoUpdateOccurred -> pure Nothing
          case mpair2 of
            Nothing -> throwIO exc
            Just pair2 -> pure pair2

Could Hackage be providing packages with missing SHA256 hashes?

[mpilgrem]

It looks to me to be intermittent; I've just re-run my failed pantry CI and, fingers crossed, it is now behaving normally. EDIT: it was intermittent.

[mpilgrem]

I think what may have happened is this: a package version could not be found in the local Hackage package index, so Stack tried to update the Hackage package index from Hackage; for some reason that update failed, so Stack threw an exception. The 'glitch' was likely Hackage not providing its package index, when requested.

[mpilgrem]

Pantry's Haddocks explain:

-- | Download the most recent 01-index.tar file from Hackage and update the
-- database tables.
--
-- This function will only perform an update once per 'PantryConfig' for user
-- sanity. See the return value to find out if it happened.
--
-- @since 0.1.0.0
updateHackageIndex ::
...

I think the origin of the problem experienced may lie in that 'for user sanity ...'.

[chreekat]

Your question about whether Hackage may be returning incomplete data reminded me that I've seen an unusual number of errors from Hackage in CI over the last couple months.

But for this particular problem, what about commercialhaskell/pantry#111? Although I had forgotten about it, it must have been knocking around in my brain, because I apparently gave it a thumbs-up at some point. :P

[chreekat]

I am tempted to close this issue as a duplicate, except it might be nice to have a pointer to it.

[mpilgrem]

@chreekat, agreed. I'll close this as a duplicate.