Support spago.lock files

[thomashoneyman]

The registry-compliant Spago specifies dependencies as either entries in a package set or via explicit version bounds of the form >=X.Y.Z <X.Y.Z, where libraries published to the registry must use version bounds. This means that it's no longer possible to tell the exact versions that a project is using from its package set alone.

Lockfiles solve this problem by listing the exact resolutions a project is currently using. They'll also typically list other metadata sufficient for the package manager to produce a build from the lockfile alone, such as:

• the dependency version (a registry version, a ref, ...)
• the location of the package source (registry packages can omit this, but we'd need this if the package is overridden, such as to a fork)
• the hash of the package contents (commonly called "integrity", and used to ensure the code downloaded exactly matches what the lockfile expects.)

There may be other fields we choose to support, if they are useful to Spago. Selfishly, I'm most interested in the lockfile storing package hashes (which we can get from the registry, or compute in the case of an override) because this is necessary for Nix tooling to be able to support Spago projects without generating additional files. However, lockfiles are useful beyond just making it easier for Nix tools to work.

[f-f]

Lockfiles are not necessary for the situations where one uses only a package set and fixed versions (which is what Spago has been dealing with this whole time), but become unavoidable when using a solver, as one would want to save the plan and not recompute it every time.

[thomashoneyman]

Right. Although you may also want lockfiles for the integrity checks as well even if only using package sets (some of this was previously done with Dhall). But with this in mind it makes sense for lock files to be optional if you are using package sets.

[f-f]

Yeah that's a good point. Right now we write the hash of the package set to the config itself, but we could write it to the lockfile instead.
On the other hand, once the package set is hashed then the build is pretty much locked down so it makes sense for the lockfile to be optional then.

[thomashoneyman]

And the hashing of the package set works even if you have overrides? It’s not just a hash of the upstream package set?

[f-f]

It is a hash of the package set, the overrides are up to the users, which are instructed to not point at branches (because they might move), but only to commits or tags.

That's not a nixified build and things might not be reproducible in certain corner cases, but it's been good enough for users so far, which is why I'm comfortable with having an optional lockfile for package-set builds.

[thomashoneyman]

In #962 we implemented initial support for writing Spago lockfiles, including:

• The lockfile format and tests
• Reading and writing lockfiles from disk
• Determining whether to write a lockfile based on a user's config and whether lock: true is set

We've got a number of things still remaining to do, including:

CLI commands

• Add a --generate-lockfile that pulls all git dependencies, regenerates lockfile
• Add a --pure for build, etc. commands that skips checking if the lockfile needs updating, just uses the lockfile directly.

Read Lockfile

• Determine whether the lockfile needs updating by checking if the 'workspace' section has changed (the dependencies listed for any workspace packages). Note: we do not update mutable references like branch tips indicated in the 'extra packages' by default; only if you do a fresh install or --generate-lockfile.

Fetch

• Read the lockfile and use it to determine all transitive dependencies in fetch

[f-f]

Determining whether to write a lockfile based on a user's config and whether lock: true is set

This is still an approximation - we now entirely avoid generating a lockfile if the user selects a specific package for compilation (with -p), but we should generate it if the user has requested so. There's a TODO left in the code

[thomashoneyman]

Weirdly, Spago is not currently writing a lockfile out unless you have multiple packages in the workspace. If you just have one spago.yaml file containing a workspace and package section, no lockfile gets written.