Monorepo, Workspaces, MSRV & CI

[nothingmuch]

The purpose of this post is to recap the current status, pointing out some inconsistencies or flaws that might be points of friction, and generate discussion WRT next steps for switching the project over to a monorepo workflow.

Unfortunately these thoughts are not very well organized. Almost nothing is urgent and not everything is important, or even actionable, so we should triage these and make issues as appropriate, focusing on improvements that would reduce barriers to contribution or improve our workflows.

Motivation for monorepo

Based on previous conversations, we've come to agreement that at least some of our separate repositories should be added to the rust-payjoin repo. A list of candidate repositories:

1. ohttp-relay - strong motivation, we use it in payjoin-test-utils, and previously needed to update it in lockstep with the directory and/or with the payjoin-cli client
2. (bitcoin-)ohttp - strong motivation, direct dependency of payjoin crate and ohttp-relay, multihop OHTTP would certainly require some API breaking changes
3. bitcoin-hpke - strong, same as ohttp
4. bitcoin_uri{,-ffi} - weak motivation, should not be tightly coupled to the payjoin project but standalone
5. payjoin{,devkit}.org - weak, updating documentation in lockstep doesn't seem that valuable

Moving to a monorepo is primarily to facilitate making atomic changes to related bits of software, instead of staging them over multiple pull requests and potentially crate releases as we have done in the past with the directory, relay and payjoin crate.

Another motivation is to provide unified nix development environments that "just work". Having to maintain a separate flake.nix for each crate means less overhead, esp. for things like scheduled CI jobs to update lockfiles. Not needing to repeat boilerplate tasks across multiple repos would be preferred, and although we can provide a reusable nix flake that dramatically reduces readability and increases complexity, necessitating deeper understanding of nix the language for contributions which would likely be self defeating.

Though BIP 77 related code is not likely to require backwards incompatible changes anymore, other things on our roadmap will definitely require such changes, minimizing the toil of experimentation should be beneficial for v3 work. For example multi-hop OHTTP would require changes to all of the related crates.

MSRV requirements, rustup

With the payjoin crate, and consequentially test-utils and perhaps to a lesser extent ffi we want to support the old MSRV that is reasonable, at least for the default features.

For the directory & cli, as well as the relay, we ship a program so we have more flexibility to require more recent toolchains.

This means we may need to track two (or more) MSRVs in the future, and so we can't really refer to msrv in the context of the repo without being ambiguous.

In the future we may also have more toolchain components, e.g. wasm. rustup's rust-toolchain.toml file is also supported by the nix rust overlay, but afaik there is no way to specify more than one supported version.

Even with a rust-toolchain.toml file unless we specify an exact version, only flake.lock can guarantee reproducibility.

Cargo.lock

For programs, including a Cargo.lock is typically recommended.

Currently we provide a Cargo-minimal.lock and Cargo-recent.lock for the rust-payjoin workspace, which afaik have always been the same. CI uses these to ensure tests pass with both. The main reasons for hypothetical deviations would be MSRV requirements of upstream dependencies.

Multiple Cargo Workspaces?

Due to the varying msrv requirements, should we set up separate cargo workspaces under a single monorepo?

CI

Currently for each of 1.63 (MSRV), stable and nightly we run the contrib/test.sh script. This iterates through the recent and minimal lockfiles, and runs contrib/test.sh in each crate in the workspace separately, which in turn run variations on cargo test typically with --locked.

contrib/test_local.sh just uses the existing Cargo.lock file. It does not run in CI. contrib/coverage.sh is checked in a separate run, only with Cargo.lock. Neither test_local.sh nor coverage.sh use --locked.

FFI only has python based tests in a separate github workflow.

Ideally we would also:

• use scheduled tasks to update lockfiles and check for things breaking due to rust nightly changes or updated upstream dependencies
  • as automated PRs to update Cargo-recent.lock and/or flake.lock?
• test all commits, not just HEAD commit of a branch to aid bisectability
• test all important FFI host languages. see also How shall we bind `payjoin` to flutter? #673
• build docker images for services & cli on each merged PR (and deploy to testing.payjo.in?)

nix flake & flake.lock

The included flake.nix provides devshells for msrv, stable & nightly rust toolchains, packages, and flake checks.

nix flake check could be used to power CI, but github's caching is no longer available, so this would require some nix caching service for fast feedback (otherwise the workspace dependencies will continually be rebuilt).

Note that nix flake check will always be slower than cargo test since it executes in a sandbox, so it is not a substitute for contrib/test{,_local}.sh.

On nixos, the bitcoind crate needs nix-ld or BITCOIN_EXE env var set. We could standardize this for all OSs, so that when using a devshell the nix provided one is used instead of downloading a release, but this means that flake.lock determines the version of bitcoin core that would be used in tests run under the dev shell which might be inconsistent with the CI environment.

Nix can also be used to build minimal docker containers, making these more reproducible than Dockerfiles without using explicit hashes (which even with Cargo.lock, these may have under-specified build toolchain dependencies, whereas flake.lock will cover the exact toolchain version).

Eventually can also consider shipping a reusable nixos module for the directory & relay services. This would allow nixos machines to serve the service with ~5 lines of config (flake input, adding the module, enabling the module and setting the hostname & email required for ACME).

.envrc

This should be renamed to envrc.default or somesuch, to allow customization of .envrc without accidental commits.