Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .envrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
use flake .
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,18 @@ cargo update -p zstd-sys --precise 2.0.8+zstd.1.5.5

## Contributing

### Nix Development Shells

Where nix is available (NixOS or
[otherwise](https://determinate.systems/nix-installer/)), development shells are provided.

The default shell uses rust nightly, and can be activated manually using `nix
develop` in the project root, or automatically with
[direnv](https://determinate.systems/posts/nix-direnv/).

To use the minimal supported version, use `nix develop .#msrv`. `.#stable` is
also provided.

### Testing

We test a few different features combinations in CI. To run all of the combinations locally, have Docker running and run `contrib/test.sh`.
Expand Down
12 changes: 5 additions & 7 deletions contrib/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,13 @@ set -e
DEPS="recent minimal"
CRATES="payjoin payjoin-cli payjoin-directory"

for dep in $DEPS
do
for dep in $DEPS; do
cargo --version
rustc --version

# Some tests require certain toolchain types.
NIGHTLY=false
STABLE=true
export NIGHTLY=false
export STABLE=true
if cargo --version | grep nightly; then
STABLE=false
NIGHTLY=true
Expand All @@ -22,10 +21,9 @@ do

cp "Cargo-$dep.lock" Cargo.lock

for crate in $CRATES
do
for crate in $CRATES; do
(
cd $crate
cd "$crate"
./contrib/test.sh
)
done
Expand Down
7 changes: 3 additions & 4 deletions contrib/test_local.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ CRATES="payjoin payjoin-cli payjoin-directory"
cargo --version
rustc --version

for crate in $CRATES
do
for crate in $CRATES; do
(
cd $crate
cd "$crate"
./contrib/test.sh
)
done
done
2 changes: 1 addition & 1 deletion contrib/update-lock-files.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ for file in Cargo-minimal.lock Cargo-recent.lock; do
cp -f "$file" Cargo.lock
cargo check
cp -f Cargo.lock "$file"
done
done
98 changes: 98 additions & 0 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

186 changes: 186 additions & 0 deletions flake.nix
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I regret to inform you running nix flake check --print-build-logs is not intuitive. Is it supposed to be for the nixpilled? IDK what crane does or what any of these changes do

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

crane parses cargo packages and produces nix package derivation for them

it works in stages:

  1. first it builds all of the dependencies of the workspace
  2. then it builds individual crate packages, with access to the previously built deps
  3. finally it provides various lints and test derivations, exposed here via nix flake check, some of which reuse the outputs of steps 1 or 2

the checks can be done individually or separately, and are quite costly since they build everything both with nightly and msrv, but should be faster to run after minor code changes due to the caching of the workspace dependencies (caching also works in CI potentially)

the main advantage over shell scripts is that nix can provide a lockfile (flake.lock) for all of the various unix dependencies (anything packaged in nixpkgs), in this case shfmt and shellcheck for example, ensuring that we can use that stuff without dealing with github's bloated and brittle ubuntu images.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like some interesting and convenient tooling to tinker with and make our dev environments and CI more reliable, AND most importantly giving new devs a path to easily get started contributing. But perhaps it's not the glaring priority of other broken parts preventing more integrations from implementing a maintainable payjoin stack?

Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
{
description = "rust-payjoin";

inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
crane.url = "github:ipetkov/crane";
};

outputs = {
self,
nixpkgs,
flake-utils,
rust-overlay,
crane,
}:
flake-utils.lib.eachDefaultSystem (
system: let
pkgs = import nixpkgs {
inherit system;
overlays = [rust-overlay.overlays.default];
};

msrv = "1.63.0";
rustVersions = with pkgs.rust-bin;
builtins.mapAttrs (_name: rust-bin:
rust-bin.override {
extensions = ["rust-src" "rustfmt"];
})
{
msrv = stable.${msrv}.default;
stable = stable.latest.default;
nightly = nightly.latest.default;
Comment on lines +35 to +37
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we document how to run these devShells? nix develop .#$VERSION was not intuitive to me

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if it makes sense in README or some sort of dev documentation, but either or should contain some examples and discuss direnv, the devshells, and the checks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definitely fixes the problem of having the right toolchain for testing so I ask that that gets documented in README and just that. Everything else is neat but doesn't exactly solve a particular problem we've experienced yet afaict

};

# Use crane to define nix packages for the workspace crate
# based on https://crane.dev/examples/quick-start-workspace.html
# default to nightly rust toolchain in crane, mainly due to rustfmt difference
craneLibVersions = builtins.mapAttrs (name: rust-bin: (crane.mkLib pkgs).overrideToolchain (_: rust-bin)) rustVersions;
craneLib = craneLibVersions.nightly;
src = craneLib.cleanCargoSource ./.;
commonArgs = {
inherit src;
strictDeps = true;

# provide fallback name & version for workspace related derivations
# this is mainly to silence warnings from crane about providing a stub
# value overridden in per-crate packages with info from Cargo.toml
pname = "workspace";
version = "no-version";

# default to recent dependency versions
# TODO add overrides for minimal lockfile, once #454 is resolved
cargoLock = ./Cargo-recent.lock;

# tell bitcoind crate not to try to download during build
BITCOIND_SKIP_DOWNLOAD = 1;
};

cargoArtifacts = craneLib.buildDepsOnly commonArgs;
individualCrateArgs =
commonArgs
// {
inherit cargoArtifacts;
doCheck = false; # skip testing, since that's done in flake check
};

fileSetForCrate = subdir:
pkgs.lib.fileset.toSource {
root = ./.;
fileset = pkgs.lib.fileset.unions [
./Cargo.toml
(craneLib.fileset.commonCargoSources subdir)
];
};

packages =
builtins.mapAttrs (
name: extraArgs:
craneLib.buildPackage (individualCrateArgs
// craneLib.crateNameFromCargoToml {cargoToml = builtins.toPath "${./.}/${name}/Cargo.toml";}
// {
cargoExtraArgs = "--locked -p ${name} ${extraArgs}";
inherit src;
})
) {
"payjoin" = "--features v2,send,receive";
"payjoin-cli" = "--features v1,v2";
"payjoin-directory" = "";
};

devShells = builtins.mapAttrs (_name: craneLib:
craneLib.devShell {
packages = with pkgs; [
cargo-edit
cargo-nextest
cargo-watch
rust-analyzer
];
})
craneLibVersions;

simpleCheck = args:
pkgs.stdenvNoCC.mkDerivation ({
doCheck = true;
dontFixup = true;
installPhase = "mkdir $out";
}
// args);
in {
packages = packages;
devShells = devShells // {default = devShells.nightly;};
formatter = pkgs.alejandra;
checks =
packages
// {
payjoin-workspace-nextest = craneLib.cargoNextest (commonArgs
// {
inherit cargoArtifacts;
partitions = 1;
partitionType = "count";
# TODO also run integration tests
# this needs --all-features to enable io,_danger_local_https features
# unfortunately this can't yet work because running docker inside the nix sandbox is not possible,
# which precludes use of the redis test container
# cargoExtraArgs = "--locked --all-features";
# buildInputs = [ pkgs.bitcoind ]; # not verified to work
});

payjoin-workspace-nextest-msrv = craneLibVersions.msrv.cargoNextest (commonArgs
// {
cargoArtifacts = craneLibVersions.msrv.buildDepsOnly commonArgs;
partitions = 1;
partitionType = "count";
});

payjoin-workspace-clippy = craneLib.cargoClippy (commonArgs
// {
inherit cargoArtifacts;
cargoClippyExtraArgs = "--all-targets --all-features --keep-going -- --deny warnings";
});

payjoin-workspace-doc = craneLib.cargoDoc (commonArgs
// {
inherit cargoArtifacts;
});

payjoin-workspace-fmt = craneLib.cargoFmt (commonArgs
// {
inherit src;
});

nix-fmt-check = simpleCheck {
name = "nix-fmt-check";
src = pkgs.lib.sources.sourceFilesBySuffices ./. [".nix"];
nativeBuildInputs = [pkgs.alejandra];
checkPhase = ''
alejandra -c .
'';
};

shfmt = simpleCheck rec {
name = "shell-checks";
src = pkgs.lib.sources.sourceFilesBySuffices ./. [".sh"];
nativeBuildInputs = [pkgs.shfmt];
checkPhase = ''
shfmt -d -s -i 4 -ci ${src}
'';
};
Comment on lines +166 to +173
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is an option it needs to be a CI check (part of ./contrib/lint.sh?) otherwise we're gonna get fat diffs with lots of formatting changes between diffs that don't run shfmt

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

./contrib/lint.sh would be tricky without nix since installing shfmt is less standard

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just track it


shellcheck = simpleCheck rec {
name = "shell-checks";
src = pkgs.lib.sources.sourceFilesBySuffices ./. [".sh"];
nativeBuildInputs = [pkgs.shellcheck];
checkPhase = ''
shellcheck -x ${src}
'';
};
};
}
);
}
Loading