NixOS support (and other FHS-challenged distros)

[shazow]

Problem

Right now the shipped binaries make assumptions about where dynamically-linked libraries live. This is a problem for NixOS and other Linux distros that use less usual FHS layouts (there are dozens of us!).

1. The binaries that ship in the release are not statically linked, so there's several shared libraries that aren't loaded successfully (I suspect this can be patchelf'd in a pinch)

$ ldd forge
	linux-vdso.so.1 (0x00007ffe0b58f000)
	libssl.so.1.1 => not found
	libcrypto.so.1.1 => not found
[...]

2. forge build downloads a solc binary which suffers from the same problem.

$ forge build
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]
$ ldd /home/shazow/.svm/0.8.2/solc-0.8.2
	linux-vdso.so.1 (0x00007ffec07b9000)
	libdl.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libdl.so.2 (0x00007fe833da6000)
	libm.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libm.so.6 (0x00007fe833c65000)
	libc.so.6 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib/libc.so.6 (0x00007fe833aa0000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/s9qbqh7gzacs7h68b2jfmn9l6q4jwfjz-glibc-2.33-59/lib64/ld-linux-x86-64.so.2 (0x00007fe834965000)

I tried using forge build --no-auto-detect but looks like it's refusing to use my local version.

$ which solc
/nix/store/5q9xn2s9pjav5fsh7qg8nj94gnj3jqdg-solc-0.8.2/bin/solc
$ forge build --no-auto-detect
compiling...
Error:
   0: "/home/shazow/.svm/0.8.2/solc-0.8.2": No such file or directory (os error 2)
[...]

Solutions

1. The low hanging fruit would be to build the shipped binaries as more statically linked than they are now. I do this for some other projects I maintain (mainly Go), it helps reduce all kinds of problems (can make it runnable on Alpine and other minimalist distros!) in exchange for larger binaries.

2. The more sustainable solution is to rely less on out-of-band binary fetching, and more on distro-friendly pre-packaging.

I understand that the current state of the project is in flux and perhaps the main demographic is people who copypasta curl-bash into their terminals (per the README), but realistically it does not need to be a big change to make this more distro packaging friendly. Even if we're not ready to start tagging releases, small things like changing the default Cargo dependencies to load specific versions rather than git main would go a long way (e.g. ethers-rs). Happy to send some PRs to this effect if it sounds desirable. :)

[onbjerg]

Unfortunately we can't statically link because of libusb. We're going to add packages on different package managers (apt, pacman, yum, brew are on my list), are there others you think we should have on our list?

[shazow]

I'd love nix support. :)

If you could include a flake.nix derivation, so that any distro that supports the nix package manager (basically everything) will be able to automagically build foundry from source in one command. This would also make it trivial to add to the nixpkgs package set as a full derivation (which would also give binary caches for free).

I started some work on this here, but still needs more work: shazow#1

Somewhat related to #315 but I wanted to clarify that you can have this declaration in a generic way that does not require any further maintenance (unless your build process changes substantially), and you don't need to use it to manage your releases or anything like that.

[Edit] The other option is to make a derivation that just downloads the latest binary releases, has all the correct inputs, and tries to patchelf the broken ldd's. It's not preferred but there are lots of binary-only packages that do this.

[Edit2] Worth noting that the solc problem will still be a thing, though. (And any other on-demand fetched binaries.)

[onbjerg]

I'll note this but do note that it is not top of my priority list currently. Feel free to open up a PR if you feel this is pressing, keeping in mind that we want to support releasechannels where possible (stable, nightly) and fixed versions for multiple package repositories, so maintainability is key here 😄

[shazow]

I'll try to make it work out of band somehow, but right now none of this works on any of my devices unfortunately. So it is equally as pressing as me (or any other NixOS/etc user) wanting to use foundry. 😅

[shazow]

One more note: I compiled forge etc from source, and I'm able to reliably patch the svm-downloaded solc binaries with patchelf --set-interpreter "$(cat $NIX_CC/nix-support/dynamic-linker)" ~/.svm/0.8.2/solc-0.8.2 for example, but unfortunately that changes the hash of the file which prompts svm to overwrite it before building again and putting me back in square one.

I'd love for there to be a place to disable that svm behaviour, or an alternative approach.

[gakonst]

@shazow all good points. You can force local solc version usage (under PATH or SOLC_PATH) with --no-autodetect when building or testing or disable them via the foundry toml.

Am supportive of making svm-rs more configurable, so that the paths issue does not manifest. cc @roynalnaruto

Further removing dynamic libraries seems hard, I've given it a lot of effort so far, but if you have any ideas for it I'd also be supportive, if we can let the user choose from static/dynamically linked to not inflate binary sizes too much or get us in violated license land.

[roynalnaruto]

@shazow Hi, if I'm not wrong, you are probably getting this error? (You could be sure by enabling tracing with RUST_LOG=ethers=trace and check if the messages are the same as here)
https://github.com/gakonst/ethers-rs/blob/e0ee03328332776f5f18a52b968ea3f62df982cb/ethers-solc/src/resolver.rs#L408-L413

Which would re-fetch the solc binary and your changed dynamic loader is reset?

Could you try the SOLC_PATH with --no-auto-detect flag?

I am not sure how to configure svm-rs in order to support NixOS, but would be happy to take any suggestions from you or if you could PR to the repo?

[gakonst]

(We just fixed a bug with --no-auto-detect btw #560)

[shazow]

Thanks, I'll give it a try soon!

[shazow]

Confirmed that the --no-auto-detect flag works as expected now.

I'm not sure if something like svm-rs is fixable for NixOS and similar distros. By nature, non-statically-linked binaries don't work. We have decent tooling for patching the dynamically linked paths, and one idea would be to allow for a hook to run on install but that modifies the hash so hash-based version detection goes out the window.

In a full Nix-ified version of this, you would specify the derivation and companion versions you want available, and nix would make sure it's available for you. If foundry has provisions for enabling this scenario (ie. allow for multiple versions of solc to be available for selection but without necessarily relying on a binary fetcher), then I think the rest can be done in the nix packaging out of band.

Would it be possible for it to be an ENV var rather than a flag with every command?

[Edit] Also would be even nicer if the solc version selection still works if present (perhaps via sub-path? or ENV var?) but it doesn't rely on hashes and fetching.

[shazow]

One more update: I made a stand-alone nix flake which uses the binary releases, it lives here for now (good chance I'll move it eventually): https://github.com/shazow/nixfiles/blob/master/flakes/foundry/flake.nix

This means that anyone who has nix installed can run nix develop git+https://github.com/shazow/nixfiles?dir=flakes/foundry to enter a shell that has forge and cast that has been patchelf'd to the current system.

A few caveats:

• As expected, it's pinned to specific release binaries/hashes (nix requires this). It would need to be updated with every release (this can be scripted).
• It doesn't work as a build input for other flakes yet, I think that's because I tossed it into my repo with my other nix configs and flakes don't like that apparently. :)