External dependencies: json-cpp, fmtlib, ranges-v3, intx, z3, cvc4

[ekpyron]

Situation:

• We have been talking about dependencies a while recently, but never came to a decision.
• It looks like we like to introduce dependencies on fmtlib and ranges-v3 and maybe intx. There also was the idea to replace json-cpp with nlohmann/json.
• Our repo contains a copy of evmc.
• For json-cpp we use the cmake ExternalProject_Add mechanism
• We also depend on boost and optionally on Z3 and CVC4. For those we use plain standard cmake find_package.

Issues with ExternalProject_Add:

• There is no way to override it and dynamically link against system json-cpp instead.
• make requires a network connection and starts downloading
• The latter complicates e.g. PPA scripts.
• We can't pre-pack dependencies to our CI docker images, increasing build time.

Apart from boost, all the other dependencies except json-cpp (including nlohmann/json as potential json-cpp replacement) are header-only or nearly header-only.

When considering to add new dependencies we could:

• Reuse the ExternalProject_Add mechanism despite its drawbacks.
• Use git submodules for the new dependencies (or all dependencies).
• Instead of submodules use subtrees.
• Switch everything to plain standard cmake find_package, while maintaining a better ìnstall_deps script to locally install dependencies in the build tree.
• Combine find_package with other mechanisms as fallback.

[axic]

We discussed this countless times on gitter, but apparently no record of that here.

I would say this issue should not block any other issues in https://github.com/ethereum/solidity/projects/40, in particular #3851 and #6900.

For those issues to proceed I think it is fine just to use ExternalProject as we do with jsoncpp.

Independently later we could clean up dependency handling. My personal preference is for git subtrees for reasonably sized projects (and intx, nlohmann-json, ranges-v3 all fit this bill).

[christianparpart]

My personal preference is for git subtrees for reasonably sized projects (and intx, nlohmann-json, ranges-v3 all fit this bill).

@axic the good thing about submodules is, you do not have to check them out and can still opt for system installed dependencies.

This ticket is also quite old, and I'm now at some ticket where fmtlib actually does make sense (solidity LSP server) for server side logging. I think I'd just use submodules for now until someone complains in the PR, as I think I can always change that at a later point.

p.s.: I favor submodules for the above reasons and can be also easily integrated into our CI.

[axic]

@christianparpart I am not sure I follow how a git submodule could "opt for system installed dependencies" ?

[cameel]

Here's another idea: why not make our cmake config agnostic to where the libs come from and just support multiple ways of getting them as a separate step before running cmake? This way we could support the C++ package managers that are getting more popular (conan, hunter, vcpkg, etc) as well as manual installation or getting them using system package manger. All without forcing any particular method on the user.

In #12077 we got a request to publish solc on conan and while I think that just having the executable there does not add much, having a conan config in the repo that one could optionally use to get dependencies would be pretty convenient.

My proposal would be this:

• Have CMake find all libraries via find_package() and not download anything by default.
• Add conan config so that users can run conan command before running cmake if they choose to. Conan apparently can install packages in such a way that CMake will be able to find them (it can generate CMake config for them or packages can provide it by themselves).
• Keep the ExternalProject_Add for the current deps as a simple alternative that does not require installing extra tools. Require a special CMake flag to enable it. Alternatively, just use a standalone script for downloading dependencies instead of doing it in CMake config.
• In the future we could add config for other package managers if we find any of them popular enough to be worth supporting.

[christianparpart]

@cameel on Windows CI (and me at home) we're already using vcpkg for installing dependencies. vcpkg is btw also available on all 3 major platforms (win, osx, linux). and now they do also support manifest files.

Personally I'd prefer to not support too many package managers (or ways to tell a project how to find them). vcpkg luckily supports find_package(), I'm not sure about conan though. I personally though had a good experience with CPM, too (which is a successor of ExternalProject_Add and FetchContent module).

[cameel]

Conan does support find_package(). I.e. it can generate the config that lets find_package() find the package or allow the package to provide it by itself.

I have a slight preference towards Conan (vcpkg is very simplistic from what I've read and I'm a bit afraid that it will pull in some heavy MS tooling with it) but I haven't really tried any of them in practice. As long as it's just one of the methods to get packages, I don't have anything against trying it out.

[christianparpart]

I have a slight preference towards Conan (vcpkg is very simplistic from what I've read and I'm a bit afraid that it will pull in some heavy MS tooling with it) but I haven't really tried any of them in practice. As long as it's just one of the methods to get packages, I don't have anything against trying it out.

I wasn't about to convince us all to hard-depend on vcpkg. It just happens to be used by CI on windows (and my personally on my windows). the solidity project does not have a single line of vcpkg outside of CircleCI files :)

[cameel]

@chriseth's opinion from the chat:

I think we should not use a package manager because we should keep our depenencies as small as possible