libhive: client build args and alternative dockerfiles

[marioevz]

Allows specifying a --client-file parameter which shall be read in YAML or JSON format and must contain the client build modifiers.

Supported modifiers:

• client: Name of the client, eg: besu, go-ethereum, etc
• dockerfile: Dockerfile to use. E.g. using dockerfile: git will build using Dockerfile.git instead of the default Dockerfile
• build_args: Build arguments passed to the docker build engine to use when building the image

Supported docker image build arguments depend on the client and the docker image being used, but common client build arguments are:

• tag: Git branch or docker tag name to use during build process.
• github: The repository containing the client source code (for git-based Dockerfiles)
• baseimage: The client image name on DockerHub.

Also adds example dockerfiles for go-ethereum: Dockerfile.git, to build from a git branch, and Dockerfile.local to build from a local source path.

This change should allow clients to introduce different Dockerfiles to test using hive in many different environments, such as the CI.

[spencer-tb]

💯 - I am a big fan of this, it will make testing each client much much easier! Especially with Dockerfile.local :D

[marioevz]

@fjl Another option for the yaml/json files is to have a build_params key which could basically be a dictionary that contains any build parameter passed to the docker build:

- name: go-ethereum
  dockerfile: git
- name: nethermind
  build_args:
    user: nethermindeth
    repo: hive
    branch: latest
- name: go-ethereum
  dockerfile: local
  build_args:
    local_path: ./path/to/local/code/base/

The problem here would be how to tag each image built.

Right now we simply append the branch to the client's name to form the tag (go-ethereum-master), but with an arbitrary amount of parameters this could become cumbersome.

One possibility could be to create the tag using the source client config yaml/json file name and index of the client in the file.

E.g. for a clients.yaml:

- name: go-ethereum
  dockerfile: git
- name: nethermind
  build_args:
    user: nethermindeth
    repo: hive
    branch: latest
- name: go-ethereum
  dockerfile: local
  build_args:
    local_path: ./path/to/local/code/base/

the client images could be named go-ethereum-clients-1, nethermind-clients-2, go-ethereum-clients-3.

[fjl]

This is almost ready now, there's just a small issue left:

Before this change, one could run multiple versions of the same client, which can be useful in sync tests, for example. This feature doesn't work anymore. For example, using this commandline

./hive -client go-ethereum,go-ethereum_v1.11.1 --sim ethereum/sync

both client images will be built, but the simulation will only see one client, "go-ethereum". I need to introduce an algorithm that assigns unique names to the clients to fix this.

We could go with the ClientDesignator.String as the name, but then the names would be kind of long. Might be better to find another method.

[marioevz]

@fjl how about hashing the names of values of the parameters in the yaml files and do a 4-byte concatenation of this to generate an unique name ?

I can implement this if you'd like.

edit: The main issue is that it might be confusing later when you want to check the logs to find the hash of the specific configuration you want.

Another possibly better solution would be to have a name field in the yaml and use that as the name of the image, if not specified, default to concatenate using the method you described.

[marioevz]

@fjl I pushed this change:

- client: nethermind
  image_name: nethermind-4844
  build_args:
    repo: nethermind
    branch: eip-4844-v6

==
image=hive/clients/nethermind-4844:latest
and

- client: nethermind
  build_args:
    repo: nethermind
    branch: eip-4844-v6

==
image=hive/clients/nethermind_branch_eip-4844-v6_repo_nethermind:latest

Let me know what you think, we can do this or think about another approach :)