Skip to content

Conversation

@nkuba
Copy link
Member

@nkuba nkuba commented Jun 7, 2022

Closes #3005
Depends on keep-network/keep-common#88
Depends on threshold-network/tbtc-v2#376

TODO:

Introduction

This PR implements Go bindings generation for Solidity contracts:

  • Threshold Network TokenStaking
  • SortitionPool
  • RandomBeacon
  • WalletRegistry
  • TBTCv2 Bridge

SortitionPool

Please note that bindings for SortitionPool are generated separately for random-beacon and ecdsa as those projects can use different dependency versions.

Contracts Artifacts

This PR changes the approach to what we consider the source for code generation.
Previously we were generating Go bindings from solidity contract sources which was causing a lot of headaches in regards to setting up solc compiler just for this task.
What is more, there are problems with solc support on machines that run on Apple M1 processors, which some folks in the team transitioned to.
Instead of generating ABIs from sources, we use contracts artifacts that are being published as part of NPM packages. We download the artifacts to _contracts directory. The directory name is prefixed with the underscore (_), so it is ignored by the go tool.

V1 Contracts

We will be removing parts of V1-related code from the client bit by bit. In this PR we remove the generation of the binding for V1 contracts. We leave the already generated code and use it in the docker image building process. This simplifies the docker build and reduces the build time.

Execution

Ensure npm at least v7.x is installed and run:

go generate ./...

nkuba added 6 commits June 7, 2022 14:15
The common Makefile is based on the Makefile used for go bindings
generation for contracts from V1.

It is a common file that should be included in other Makefiles for
projects.

The main difference is that solc generation was extracted to a function
that expects dependencies imports mapping and source file as arguments.
Configuration for Go bindings generator for ECDSA contracts.
Configuration for Go bindings generator for Random Beacon contracts.

It also contains contracts pulled as dependencies: Token Staking and
Sortition Pool.
Configuration for Go bindings generator for TBTC v2 contracts.
Configuration for Go bindings generator for Threshold Network contracts.
We don't want to push files installed from npm like node_modules.
@nkuba nkuba changed the title Go bindings for v2 contracts Generate Go bindings for V2 contracts Jun 7, 2022
@nkuba nkuba self-assigned this Jun 7, 2022
@nkuba nkuba added this to the v2.0.0-m1 milestone Jun 7, 2022
nkuba added 9 commits June 9, 2022 22:41
We don't need to define the paths to the solidity source files as this
can be simplified and we can look for the contract file names.
We can use the default `contracts/` directory for internal projects, but
for external ones we need to define the path inside the `node_modules`.
As `**/*` pattern is not supported we have to define subdirectories for
recursive search.
To ignore a directory in go build we have to prefix it with the
underscore.
Move code out ouf `v2` directory.
Mark the direcotries that has to be revisited when working on V2 and
removed or refactored.

We leave the code for now so we don't break the build.
nkuba added 2 commits June 23, 2022 12:03
Updated the dependendcy to main branch commit after
keep-network/keep-common#88 is merged.
This commit contains bindings generated by `go generate` command for V2
solidity contracts.
@nkuba nkuba marked this pull request as ready for review June 23, 2022 10:31
@dimpar
Copy link
Contributor

dimpar commented Jun 27, 2022

Are you sure we don't miss anything in go.mod to make go generate ./... work? I had the following error:

Screenshot 2022-06-27 at 12 47 24

I did the suggested commands above and then it worked.

nkuba added 10 commits June 28, 2022 17:01
For go bindings generation we need contracts artifacts.
Npm pack outputs a filename that we can use in tar command to extract
it.
Version 7.x introduced `pack-destination` flag that we use.
We don't need to set GO111MODULE as it's already set in the version of
go we use (1.18)
We download the artifacts for V2 contracts in the docker image building
process.

There is an optional argument `CONTRACTS_NPM_PACKAGE_TAG` that can be
set to a tag with which npm packages should be downloaded (e.g. ropsten,
mainnet). By default if the argument is not set the `development`
artifacts will be downloaded.
It's tiny change, but looks cleaner.
We use existings bindings that are commited to the repository under
`pkg/chain/gen` instead of generating the new ones.
We don't want to regenerate V1 bindings. We leave the existing code so
the build is not broken.
The outputs is not needed.
@nkuba nkuba requested a review from pdyraga June 28, 2022 22:13
@dimpar
Copy link
Contributor

dimpar commented Jun 29, 2022

Are you sure we don't miss anything in go.mod to make go generate ./... work? I had the following error:
Screenshot 2022-06-27 at 12 47 24
I did the suggested commands above and then it worked.

Do you use Go 1.18?

go version go1.18.3 darwin/amd64

After the recent changes, it works now.

@dimpar
Copy link
Contributor

dimpar commented Jun 29, 2022

I was able to successfully download artifacts, generate bindings and build an image from Dockerfile. Looks good.
go version go1.18.3 darwin/amd64
npm v8.13.1

@pdyraga
Copy link
Member

pdyraga commented Jun 29, 2022

👀

nkuba added 4 commits June 29, 2022 15:09
Having a separate bash script for getting the artifacts is another
complication. To simplify the generation we include the download
artifact in the go generate command execution.
It makes sense to pull the artifacts each time the go generate is run as
we want to build the fresh bindings in case the packages were updated.
We want to force the rm so it doesn't error if the directories don't
exist.
@pdyraga
Copy link
Member

pdyraga commented Jun 29, 2022

Potential improvements we discussed:

  1. We could have a global make clean that would execute a cleanup on each level of gen/. Right now, to regenerate files from scratch, we need to:

    cd pkg/chain
    cd random-beacon/gen && make clean && cd ../..
    cd ecdsa/gen && make clean && cd ../..
    cd threshold-network/gen && make clean && cd ../..
    cd tbtc-v2/gen && make clean && cd ../..
    

    Captured in Global cleanup for smart contract Go bindings generator #3049.

  2. We need to ensure in future all NPM packages used by the client are tagged with testnet (or ropsten/kovan/...) and mainnet. CONTRACTS_NPM_PACKAGE_TAG environment variable determines the ABI version to download based on a common tag. All dev packages are tagged with development. We need tags for mainnet and testnet as well. Packages affected are: @keep-network/random-beacon, @keep-network/ecdsa, @keep-network/tbtc-v2, @threshold-network/solidity-contracts.

@pdyraga pdyraga enabled auto-merge June 29, 2022 13:35
@pdyraga pdyraga merged commit c2f1e9d into main Jun 29, 2022
@pdyraga pdyraga deleted the go-bindings-for-v2-contracts branch June 29, 2022 13:38
@pdyraga pdyraga modified the milestones: v2.0.0-m0, v2.0.0-m4 Sep 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regenerate contract bindings

4 participants