ethtest: support block chain & transactions tests

[winsvega]

I seems that currently GO doing only state tests by ethtest tool.
I am planning to expand our fuzz testing from just only state tests to other test suites as well.
(https://github.com/ethereum/tests/wiki)

As we need GO, Python and Cpp clients be in consensus, they should pass all the tests suites. Also we are running fuzz test server in attempt to discover new possible bugs, so we need go ethtest tool to be able to parse other tests.

[CJentzsch]

Basically, we need an option for the ethtest executable so we can define whether a VMTests, StateTest, TransactionTests or BlockChainTest will follow.

[winsvega]

VMTests needed too.

[tgerring]

Is there any documentation on how this is currently run? Is it automated in any way or manual only?

[winsvega]

https://github.cohttps://github.com/ethereum/tests/wikim/ethereum/tests/wiki
TestTypes

[tgerring]

I see the specifications, but I couldn't find any documentation on the wiki about syntax or how ethtest is being used either in automated or manual environments.

I've made some changes on a local branch so that it accepts a test type parameter, but the internal function and type names (Vm) don't match what you're saying that only state tests are currently supported. It might just be a code cruft issue, but I need verification from someone who is actively using it.

[CJentzsch]

We are currently using it for the fuzz testing with state tests. Here we do ethtest '<content of json state test>'. It returns 0/1 for success/failure. We run thousands of tests a day with it. Now we want to expand that to the other test suites. Most importantly transaction tests and blockchain tests. We want to do something like: ethtest -txtest '<content of tx test json>' . Similiar for BlockChainTests. Since go is already able to run Transactiontests and BlockChainTests (https://github.com/ethereum/tests/tree/develop/TransactionTests , https://github.com/ethereum/tests/tree/develop/BlockTests and specified here: https://github.com/ethereum/tests/wiki (see test tyspes on the right)), it should not be too much changes. Only the way go gets the tests and how it does report back for success/failure needs to be adopted.

[obscuren]

gustav is the right person for the job. He'll get around to it once he's finished up his other tasks

[CJentzsch]

@Gustav-Simonsson , let me know when you need some help or clarification.

[tgerring]

@CJentzsch @winsvega I've added initial integration of all test types to the ethtest tool in a branch (see linked PR).

The syntax is currently ethtest <testType> <path/to/dir/or/file>, for example ethtest tx ./files/TransactionTests/ will run the transactions tests on all the files in that directory or the file itself. The valid test types are as follows (either form works):

case "vm", "VMTests":
case "state", "StateTest":
case "tx", "TransactionTests":
case "bc", "BlockChainTest":

I can continuing working on the PR, so please provide any feedback about your scenarios, etc. :)

[tgerring]

CLI has been updated and ethtest will run all 4 defined tests if tun from the root of the tests directory without any switches. Some examples:

Run all tests from current directory, looking in their respective sub directories for json files:
ethtest

Run all VM json tests from ./VMTests/ directory
ethtest --test "vm" --file "./VMTests/"

Run all tests in a cousin directory supplied by environment var
ETHEREUM_TEST_PATH="../../tests/files" ethtest --test "all"

Run a single transaction test
ethtest --test "tx" --file "./TransactionTests/ttTransactionTest.json"

Flags:

   --test "all"     Test type (string): VMTests, TransactionTests, StateTests, BlockTests
   --file "."       Test file or directory. Directories are searched for .json files 1 level deep [$ETHEREUM_TEST_PATH]
   --continue       Continue running tests on error (true) or exit immediately (false)