Universal specs for JSON-RPC API methods

[cdetrio]

Universal specs for JSON-RPC API methods

This is a pre-draft outline of a Hive-based multi-client test suite for the JSON-RPC API. The specification format is JSON schema (inspired by the Kodi API description).

Existing RPC test suites include ethereum/rpc-tests and bas-vk/hive/tree/rpc. The former is based on mocha.js and the latter on go-ethereum's ethclient.Client api. This proposal intends to be a language and client-agnostic test and specification format.

Here's an example JSON schema specification, for eth_getBlockTransactionCountByHash, annotated for explanation:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "id": "eth_getBlockTransactionCountByHash",
    "title": "eth_getBlockTransactionCountByHash",
    "description": "eth_getBlockTransactionCountByHash JSON-RPC method request and response schema.",

    "request": {
      "id": "#request",
      "allOf": [ // allOf means the request must pass both sub-schemas
        { "$ref": "jsonrpc-request.json" }, // first-pass generic validation, see file https://github.com/cdetrio/interfaces/blob/master/rpc-specs-tests/schemas/jsonrpc-request.json
        { "$ref": "#/definitions/request-obj" } // second-pass validation, specific to the RPC method
      ],
      "definitions": {
        "request-obj": {
          "id": "#request-obj",
          "properties": {
            "method": {
              "type": "string",
              "enum": ["eth_getBlockTransactionCountByHash"]
            },
            "params": {
              "type": "array",
              "items": [
                {
                  "type": "string",
                  "description": "DATA, 32 Bytes - Hash of a block."
                }
              ]
            }
          }
        }
      }
    },

    "response": {
      "id": "#response",
      "allOf": [
        { "$ref": "jsonrpc-response.json" },
        { "$ref": "#/definitions/response-obj" }
      ],
      "definitions": {
        "response-obj": {
          "id": "#response-obj",
          "properties": {
        		"result": {
              "type": "string",
              "description": "QUANTITY - integer of the number of transactions in this block."
        		}
        	}
        }
      }
    }
}

The description fields are not used for validation, but for generating RPC API method documentation as seen on the wiki.

The JSON schema specifications are paired with test cases. Here's an example file with test cases for eth_getBlockTransactionCountByHash:

{
  "title" : "eth_getBlockTransactionCountByHash",

  "schema": {
    "$ref": "../schemas/eth_getBlockTransactionCountByHash.json"
  },

  "chainConfig" : {
    "$ref": "../configs/bcRPC_API_Test.json"
  },

  "tests": [
    {
      "title": "eth_getBlockTransactionCountByHash for block with one tx",
      "request" : {
        "method" : "eth_getBlockTransactionCountByHash",
        "params" : ["0x4e9a67b663f9abe03e7e9fd5452c9497998337077122f44ee78a466f6a7358de"]
      },
      "expectedResponse" : {
        "result": "0x1"
      },
      "asserts": [
        {
          "description": "response is not empty",
          "program": ".receivedResponse.result != null"
        },
        {
          "description" : "transaction count should be equal",
          "program" : ".receivedResponse.result == .expectedResponse.result"
        }
      ]
    }
  ]
}

The asserts array specifies jq programs or "filters", which are used to compare the received response against the expected response.

The chainConfig option points to a set of blocks/transactions to import. The current chain configuration is inherited from ethereum/rpc-tests, which in turn borrows from the state tests repo. This configuration file is loaded by the test runner, and imported by the hive clients.

After hive runs the RPC tests against a set of clients, the results are displayed as a compatibility table: http://cdetrio.github.io/eth-compat-table/ (inspired by the table for ES6).

Todos

One remaining task is to specify a test format for the pub/sub methods. One possibility is a transitionEvent field to specify an event (e.g. arrival of a block or a pending transaction), and an onTransition field to describe the expected notifications.

Test suite maintenance

An automated maintenance process for the RPC test suite will be described in a forthcoming Meta-EIP. The current plan is to maintain updates to the RPC test suite through a voting contract, with one-developer one-vote where developer identities are verified using devcon2 tokens. (There is a separate plan to expand the Devcon2 tokens into a web of trust, in order to enable participation by any and all Ethereum developers/stakeholders). Tentatively named EipSignal, this automated governance process is somewhat in the spirit of the Yellow Paper Committee; however, the outcome of EipSignal votes will only mandate changes to the RPC API specification and test suite as described above, and will NOT govern "core/consensus" protocol changes. The work-in-progress on EipSignal can seen here: MetaMask/IPFS-Ethereum-Hackathon#14.

[axic]

👍

The asserts array specifies jq programs or "filters", which are used to compare the received response against the expected response.

Is it possible to include strict verifications, such as a subset of regular expressions on some data fields? Thinking here about validating that an actual address was returned for example by personal_newAccount.

The chainConfig option points to a set of blocks/transactions to import. The current chain configuration is inherited from ethereum/rpc-tests, which in turn borrows from the state tests repo.

Have you seen cpp-ethereum's test_setChainParams RPC method? I think that might be the successor of that format.

[danfinlay]

This is very exciting to me. Developers need a place they can look up stable methods, and client devs need a smooth way to evolve their interfaces.

Other clients that will be nice to add to the tested columns:

• testrpc
• provider-engine (Wrapped with an RPC server, backed by different clients)

And I'm sure others, too. Once we have a strong singular test suite like this, building new RPC providers should become a much more stable process.

[bobsummerwill]

Really great to see this happening, @cdetrio, @FlySwatter!

Starting with the RPCs makes sense, and I hope that the patterns which are built can be extended to broader specifications over time. We've had discussions along very similar lines for Enterprise using Boardroom.

See also http://html5test.com.

[cdetrio]

Is it possible to include strict verifications, such as a subset of regular expressions on some data fields?

Yes, actually there's two ways to enforce a regexp. It can be done in the JSON schema using pattern, or as a test case using a jq filter.

[cdetrio]

Have you seen cpp-ethereum's test_setChainParams RPC method? I think that might be the successor of that format.

Ah, here's an issue on that: ethereum/interfaces#4

[cdetrio]

There's also https://github.com/ethcore/ethereum-rpc-json, which generates documentation but doesn't include tests to run against clients.

[rphmeier]

http://cdetrio.github.io/eth-compat-table/

Seems like it measures compatibility against Geth, not against any spec.

[bobsummerwill]

Hey @rphmeier.

See "Test suite maintenance" at the very top of this issue.

The existing tests are incomplete placeholders. The plan is to expand/refine those to become the missing client-neutral executable specification.

An automated maintenance process for the RPC test suite will be described in a forthcoming Meta-EIP. The current plan is to maintain updates to the RPC test suite through a voting contract, with one-developer one-vote where developer identities are verified using devcon2 tokens. (There is a separate plan to expand the Devcon2 tokens into a web of trust, in order to enable participation by any and all Ethereum developers/stakeholders). Tentatively named EipSignal, this automated governance process is somewhat in the spirit of the Yellow Paper Committee; however, the outcome of EipSignal votes will only mandate changes to the RPC API specification and test suite as described above, and will NOT govern "core/consensus" protocol changes. The work-in-progress on EipSignal can seen here: MetaMask/IPFS-Ethereum-Hackathon#14.

[danfinlay]

@bobsummerwill beat me to it, but I'll still reword:

@rphmeier

Seems like it measures compatibility against Geth, not against any spec.

There is no current spec to measure against, which is why this EIP exists.

The current plan is to manage this test suite with a form of governance, at which point non-geth-compliant specs may emerge, or even become dominant.

Until then, the linked test suite should be treated as a template for specs, not as a proposed spec itself.

[rphmeier]

A test suite isn't a specification, it just tests conformity to some specification. The schema described here is not sufficient to specify certain primitives which all methods rely upon. This is exactly how you get into the situation of strange corner cases that each client implements differently while still being "to-spec".

The places where I see the current documentation lacking the most:

• definition of pending state
• definition of error messages
• lack of consensus between client devs over changes

I'm glad that this EIP is working on point 3. But we need strong definitions of the primitives used to build the RPC methods rather than specific test cases which will just lead to overfitting.

It's misleading to label one specific implementation of ambiguous methods as "correct".

[danfinlay]

@rphmeier

I think we're actually starting to stray beyond the scope of this issue, but there isn't a better place for it. This issue is really about an RPC test format. We're beginning to talk about the governance process of managing those tests.

I understand the initial alarm, it could sound like someone is trying to hijack what "correct" means, but that's not our intention at all. We're not trying to shame Parity over a specific difference from Geth, we're trying to create a tool for all client devs to better coordinate their implementations.

The (current, in flux) plan is that a given suite of tests would accompany an EIP spec, not replace it. For example, for a given EIP, a variety of accompanying test suites could be nominated, and the most popular one(s?) could be displayed on the compatibility table.

Since the most popular test suite for a given EIP would be displayed, the test would not itself be an endorsement of that EIP spec, it would merely be a representation of it, and a way of automatically measuring compliance to it, which better represents the sort of ad-hoc standards process we actually have, where endorsement is almost solely represented through client implementation.

It certainly would not be a spec itself, since the table could include distinct, incompatible implementations of the same method, but it would provide a singular place to view and compare those incompatibilities. This table would not be about labeling any one implementation as correct, but instead, be a way of highlighting differences of implementation, for the benefit of both dapp and client developers.

I don't think a test suite has to encourage over-fitting, either. In the test format that @cdetrio has proposed, specific values of the response can be used to validate results, so it does not need to blindly enforce bit-conformity.

In the case of personal_sign, a reasonable test might check if a certain account with a published private key, given a certain message, produces a consistent signed output, and no more. This is not a spec, but adhering to it, and publishing this sample data, could have alleviated many headaches that came from having an under-specified feature adopted by a client.

I'm very interested in seeing solutions to 1 & 2 as well, and while a test suite may not be a complete solution for them, I think it may still be a useful tool for even those issues, when combined with EIP specifications.