ethereumProvider API

[frozeman]

Abstract

This draft is to discuss the ethereum API, an object exposed by ethereum native browsers and tools like Mist, MetaMask and status.im.

The API should be small and clean and allow all kind of libraries to attach to it.

The class of the object should be EthereumProvider, the object name ethereum.

API

The ethereum should inherit from a EventEmitter type class, and provide add/removeListener, on/once/off etc.

// callback: a error first callback
ethereum.send(methodName, params, callback)
> undefined

// CALLBACK return
> err, result

// returns subscription notification,
ethereum.on(subscriptionType, handler)

subscriptionType = `eth_subscription`

// HANDLER return
> {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }

// react on connection start
ethereum.on('connect', handler)

// HANDLER return
null

// react on connection end
ethereum.on('end', handler)

// HANDLER return
Error Object with message (timeout, error, drop, etc)

[frozeman]

@kumavis @jarradh

[kumavis]

I suggest that connect() and the relevant events are removed from the standard.
If a special subclass of ethereumProvider needs it, it can implement it.

Why did you include those?

Any thoughts on returning a Promise in ethereumProvider.send(payload, callback)?

Looks good!

[kumavis]

small nitpick ethereumProvider.on('timeout', callback) the callback reference here should be called handler, as callback implies that it should only be called once

[frozeman]

changed to handler.

Hm yeah, if the provider always takes care of connecting and re-connecting automatically we wouldn't need connect()

Concerning promise. i would like to keep this lib small and a promise is an extra dependency.
Apart from that i'd follow node.js defaults here, which is error first callback :)
But im open for debate..

[kumavis]

here's a version with the connection semantics in a separate subclass

EthereumProvider : EventEmitter

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(payload, callback)
> undefined

ethereumProvider.on('data', handler)
> undefined

// The subscription received in the handler:
{
  "jsonrpc":"2.0",
   "method":"eth_subscription",
   "params": {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }
}

EthereumWebsocketProvider : EthereumProvider

// Will try to connect, the ethereumProvider should be already connected when the webpage is loaded and firse the "connect" event at start.
ethereumProvider.connect()
> undefined

// react on connection start
ethereumProvider.on('connect', handler)

// react on connection end
ethereumProvider.on('end', handler)

[tcoulter]

My biggest ask is that dapp developers not have to code for specific provider types. For instance, a dapp developer should not have to maintain a specific code block for Metamask-style providers where no connection information needs to be managed, and another code block for Mist that, say hypothetically issues a provider where connection information does need to be managed. (If unclear, I interpret @kumavis's suggestion above as requiring dapp developers to do exactly that).

Creating a library that supports that ask would require either we:

1. Expose a connection abstraction that every provider must implement, for all provider types; or
2. Expose no connection information at all.

My vote is on a modified hybrid approach, as described below, as I think the current connection abstraction is too specific to a certain connection type, and exposes too many temporary error states. Instead, I think we should focus on whether a provider has been "started" and "stopped" -- i.e., is it ready to provide you the information you need, or not. Here's what I propose:

// Implemented by all providers; is a no-op for providers that don't need to 
// connect to anything or don't require any type of polling. If there's base
// code we decide on, it could implement this no-op by default.
//
// The callback is fired when the provider is successfully started.
// This is important for providers that work over an established connection
// or that are in-process and connect to a database backend.
ethereumProvider.start(callback)

Similarly, we could implement a stop method that can also be a no-op if not necessary.

// The callback is fired when the connection is successfully stopped.
// Important for established connections as well as database backends.
ethereumProvider.stop(callback)

Sending data is the same as the examples above, but the handler has changed (more on that below):

// payload: is a valid jsonrpc 2.0 object
// callback: see below
ethereumProvider.send(payload, callback)

Also, on('data') can be more generic. Instead of being limited to subscriptions, which assume a specific transport, on('data') can provide a handler for all data coming through the pipe. See the note on handlers below.

// returns payload and response, see handler details below
ethereumProvider.on('data', handler)

As well, theon('error') event has been modified:

// Will return an error as a result of any payload; this includes connection
// errors as well as rpc errors. See below for handler details.
ethereumProvider.on('error', handler)

All callbacks and handlers on send, on('error'), on('data') are of this form:

function(err, payload, response) {
  // connection error
  // payload (request)
  // response: response body returned by the RPC server
}

Note that err above would only include connection errors as, if RPC clients are following the specification, RPC errors are part of the response body.

An http provider pointed to an invalid non-responsive or incorrect url, for example, would expose connection errors in the err object when a request 400's or 500's. Websocket errors would return a connection err (err) when the connection hasn't reconnected after a specific period of time. The TestRPC provider would return an err when there's a runtime exception that should never have occurred (i.e., the TestRPC provider would run in the same process, so wouldn't need to make use of this error path as a connection error).

I see all of this discussion as akin to leveldown. An output of this discussion should not only be an abstraction that works for any connection type (or none at all) with the littlest effort on its users; but we should also produce an abstract-leveldown-like codebase (we'd call it abstract-ethereum-provider) that lets any data layer fit in and the connection type inconsequential.

[kumavis]

I interpret @kumavis's suggestion above as requiring dapp developers to do exactly that

@tcoulter you seemed to have misunderstood my proposal. I specified a standard EthereumProvider and an example subclass of that standard EthereumWebsocketProvider. The standard is what dapp developers would expect to be injected by the browser. The subclass would be for the case where the there was no standard api object injected and the dapp dev wanted to set up their own fallback that would necessarily require connect configuration be handled by the dapp dev.

[kumavis]

Also, on('data') can be more generic.

Instead of being limited to subscriptions, which assume a specific transport,

its not a specific transport, its a feature of the ethereum json rpc built on top of json rpc notifications.

on('data') can provide a handler for all data coming through the pipe. See the note on handlers below.

its nice having just a handler for server sent notifications, but interesting idea

[kumavis]

Creating a library that supports that ask would require either we:

for the sake of simplicity I advocate for 2. Expose no connection information at all. in the base standard

[tcoulter]

for the sake of simplicity I advocate for 2. Expose no connection information at all. in the base standard

I went down that path too when I originally started writing my original comment. If you as a dapp developer want anything extra it means you have to explicitly code for a specific environment (i.e., the websocket provider), which is why I later chose the hybrid/more general abstraction approach. However, if the custom provider classes are meant to only be used as a fallback, then I could get into that.

its not a specific transport, its a feature of the ethereum json rpc built on top of json rpc notifications.

👍 Good to know, thanks. I find the general on("data") to more useful (at least theoretically), but could go either or as the behavior I'm looking for could easily be shimmed.

[frozeman]

Ok sorry for the late response, but i was on a holiday. @tcoulter @kumavis

So concerning exposing no connection data:
IMO thats wrong. I do agree we can consolidate "timeout" and "error" to simply "end". But a library like web3.js can and must be able to know about connection drops, which can happen in mist, as well as any internet connected device. Web3.js and other libraries in this case must know about it and provide subscription regains strategies and lost connection behaviour.

Or the dapp developer itself must be able to act on connection drops as well.

Its not feasable to try to mitigate all of that under the hood in the EthereumProvider itself.

Concerning the generic on("data"):

This makes building libs like web3.js a lot more complicated, as each of those libraries needs to handle the mapping between request and responses and do the mapping of JSON rpc IDs themselves. This is easily taken away by ethereumProvider.send(payload, callback), and i think thats a good thing and makes it more fun to build web3.js like libraries.

Concerning the ethereumProvider.connect()

This is necessary to force regaining connection tries, but we can get rid of it, if we make sure to call the "connect" event on reconnect, and try to reconnect under the hood automatically.

Concerning Separate WS provider

The ethereumProvider above is on purpose generic for all type of providers, be it HTTP, IPC or WS under the hood. Its kind of its own type and can be used in web3 like libs, as they would a WS provider.

If ethereumProvider is not defined the dapp has to load its own WS or whatever provider to get a connection. The browser can't take care of this.

So given the above feedback i would suggest the following API:

// payload: is a valid jsonrpc 2.0 object
// callback: a error first callback
ethereumProvider.send(payload, callback)
> undefined

// returns subscription notification,
ethereumProvider.on('data', handler)

// The subscription received in the callback:
> {
  "jsonrpc":"2.0",
   "method":"eth_subscription",
   "params": {
       "subscription":"0x2acffa11d68280c2954daeb77cb849d9",
       "result": {...}
    }
}

// react on connection start
ethereumProvider.on('connect', handler)

// react on connection end
ethereumProvider.on('end', handler)

[danfinlay]

What happened to your PromEvent pattern? I was fond of that. I'm not as big a fan of a global data event.

I'd much rather that each individual query is subscribable, allowing reactive state updates.

Example:

token.getBalance(myAccount)
.on('update', updateUI)

Also, I think a single PR is too simplistic a form to discuss an entire spec. I would propose the spec proposal be a repository, and then individual features & changes could be pull requests, discussed individually.

[frozeman]

This here is about a basic provider provided by mist, metamask and others, not how web3.js or any other library on top will work ;)

Off Topic:
Tho as the subscriptions we currently have are rather limited in the RPC, things like you propose there, are nice but won't be available in web3.js 1.0.
The PromiEvent is only working on the send function right now: http://web3js.readthedocs.io/en/1.0/web3-eth.html#id62
And for contract methods:
http://web3js.readthedocs.io/en/1.0/web3-eth-contract.html#id21

[frozeman]

To get back to this and finalise it soon. @kumavis @tcoulter

Here is an idea from @MaiaVictor which is what you @kumavis probably mentioned in one of these calls, to have a simple function object which can handle the calls, without exposing formatting the RPC calls yourself.

I could imagine something like this, but with the subscription addition:
https://github.com/MaiaVictor/ethereum-rpc