Vision for cleaner substrate-node's developer-facing interface

[kianenigma]

related to #186 #1337

Vision 1

As it stands now, we take it for granted that that one builds a runtime, then pulls the substrate-node-template (or some similar repository of code), integrate it in a "blind" fashion (using probably nasty trial and error, without really knowing what they are doing) and never look at the node side code ever again.

Taking the same mindset that I talked about for a "substrate-node-builder-cli" here, Imagine an alternative like this:

• You build your pallet and runtime using feat: FRAME umbrella crate. #1337
• Then, you build a new project like:

[dependencies]
my-runtime = "1.0.0"
substrate-node-builder = "1.0.0"

• a main.rs that looks like:

use substrate_node_builder::{Builder, Consensus, Subcommand};
 
let node = Builder::default()
    // wire your node and runtime. All of the random types that the node needs from the runtime should not 
    // be raw re-exports anymore, and should be done via a `trait`.
    // this should also be helpful 
    .runtime(my_runtime::MyRuntime)
    // define your consensus engine. Can only be one of a few options. 
    .consensus(Consensus::ManualSeal)
    // Tweak the list of sub-commands from `Default`
    .add_subcommand(Subcommand::TryRuntime)
    .add_subcommand(Subcommand::Revert)
    .remove_subcommand(Subcommand::Benchmarks)
    // Tweak the list of RPCs endpoints
    .add_rpc(Rpc::Foo)
    .remove_rpc(Rpc::Bar)
    .build()
    .unwrap();

fn main() {
    node.run();
}

This is basically what a "creat-substrate-app" CLI would do for you, but done via code. Once we have this, the CLI would be rather trivial to build, and would merely be something that converts a YAML/JSON file t the above piece of code.

Vision 2

I have little clue about what are the blockers to reach the above. Historically I knew that code in service.rs has been quite a difficult one to play with. But, looking at a very simple node-template with manual seal, I think I can wrap my head around it and conclude that the above is feasible.

But, if the above is not possible, what I would be a good improvement to the existing quo, especially from a DevEx perspective is to replace the need to clone any code for the node software with pure binaries.

That is, once #62 is done, and there is less (hopefully none!) strict dependency between the node and the runtime, the process to learn FRAME and run a basic chain would be:

1. build your runtime, again with feat: FRAME umbrella crate. #1337
2. go to a release/download page where you are presented a list of node software different pre-configurations, such as consensus options.
3. You download the software, for example node and run ./node --runtime path/to/wasm.wasm <rest of substrate cli opts> and the rest should work.

I am not sure if this is actually much simpler than the above, as it would require some checks like seeing which runtime api/rpcs are available to become runtime (os opposed to the current compile-time) checks.

[kianenigma]

(Once I get some initial feedback, unless if this is totally crazy, I would like to post this in the parity forum for other eco-dev related departments to also see.)

[xlc]

I don't see much reason to have custom subcommand.
We don't want custom RPC.
The consensus is the same for all the parachains.
Runtime can be fetched from chain spec.
How about just create a universal client that works with all the parachains?

[arturgontijo]

We don't want custom RPC.

Why not? I mean, why impose that limit to the builders?
I see that the best practice would be calling RuntimeApi but I see no reason to limit it in a development tool like this.

[xlc]

It is all about trade offs. If you believe support custom RPC have more advantages than disadvantages, sure, list them out and we good.

Please keep in mind that Substrate is so flexible to the point it is pain to use and we still haven’t learned lessons from this?

[bkchr]

Yeah, custom RPCs are not really needed. This simple kind of interface should be to onboard people really easy. There would be some advantaged mode where you can still add your custom RPCs and whatever. So, there will be some "hard mode" and some "easy mode".

I already spoke with @kianenigma and I support this for parachains: paritytech/cumulus#2181

How about just create a universal client that works with all the parachains?

Yes, but to make this properly it will require quite a lot of work.

[arturgontijo]

So, there will be some "hard mode" and some "easy mode".

Ok, yeah...makes sense.

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/psa-parity-is-currently-working-on-merging-the-polkadot-stack-repositories-into-one-single-repository/2883/14

[gilescope]

@expenses and @gnunicorn were also keen on this happening.

[rphmeier]

Strong disagree on any Consensus subcommand.

There's a lot of stuff baked into consensus that doesn't usually get considered:

• Custom networking protocols
• Fork choice rules
• Many long-running tasks
• Running with multiple consensus engines

Consensus is usually not just one task but many things stitched together.

In my view, we should separate the infrastructure that Substrate (interfaces to libp2p, a block database, maybe stuff like sync or tx-gossip) from everything else. These should be provided as context into long-running futures that all just have absolute freedom in how they use these things.

Also, I don't see why the runtime should factor into this at all. The runtime should be encapsulated entirely by the chain-spec once the native runtime is gone, right? chain-spec is a CLI flag and can be handled internally.

Another goal is that we should not only abstract complexity at the very outer layer, but also make extending these functionalities very simple. The simplest thing is just letting users spawn futures.

For example - starting with an Aura parachain, but then adding a bunch of off-chain logic and networking protocols for e.g. offchain storage or oracles. This should be an easy tweak to the templates we provide.

My ideal here would be something like

// Some "mega object" that gives you access to:
//  - Registering new network protocols
//  - Adding block import hooks and notifications
//  - Spawning tasks
//  - Importing blocks
//  - Finalizing blocks
//  - Generating storage proofs
//  - Calling runtime APIs
//  - Setting the best block
//  - Adding RPCs
//  - Reading blocks and headers
//  - Reading and writing arbitrary items in the DB
//  - Reading state
//  - Executing Wasm
//  - etc.
//   
// It's not intended to be `Clone`d, but will give shared ownership of handles for all these things to avoid leaking a 
// god-object everywhere.
let raw_node: Node = RawNode::<Block>::new(cli_params);

The Raw Node does nothing on its own. It's just a bag of all the raw components you need to do anything interesting. This makes starting consensus logic simple.

Here are some examples (which could be library functions exposed by top-level Substrate/Cumulus crates).

A single function for running a standalone Babe/Grandpa node:

// Example for a standalone Babe/Grandpa node
fn vanilla_substrate() {
  let raw_node = RawNode::new(cli_params);

  // Under the hood, this is setting up new networking protocols, block import hooks, new DB
  spawn(run_grandpa(&raw_node));
  spawn(run_babe(&raw_node));

  // wait...
}

How it might be used in Polkadot or other complex systems with extra subsystems added:

fn polkadot() {
  let raw_node = RawNode::new(cli_params);

  // Under the hood, this is setting up new networking protocols, block import hooks, new DB
  spawn(run_grandpa(&raw_node));
  spawn(run_babe(&raw_node));
  spawn(run_parachain_consensus(&raw_node));

  // wait...
}

How it might be used for a one-size-fits-all parachain with Aura:

fn aura_parachain(cli_params) {
  // This may be starting a Polkadot raw node underneath. Or not. 
  let relay_chain_interface = make_relay_chain_interface(cli_params);

  let parachain_raw_node = RawNode::new(cli_params);

  // Under the hood, this is setting up new networking protocols, block import hooks, new DB
  spawn(run_parachain_general(&raw_node, relay_chain_interface));
  spawn(run_parachain_aura(&raw_node, relay_chain_interface));

  // wait...
}

This is pretty dead-simple and would hide all the complexity of registering network protocols, starting services, etc. inside of specialized crates that aren't exposed to the user except through a simple one or two parameter function.

I could see things getting a little more complex, i.e. if we let people swap in Proposers for consensus algorithms, but it still all stays very simple. Substrate could expose a DefaultProposer as it currently does in basic_authorship, for instance. Or even extracting out "standard" networking protocols like sync and transaction pooling, but having a utility function that spawns these things under the hood.

We need to make the easy stuff easy while keeping the hard stuff possible.

From above:

The stuff that the node provides ... maybe stuff like sync or tx-gossip

It's not clear to me that sync or tx-gossip shouldn't just be long-running futures themselves instead of baked into the service crate. These are also just components that build on top of the raw node.

[rphmeier]

@bkchr I am interested in sketching out some issues related to the above refactoring approach, if it's an agreeable direction. I am coming off of a larger project and would like to spend some time making Substrate more fun to work with for end-users, and this is one of the higher impact things I feel I can reason well about. Furthermore, many of these issues are the result of my own decisions in 2018/19 and I have some personal responsibility to improve the situation as a result.

Some initial suggestions with the goal of creating the struct RawNode:

1. Remove reliance on static dispatch in BlockImport. This is currently one of the main sources of setup hell, as we first set up BABE, GRANDPA, BEEFY, etc. BlockImport implementations and then spawn the tasks. I'd advocate for a simplification of the BlockImport trait and an implementation that wraps a Vec<dyn BlockImport> instead of setting them up like a composable onion.
2. Likewise, we amend the ImportQueue interface to accept an arbitrary amount of custom Verifiers via dynamic dispatch.
3. Remove the Proposer/Environment split. There really should just be one Proposer trait: fn propose(&self, parent_hash) -> Block
4. Create a general-purpose AllRuntimeApis: ProvideRuntimeApi type as described in Remove sp-api requirement on compile time bounds #27 which we can use to statically type the client. This is so the RawNode<Block> can have an internal client: Client<Block, SpecificBackend, SpecificExecutor, AllRuntimeApis>.
5. Decouple any enshrined network protocols from the starting of the network and add utility functions for spawning them.