Separation of app logic and framework code

[eddyashton]

Each CCF node currently executes a single binary containing both framework systems and application code. This means an update to either component requires a rebuild of this binary, the application code has a limited API and dependency list, and the application code must be written in C++ (or a scripting language via an embeddable interpreter).

We want to remove this coupling in future to support independent versioning and updates, broad support for alternative languages and dependencies, and separate scaling of application capacity.

The plan is to do this by separating each CCF node into multiple binaries, each deployed and managed as a separate container: a CCF-Core container written and maintained by the framework, communicating over a well-defined interface with Business Logic containers written in any other language. We will use gRPC for this inter-container communication interface, so that the Business-Logic containers can easily be written in any language with gRPC support.

Business-Logic containers will need to attest to their own identity, and CCF governance will determine the set of acceptable identities - those containers will then have full access to the confidential ledger contents, in the same way that application code does today.

A Business-Logic container will register to handle some set of (HTTP-method, HTTP-url) pairs (as endpoint handlers currently do), will receive requests parsed by a CCF-Core node over a gRPC interface, will execute their business logic using a gRPC KV API to read and write ledger entries, and produce some HTTP response to be returned when the transaction is to be applied. Beyond that, the containers may do any additional indexing or book-keeping they wish inside or outside that container (while considering confidentiality of everything they process). They may execute long-lived async requests to resolve user queries, or utilise external hardware for expensive computations.

This discussion is intended to be a central collection of the various decisions, designs, and tasks to achieve this.

[eddyashton]

Tasks

• Creating the definition of a gRPC service for CCF-Core to Business-Logic communication

  • [BL->CCF] Registration of new Business-Logic instances

  • [CCF->BL] Delivery of parsed HTTP requests to be executed

  • [BL->CCF->BL] KV interaction

    • We may start with a barebones get() + put()
    • Deals with raw byte buffers, deserialization is left to the BL
    • Need parity with current API eventually, including foreach(), has(), delete()

  • [BL->CCF] Return transaction result

  • Mermaid diagram demonstrating showing sequence diagram of these steps

• Running any gRPC service inside CCF

  • HTTP/2 support

  • An async runtime inside CCF

    • We can't use the entire default gRPC stack in-enclave, we need to overwrite parts of it ourselves.
    • We need to handle long-lived requests to remote services.
    • Simplest way to reason about these is in a robust async runtime.
    • Not a goal to expose this to existing applications directly, or to rewrite framework details with it, but some of this may be unavoidable.

  • A custom gRPC transport implementation, communicating over the ringbuffer

• Running this gRPC service inside a CCF node
  • CCF governance for defining acceptable BL code IDs
  • Dispatching incoming requests to the set of available BL workers, with retries and timeouts
  • Long-lived Tx objects
• Sample Business-Logic containers
  • Fetching and submitting an attestation
  • Multiple languages
  • Samples of automatically scaling deployments?
• Historical queries and indexing
  • The goal is for indexing to be done entirely in dedicated BL containers
  • gRPC API for BL to request an old ledger entry? KV reads at a target index?
  • gRPC API to register a callback for every ledger entry? Per-transaction, per-map, per-key?

[eddyashton]

Adding a few more recent thoughts, I'll try to combine these when I've settled on a de-risked approach:

• We don't need a new async runtime for an MVP. For a minimal impact implementation, we can reuse the existing forwarding code paths to hand off execution to a BL instance (ie - step out of processing a request, store some context, look-up and respond later). There's a little complexity in implementing commit of a transaction via this, but I think its possible.
  • Advantages of a richer async runtime would be better performance (work stealing), safety help (tracked ownership), and from those we could parallelize work in more places.
  • Those aren't required for this task, and we get the core ability to run in multiple threads from the existing ThreadMessaging scheduler.
• Registration of BL instances needs to be transactional, including removal when BL code IDs are revoked. But those are tightly bound to a type (channel state) that we don't want to store directly in the KV. Still working on the terminology for this, but I think we want an in-memory instantiation/manifestation of the KV state, which is a generic pattern we can use for other things. Essentially "read from this table, and if it's the same version as your current instantiation, reuse it, else rebuild it fresh". So it always has a correct read dependency on the state, but the deserialized value (or expensive thing derived from the deserialized value) can be reused.
• We may avoid including gRPC in the enclave, and instead manually parse/dispatch the messages ourselves from the raw protobuf. I'd like to use the generated types for constructing/sending messages, but they also have a large dependency footprint. The barrier here is how hard decoding gRPC actually is (in particular, I think they are usually zipped), and testing compatibility with multiple clients (its likely each language's client implementation has its own quirks, and we'll miss these in our minimal parser).

[eddyashton]

I've been experimenting with gRPC APIs in this repo, initially in Python. The code there's extremely sketchy, but should be a testing ground for a PoC protobuf service definition.

[eddyashton]

I've added some Mermaid diagrams to this repo sketching the currently planned flow. Including here with some comments on implications and open decisions.

Registration of a new Business-Logic instance

The initialisation block is something that each Business-Logic container must be able to do, so should be achievable with standard libraries in supported languages (ie - not require CCF code in each language). It's possible we do that through a sidecar.

I show a single BL instance here for simplicity, but really we'd expect it to shutdown on a rejected registration rather than retrying - imagine it's a new instance for the successful attempt.

Each BL generates a new secure identity, in-enclave, and uses that to establish a secure channel with CCF. CCF needs to listen for registration traffic. The BL instance should be given the CCF service identity, so it does not connect to arbitrary services (if it contains sensitive/confidential code).

A big design choice I haven't solved yet - is the set of supported endpoints part of the attestation (ie - determined by governance), or dynamically set by each BL when it registers? Either approach needs a new dispatch mechanism. The latter is more generic/powerful (allows js_generic-like general interpreters to be registered), but also makes code audit harder (actual endpoint implementation is out-of-attestation?). Also need to think about dispatch collisions - governance could disallow multiple code IDs claiming the same endpoint, whereas if the endpoints are provided by each BL instance we can't know if they're actually doing the same thing! Suggests this needs to be dictated by governance. May be solved by some generic policy language allowing either approach, per-service, dictated by constitution.

Execution of a user request

Activations are trying to show the presumed-to-be "blocking" bits, but this is missing some subtleties. In particular, when the request first arrives, there's technically an async wait before the CCF node actually begins processing it, and it's likely that this processing is itself several async steps.

Eventually it finds a BL instance to dispatch to. This dispatch lookup needs to be transactional, so that if the BL's code ID is revoked mid-execution, this transaction automatically fails.

This tries to show requests as solid lines, and responses in dotted lines. The CCF->BL execute..done block is a bit odd here, because CCF is actually the server for this gRPC service. This may be a server-streaming RPC, or we may expect the BL to submit an initial "Ready" request, or maybe that can be implicit in TLS connection/registration? TBC.