Constitution documentation (#2485)

CMakeLists.txt

@@ -388,14 +388,6 @@ if(BUILD_TESTS)
       ${CMAKE_CURRENT_SOURCE_DIR}/src/node/rpc/test/tx_status_test.cpp
     )
 
-    # TODO: Port or remove entirely? add_unit_test( member_voting_test
-    # ${CMAKE_CURRENT_SOURCE_DIR}/src/js/wrap.cpp
-    # ${CMAKE_CURRENT_SOURCE_DIR}/src/node/rpc/test/member_voting_test.cpp )
-    # target_link_libraries( member_voting_test PRIVATE
-    # ${CMAKE_THREAD_LIBS_INIT} http_parser.host sss.host ccf_endpoints.host
-    # quickjs.host ) set_tests_properties( member_voting_test PROPERTIES
-    # ENVIRONMENT RUNTIME_CONFIG_DIR=${CMAKE_SOURCE_DIR}/src/runtime_config )
-
     add_unit_test(
       proposal_id_test ${CMAKE_CURRENT_SOURCE_DIR}/src/js/wrap.cpp
       ${CMAKE_CURRENT_SOURCE_DIR}/src/node/rpc/test/proposal_id_test.cpp

doc/glossary.rst

@@ -10,10 +10,10 @@ Glossary
     Intel SGX Data Centre Attestation Primitives which allows SGX attestation to be used within Microsoft Azure.
 
   Consensus Protocols
-    The term Consensus protocol refers to either :ref:`Raft <overview/consensus:CFT Consensus Protocol>` or :ref:`BFT <overview/consensus:BFT Consensus Protocol>`. Generic Consensus terminology will use `primary` node and `backup` node to indicate node responsibility in carrying out the protocol(s). These correspond in Raft to `leader` and `follower`. More information about consensus protocols can `be found here <https://en.wikipedia.org/wiki/Consensus_(computer_science)>`_.
+    The term Consensus protocol refers to either :ref:`CFT <overview/consensus:CFT Consensus Protocol>` or :ref:`BFT <overview/consensus:BFT Consensus Protocol>`. Generic Consensus terminology will use `primary` node and `backup` node to indicate node responsibility in carrying out the protocol(s). These correspond in Raft to `leader` and `follower`. More information about consensus protocols can `be found here <https://en.wikipedia.org/wiki/Consensus_(computer_science)>`_.
 
   Constitution
-    Set of rules expressed in a JavaScript module that define how members' proposals are validated, resolved and applied.
+    JavaScript module that defines possible governance actions, and how members' proposals are validated, resolved and applied to the service.
 
   FLC
     `Flexible Launch Control <https://github.com/intel/linux-sgx/blob/master/psw/ae/ref_le/ref_le.md#flexible-launch-control>`_ is a feature of the Intel :term:`SGX` architecture.

doc/governance/accept_recovery.rst

@@ -102,7 +102,7 @@ Summary Diagram
         Member 0->>+Node 2: Propose accept_recovery
         Node 2-->>Member 0: Proposal ID
         Member 1->>+Node 2: Vote for Proposal ID
-        Node 2-->>Member 1: State: ACCEPTED
+        Node 2-->>Member 1: State: Accepted
         Note over Node 2, Node 3: accept_recovery proposal completes. Service is ready to accept recovery shares.
 
         Member 0->>+Node 2: GET recovery_share

doc/governance/common_member_operations.rst

@@ -6,7 +6,7 @@ Trusting a New Node
 
 As opposed to an opening network in which nodes are trusted automatically, new nodes added to an open network must become trusted through a governance proposal and vote before becoming part of the network.
 
-When an operator starts a new node with the ``join`` option (see :ref:`operations/start_network:Adding a New Node to the Network`), the node is recorded in state ``PENDING``. Then, members can vote to accept the new node, using the unique node id (hex-encoded string of the SHA-256 digest of the node's identity public key). See :ref:`governance/proposals:Proposing and Voting for a Proposal` for more detail.
+When an operator starts a new node with the ``join`` option (see :ref:`operations/start_network:Adding a New Node to the Network`), the node is recorded in state ``Pending``. Then, members can vote to accept the new node, using the unique node id (hex-encoded string of the SHA-256 digest of the node's identity public key). See :ref:`governance/proposals:Proposing and Voting for a Proposal` for more detail.
 
 Once the proposal successfully completes, the new node automatically becomes part of the network.
 
@@ -17,7 +17,7 @@ Updating Code Version
 
 For new nodes to be able to join the network, the version of the code they run (as specified by the ``--enclave-file``) should be first trusted by the consortium of members.
 
-If the version of the code being executed needs to be updated (for example, to support additional endpoints), members can create a ``new_node_code`` proposal, specifying the new code version.
+If the version of the code being executed needs to be updated (for example, to support additional endpoints), members can create an ``add_node_code`` proposal, specifying the new code version.
 
 .. note:: For a given :term:`Open Enclave` enclave library, the version of the code (``mrenclave``) can be found by running the ``oesign`` utility:
 

doc/governance/index.rst

@@ -1,7 +1,7 @@
 Governance
 ==========
 
-.. warning::
+.. note::
     See :doc:`/governance/js_gov` for pointers on converting from Lua to JS.
 
 This section describes how a consortium of trusted :term:`Members` governs an existing CCF network. It explains how members can submit proposals to CCF and how these proposals are accepted based on the rules defined in the :term:`Constitution`.

doc/governance/open_network.rst

@@ -7,9 +7,9 @@ This sections assumes that a set of nodes has already been started by :term:`Ope
 Adding Users
 ------------
 
-Once a CCF network is successfully started and an acceptable number of nodes have joined, members should vote to open the network to :term:`Users`. First, :doc:`the identities of trusted users should be generated </use_apps/index>`.
+Once a CCF network is successfully started and an acceptable number of nodes have joined, members should vote to open the network to :term:`Users`. First, the identities of trusted users should be generated, see :ref:`governance/adding_member:Generating Member Keys and Certificates`.
 
-Then, the certificates of trusted users should be registered in CCF via the member governance interface. For example, the first member may decide to make a proposal to add a new user (here, ``user_cert`` is the PEM certificate of the user -- see :ref:`overview/cryptography:Cryptography` for a list of supported algorithms):
+Then, the certificates of trusted users should be registered in CCF via the member governance interface. For example, the first member may decide to make a proposal to add a new user (here, ``cert`` is the PEM certificate of the user -- see :ref:`overview/cryptography:Cryptography` for a list of supported algorithms):
 
 .. code-block:: bash
 
@@ -63,7 +63,7 @@ Other members are then allowed to vote for the proposal, using the proposal id r
         "state": "Accepted"
     }
 
-The user is successfully added once a the proposal has received enough votes under the rules of the :term:`Constitution` (indicated by the response body showing a transition to state ``ACCEPTED``).
+The user is successfully added once the proposal has received enough votes under the rules of the :term:`Constitution` (indicated by the response body showing a transition to state ``Accepted``).
 
 The user can then make user RPCs.
 
@@ -99,6 +99,7 @@ Members configure this permission with ``set_user_data`` proposals:
         ]
     }
 
+
 Once this proposal is accepted, the newly added user (with ID ``529d0f48287923e7536a708c0b7747666f6b904d3fd4b84739f7d2204233a16e``) is able to use this endpoint:
 
 .. code-block:: bash
@@ -144,4 +145,4 @@ Once users are added to the opening network, members should create a proposal to
 
 Other members are then able to vote for the proposal using the returned proposal id.
 
-Once the proposal has received enough votes under the rules of the :term:`Constitution`, the network is opened to users. It is only then that users are able to execute transactions on the business logic defined by the enclave file (``--enclave-file`` option to ``cchost``).
+Once the proposal has received enough votes under the rules of the :term:`Constitution` (ie. ballots which evaluate to ``true``), the network is opened to users. It is only then that users are able to execute transactions on the business logic defined by the enclave file (``--enclave-file`` option to ``cchost``).

doc/governance/proposals.rst

@@ -1,35 +1,66 @@
 Proposing and Voting for a Proposal
 ===================================
 
-.. warning::
+.. note::
     See :doc:`/governance/js_gov` for pointers on converting from Lua to JS.
 
-This page explains how members can submit and vote for proposals.
+Summary
+-------
 
 Proposals are submitted as JSON documents, which if resolved successfully are applied atomically to the KV state.
 
-Ballots are submitted as JavaScript modules, executed transactionally, and are able to read from the current KV state but not write to it.
-Each vote script is given the proposal as a list of actions, and returns a Boolean indicating whether it supports or rejects it.
+Ballots are submitted as JavaScript modules exporting a single ``vote()`` function, executed transactionally, and are able to read from the current KV state but not write to it.
+Each vote script is given the proposal as a JSON document, typically containing list of actions, and returns a Boolean value indicating whether it supports or rejects it.
 
 Any member can submit a new proposal. All members can then vote, once at most, on this proposal using its unique proposal id.
-The proposer has the ability to `withdraw` a proposal while it is open.
+The proposer has the ability to `withdraw` a proposal as long as it is open.
 
-Each time a vote is submitted, all vote ballots for this proposal are re-executed on the current state to determine whether they are `for` or `against` the proposal. This vote tally is passed to the :term:`Constitution`, which determines whether the proposal is accepted or remains open. Once a proposal is accepted under the rules of the :term:`Constitution`, it is executed and its effects are recorded in the ledger.
+Each time a vote is submitted, all vote ballots for this proposal are re-executed on the current state to determine whether they are `for` or `against` the proposal.
+This vote tally is passed to the ``resolve()`` call in the :term:`Constitution`, which determines whether the proposal is accepted or remains open.
+Once a proposal is accepted under the rules of the :term:`Constitution`, it is executed via ``apply()`` and its effects are applied to the state and recorded in the ledger.
 
 For transparency and auditability, all governance operations (including votes) are recorded in plaintext in the ledger and members are required to sign their requests.
 
+.. mermaid::
+
+    sequenceDiagram
+        participant Member 0
+        participant Member 1
+        participant MemberFrontend
+        participant Constitution
+
+        Note over MemberFrontend, Constitution: CCF
+        Member 0->>+MemberFrontend: Submit Proposal to /gov/proposals
+        MemberFrontend->>+Constitution: call validate(Proposal)
+        Constitution-->>-MemberFrontend: no exception
+        MemberFrontend->>+Constitution: call resolve(Proposal, {})
+        Constitution-->>-MemberFrontend: not enough votes, return Proposal is Open
+        MemberFrontend-->>-Member 0: Proposal is Open
+
+        Member 1->>+MemberFrontend: Submit Ballot containing vote() to /gov/proposals/ProposalID/ballots
+        MemberFrontend->>MemberFrontend: evaluate vote(Proposal, KV State) to boolean Vote
+        MemberFrontend->>+Constitution: call resolve(Proposal, {Member 1: Vote})
+        Constitution-->>-MemberFrontend: enough positive votes, return Proposal is Accepted
+        MemberFrontend->>+Constitution: call apply(Proposal) to perform side-effects
+        Constitution-->>-MemberFrontend: no exception
+        MemberFrontend-->>-Member 1: Proposal is Accepted, has successfully been applied
+
+
 Creating a Proposal
 -------------------
 
-For custom proposals with multiple actions and precise conditional requirements you will need to write the proposal script by hand. For simple proposals there is a helper script in the CCF Python package - `proposal_generator.py`. This can be used to create proposals for common operations like adding members and users, without writing any JSON. It also produces sample vote scripts, which validate that the executed proposed actions exactly match what is expected. These sample proposals and votes can be used as a syntax and API reference for producing more complex custom proposals.
+For custom proposals with multiple actions and precise conditional requirements you will need to write the proposal script by hand.
+For simple proposals there is a helper script in the CCF Python package - ``proposal_generator.py``.
+This can be used to create proposals for common operations like adding members and users, without writing any JSON.
+It also produces sample vote scripts, which validate that the executed proposed actions exactly match what is expected.
+These sample proposals and votes can be used as a syntax and API reference for producing more complex custom proposals.
 
 Assuming the CCF Python package has been installed in the current Python environment, the proposal generator can be invoked directly as ``ccf.proposal_generator``. With no further argument it will print help text, including the list of possible actions as subcommands:
 
 .. code-block:: bash
 
-    python -m ccf.proposal_generator
     usage: proposal_generator.py [-h] [-po PROPOSAL_OUTPUT_FILE] [-vo VOTE_OUTPUT_FILE] [-pp] [-i] [-v]
-                             {add_node_code,new_node_code,remove_ca_cert_bundle,remove_js_app,remove_jwt_issuer,remove_member,remove_node,remove_node_code,remove_user,retire_node,retire_node_code,set_ca_cert_bundle,set_constitution,set_js_app,set_jwt_issuer,set_jwt_public_signing_keys,set_member,set_member_data,set_recovery_threshold,set_user,set_user_data,transition_node_to_trusted,transition_service_to_open,trigger_ledger_rekey,trigger_recovery_shares_refresh,trust_node}
+                                {add_node_code,remove_ca_cert_bundle,remove_js_app,remove_jwt_issuer,remove_member,remove_node,remove_node_code,remove_user,set_ca_cert_bundle,set_constitution,set_js_app,set_jwt_issuer,set_jwt_public_signing_keys,set_member,set_member_data,set_recovery_threshold,set_user,set_user_data,transition_node_to_trusted,transition_service_to_open,trigger_ledger_rekey,trigger_recovery_shares_refresh}
 
 Additional detail is available from the ``--help`` option. You can also find the script in a checkout of CCF:
 
@@ -41,17 +72,12 @@ Some of these subcommands require additional arguments, such as the node ID or u
 
 .. code-block:: bash
 
-    $ python -m ccf.proposal_generator trust_node 5
+    $ python -m ccf.proposal_generator transition_node_to_trusted 6d566123a899afaea977c5fc0f7a2a9fef33f2946fbc4abefbc3e10ee597343f
     SUCCESS | Writing proposal to ./trust_node_proposal.json
     SUCCESS | Wrote vote to ./trust_node_vote_for.json
 
     $ cat trust_node_proposal.json
-    {"script": {"text": "tables, args = ...; return Calls:call(\"trust_node\", args)"}, "parameter": "5"}
-
-    $ cat trust_node_vote_for.json
-    {
-      "ballot": "export function vote (rawProposal, proposerId) {\n  let proposal = JSON.parse(rawProposal);\n  if (!('actions' in proposal)) { return false; };\n  let actions = proposal['actions'];\n  if (actions.length !== 1) { return false; };\n  let action = actions[0];\n  if (!('name' in action)) { return false; };\n  if (action.name !== 'transition_node_to_trusted') { return false; };\n  if (!('args' in action)) { return false; };\n  let args = action.args;\n  {\n    if (!('node_id' in args)) { return false; };\n    let expected = \"cc6e776911230e4c419475b528ae272c655b1133c513476783daea67c59d9ffa\";\n    if (JSON.stringify(args['node_id']) !== JSON.stringify(expected)) { return false; };\n  }\n  return true;\n}"
-    }
+    {"actions": [{"name": "transition_node_to_trusted", "args": {"node_id": "6d566123a899afaea977c5fc0f7a2a9fef33f2946fbc4abefbc3e10ee597343f"}}]}
 
     $ python -m ccf.proposal_generator --pretty-print --proposal-output-file add_pedro.json --vote-output-file vote_for_pedro.json set_user pedro_cert.pem
     SUCCESS | Writing proposal to ./add_pedro.json
@@ -69,11 +95,6 @@ Some of these subcommands require additional arguments, such as the node ID or u
       ]
     }
 
-    $ cat vote_for_pedro.json
-    {
-      "ballot": "export function vote (rawProposal, proposerId) {\n  let proposal = JSON.parse(rawProposal);\n  if (!('actions' in proposal)) { return false; };\n  let actions = proposal['actions'];\n  if (actions.length !== 1) { return false; };\n  let action = actions[0];\n  if (!('name' in action)) { return false; };\n  if (action.name !== 'set_user') { return false; };\n  if (!('args' in action)) { return false; };\n  let args = action.args;\n  {\n    if (!('cert' in args)) { return false; };\n    let expected = \"-----BEGIN CERTIFICATE-----\\nMIIBsjCCATigAwIBAgIUOiTU32JZsA0dSv64hW2mrKM0phEwCgYIKoZIzj0EAwMw\\nEDEOMAwGA1UEAwwFdXNlcjIwHhcNMjEwNDE0MTUyODMyWhcNMjIwNDE0MTUyODMy\\nWjAQMQ4wDAYDVQQDDAV1c2VyMjB2MBAGByqGSM49AgEGBSuBBAAiA2IABBFf+FD0\\nUGIyJubt8j+f8+/BP7IY6G144yF/vBNe7CJpNNRyiMZzEyN6wmEKIjsn3gU36A6E\\nqNYBlbYbXD1kzlw4q/Pe/Wl3o237p8Es6LD1e1MDUFp2qUcNA6vari6QLKNTMFEw\\nHQYDVR0OBBYEFDuGVragGSHoIrFA44kQRg/SKIcFMB8GA1UdIwQYMBaAFDuGVrag\\nGSHoIrFA44kQRg/SKIcFMA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwMDaAAw\\nZQIxAPx54LaqQevKrcZIr7QSCZKGFJgSxfVxovSfEqTMD+sKdWzNTqJtJ1SDav1v\\nImA4iwIwBsrdevSQj4U2ynXiTJKljviDnyc47ktJVkg/Ppq5cMcEZHO4Q0H/Wq3H\\nlUuVImyR\\n-----END CERTIFICATE-----\\n\";\n    if (JSON.stringify(args['cert']) !== JSON.stringify(expected)) { return false; };\n  }\n  return true;\n}"
-    }
-
 These proposals and votes should be sent as the body of HTTP requests as described below.
 
 Creating Proposals in Python
@@ -143,6 +164,7 @@ Here a new proposal has successfully been created, and nobody has yet voted for
       "ballot": "export function vote (proposal, proposerId) { return true }"
     }
 
+
     # Member 1 approves the proposal (votes in favour: 1/3)
     $ scurl.sh https://<ccf-node-address>/gov/proposals/d4ec2de82267f97d3d1b464020af0bd3241f1bedf769f0fee73cd00f08e9c7fd/ballots --cacert network_cert --key member1_privk --cert member1_cert --data-binary @vote_accept.json -H "content-type: application/json"
     {
@@ -152,6 +174,7 @@ Here a new proposal has successfully been created, and nobody has yet voted for
       "state": "Open"
     }
 
+
     # Member 2 rejects the proposal (votes in favour: 1/3)
     $ scurl.sh https://<ccf-node-address>/gov/proposals/d4ec2de82267f97d3d1b464020af0bd3241f1bedf769f0fee73cd00f08e9c7fd/ballots --cacert network_cert --key member2_privk --cert member2_cert --data-binary @vote_reject.json -H "content-type: application/json"
     {
@@ -206,7 +229,7 @@ At any stage during the voting process, before the proposal is accepted, the pro
 
 .. code-block:: bash
 
-    $ scurl.sh https://<ccf-node-address>/gov/proposals/<proposal-id>/withdraw --cacert networkcert.pem --key member1_privk.pem --cert member1_cert.pem -H "content-type: application/json"
+    $ scurl.sh https://<ccf-node-address>/gov/proposals/d4ec2de82267f97d3d1b464020af0bd3241f1bedf769f0fee73cd00f08e9c7fd/withdraw --cacert networkcert.pem --key member1_privk.pem --cert member1_cert.pem -H "content-type: application/json"
     {
       "ballot_count": 1,
       "proposal_id": "d4ec2de82267f97d3d1b464020af0bd3241f1bedf769f0fee73cd00f08e9c7fd",

doc/index.rst

@@ -71,6 +71,13 @@ CCF Documentation
 
     Contribute to the CCF GitHub repository.
 
+    ---
+
+    :fa:`book` :doc:`glossary`
+    ^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+    Common terms used throughout the documentation and their definitions.
+
 .. toctree::
     :maxdepth: 1
     :hidden: