Add Publish & Custom Genesis REST documentation. (#1176)

* Update REST api documentation to include Publish & Custom Genesis. Also update sui-move documentation for REST publishing

* Response to PR comments

* Fix small typo

* Update rest-api.md

Make my direct edits to rest-api.md

* Update wallet.md

Make small edits directly to wallet.md

* merge with main

* revert unrelated changes triggered by pre-commit

* revert unrelated change

* Modify sui-move to output as base64 instead of hex to be used in RESTendpoint

* Small field name change

Co-authored-by: Clay-Mysten <100217682+Clay-Mysten@users.noreply.github.com>

doc/src/build/rest-api.md

@@ -2,9 +2,9 @@
 title: Local REST Server & REST API Quick Start
 ---
 
-Welcome to the Sui REST API quick start. 
+Welcome to the Sui REST API quick start.
 
-This document walks you through setting up your own local Sui REST Server and using the Sui REST API to interact with a local Sui network. This guide is useful for developers interested in Sui network interactions via API. For a similar guide on Sui network interactions via CLI, refer to the [wallet](wallet.md) documentation.  
+This document walks you through setting up your own local Sui REST Server and using the Sui REST API to interact with a local Sui network. This guide is useful for developers interested in Sui network interactions via API. For a similar guide on Sui network interactions via CLI, refer to the [wallet](wallet.md) documentation.
 
 Full [API documentation](https://app.swaggerhub.com/apis/MystenLabs/sui-api) can
 be found on SwaggerHub.
@@ -51,6 +51,24 @@ Retrieve OpenAPI documentation:
 curl --location --request GET $SUI_GATEWAY_HOST/docs | json_pp
 ```
 
+### POST /api/sync_account_state
+
+Synchronize client state with authorities:
+
+```shell
+curl --location --request POST $SUI_GATEWAY_HOST/api/sync_account_state \
+--header 'Content-Type: application/json' \
+--data-raw '{
+    "address": "{{address}}"
+}'
+```
+
+You should replace `{{address}}` in the command above with an actual
+address value, for example one obtained from `wallet.conf` (without quotes).
+
+This will fetch the latest information on all objects owned by each
+address that is managed by this server. This command has no output.
+
 ### GET /api/objects
 
 Return the list of objects owned by an address:
@@ -59,8 +77,9 @@ curl --location --request GET $SUI_GATEWAY_HOST'/api/objects?address={{address}}
 ```
 
 You should replace `{{address}}` in the command above with an actual
-address value, you can retrieve the list of the addresses created during 
-genesis from `wallet.conf`.
+address value, you can retrieve the list of the addresses created during
+genesis from `wallet.conf`. Ensure you have run [`POST
+/api/sync_account_state`](#post-apisync_account_state)
 
 The output you see should resemble the following (abbreviated to show only two objects):
 
@@ -158,10 +177,9 @@ You should replace `{{owner_address}}` and `{{to_address}}` in the
 command above with an actual address values, for example one obtained
 from `wallet.conf`. You should also replace
 `{{coin_object_id}}` and `{{gas_object_id}}` in the command above with
-an actual object ID, for example one obtained from [`GET
-/objects`](#get-apiobjects) (from `objectId` in the output of [`GET
-/objects`](#get-apiobjects). You can see that all objects generated
-during genesis are of `Coin/GAS` type). For this call to work, objects
+an actual object ID, for example one obtained from `objectId` in the output
+of [`GET /objects`](#get-apiobjects). You can see that all gas objects generated
+during genesis are of `Coin/SUI` type). For this call to work, objects
 represented by both `{{coin_object_id}}` and `{{gas_object_id}}` must
 be owned by the address represented by `{{owner_address}}`.
 
@@ -210,31 +228,76 @@ objects). Consequently, you should fill out argument placeholders
 would for [`POST /api/transfer`](#post-apinew_transfer) - please not additional
 `0x` prepended to function arguments.
 
-NOTE: A Publish endpoint is in the works, but for now the only way to add a new module is to have it included as part of genesis. To do this, add your Move module to  `sui_programmability/framework/sources` before you hit the genesis endpoint. Once you have done this you will be able to use `"packageObjectId": "0x2"` in the call endpoint to find your Move module.
+NOTE: A Publish endpoint is in the works, but for now the only way to add a new module is to have it included as part of genesis. To do this, add your Move module to `sui_programmability/framework/sources` before you hit the genesis endpoint. Once you have done this you will be able to use `"packageObjectId": "0x2"` in the call endpoint to find your Move module.
 
 To learn more about what `args` are accepted in a Move call, refer to the [SuiJSON](sui-json.md) documentation.
 
-### POST /api/sync_account_state
+### POST /api/publish
 
-Synchronize client state with authorities:
+Publish Move module.
 
 ```shell
-curl --location --request POST $SUI_GATEWAY_HOST/api/sync_account_state \
+curl --location --request POST $SUI_GATEWAY_HOST/publish \
 --header 'Content-Type: application/json' \
 --data-raw '{
-    "address": "{{address}}"
+    "sender": "{{owner_address}}",
+    "compiledModules": {{vector_of_compiled_modules}},
+    "gasObjectId": "{{gas_object_id}}",
+    "gasBudget": 10000
 }' | json_pp
 ```
 
-You should replace `{{address}}` in the command above with an actual
-address value, for example one obtained from `wallet.conf` (without quotes).
+This endpoint will perform proper verification and linking to make
+sure the package is valid. If some modules have [initializers](move.md#module-initializers), these initializers
+will also be executed in Move (which means new Move objects can be created in
+the process of publishing a Move package). Gas budget is required because of the
+need to execute module initializers.
 
-This will fetch the latest information on all objects owned by each
-address that is managed by this server. This command has no output.
+You should replace `{{owner_address}}` in the
+command above with an actual address values, for example one obtained
+from `wallet.conf`. You should also replace `{{gas_object_id}}` in the command above with
+an actual object ID, for example one obtained from `objectId` in the output
+of [`GET /objects`](#get-apiobjects). You can see that all gas objects generated
+during genesis are of `Coin/SUI` type). For this call to work, the object
+represented by `{{gas_object_id}}` must be owned by the address represented by
+`{{owner_address}}`.
+
+To publish a Move module, you also need `{{vector_of_compiled_modules}}`. To generate the value of this field, use the `sui-move` command. The `sui-move` command supports printing the bytecodes as base64 with the following option
+
+```
+sui-move --path <move-module-path> build --dump-bytecode-as-base64
+```
+
+Assuming that the location of the package's sources is in the `PATH_TO_PACKAGE` environment variable an example command would resemble the following
+
+```
+sui-move --path $PATH_TO_PACKAGE/my_move_package build --dump-bytecode-as-base64
+
+["oRzrCwUAAAAJAQAIAggUAxw3BFMKBV1yB88BdAjDAigK6wIFDPACQgAAAQEBAgEDAAACAAEEDAEAAQEBDAEAAQMDAgAABQABAAAGAgEAAAcDBAAACAUBAAEFBwEBAAEKCQoBAgMLCwwAAgwNAQEIAQcODwEAAQgQAQEABAYFBgcICAYJBgMHCwEBCAALAgEIAAcIAwABBwgDAwcLAQEIAAMHCAMBCwIBCAADCwEBCAAFBwgDAQgAAgsCAQkABwsBAQkAAQsBAQgAAgkABwgDAQsBAQkAAQYIAwEFAgkABQMDBwsBAQkABwgDAQsCAQkAAgsBAQkABQdNQU5BR0VEBENvaW4IVHJhbnNmZXIJVHhDb250ZXh0C1RyZWFzdXJ5Q2FwBGJ1cm4EaW5pdARtaW50DHRyYW5zZmVyX2NhcAtkdW1teV9maWVsZA9jcmVhdGVfY3VycmVuY3kGc2VuZGVyCHRyYW5zZmVyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgACAQkBAAEAAAEECwELADgAAgEAAAAICwkSAAoAOAEMAQsBCwAuEQY4AgICAQAAAQULAQsACwI4AwIDAQAAAQQLAAsBOAQCAA==", "oRzrCwUAAAALAQAOAg4kAzJZBIsBHAWnAasBB9IC6QEIuwQoBuMECgrtBB0MigWzAQ29BgYAAAABAQIBAwEEAQUBBgAAAgAABwgAAgIMAQABBAQCAAEBAgAGBgIAAxAEAAISDAEAAQAIAAEAAAkCAwAACgQFAAALBgcAAAwEBQAADQQFAAIVCgUBAAIICwMBAAIWDQ4BAAIXERIBAgYYAhMAAhkCDgEABRoVAwEIAhsWAwEAAgsXDgEAAg0YBQEABgkHCQgMCA8JCQsMCw8MFAYPBgwNDA0PDgkPCQMHCAELAgEIAAcIBQILAgEIAwsCAQgEAQcIBQABBggBAQMEBwgBCwIBCAMLAgEIBAcIBQELAgEIAAMLAgEIBAMLAgEIAwEIAAEGCwIBCQACCwIBCQAHCwcBCQABCAMDBwsCAQkAAwcIBQELAgEJAAEIBAELBwEIAAIJAAcIBQELBwEJAAEIBgEIAQEJAAIHCwIBCQALAgEJAAMDBwsHAQkABwgFAQYLBwEJAAZCQVNLRVQHTUFOQUdFRARDb2luAklEA1NVSQhUcmFuc2ZlcglUeENvbnRleHQHUmVzZXJ2ZQRidXJuBGluaXQObWFuYWdlZF9zdXBwbHkEbWludApzdWlfc3VwcGx5DHRvdGFsX3N1cHBseQtkdW1teV9maWVsZAJpZAtWZXJzaW9uZWRJRAx0cmVhc3VyeV9jYXALVHJlYXN1cnlDYXADc3VpB21hbmFnZWQFdmFsdWUId2l0aGRyYXcPY3JlYXRlX2N1cnJlbmN5Bm5ld19pZAR6ZXJvDHNoYXJlX29iamVjdARqb2luAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgMIAAAAAAAAAAAAAgEOAQECBA8IBhELBwEIABMLAgEIAxQLAgEIBAABAAAIFg4BOAAMBAsBCgAPADgBCgAPAQoECgI4AgwFCwAPAgsECwI4AwwDCwULAwIBAAAAEA8JEgAKADgEDAEKABEKCwEKADgFCwA4BhIBOAcCAgEAAAMECwAQAjgIAgMBAAAFHA4BOAkMBAoEDgI4CCEDDgsAAQsDAQcAJwoADwELATgKCgAPAgsCOAsLBAsADwALAzgMAgQBAAADBAsAEAE4CQIFAQAAAwQLABAAOA0CAQEBAgEDAA=="]
+Build Successful
+```
 
+Copy the outputting base64 representation of the compiled Move module into the
+REST publish endpoint.
+
+#### 2, Sign the transaction
+Follow the instructions to [sign the transaction](rest-api.md#2-sign-the-transaction-using-the-sui-signtool).
+
+#### 3, Execute the transaction
+Follow the instructions to [execute the transaction](rest-api.md#3-execute-the-transaction-using-the-transaction-data-signature-and-public-key).
+
+Below you can see a truncated sample output of [POST /publish](#post-publish). One of the results of executing this command is generation of a package object representing the published Move code. An ID of the package object can be used as an argument for subsequent Move calls to functions defined in this package.
+
+```
+{
+    "package": [
+            "13e3ec7279060663e1bbc45aaf5859113fc164d2",
+    ...
+}
+```
 
 ## Postman setup
-The recommended way to test the Sui REST API is to use Postman. 
+The recommended way to test the Sui REST API is to use Postman.
 
 Postman is an API platform for building and using APIs. Postman provides an alternative solution to accessing APIs over issuing `curl` commands in a terminal. You can use variables rather than copy-pasting addresses and object IDs for each call in a terminal. We have provided a sample Postman runbook for you to use.
 

doc/src/build/wallet.md

@@ -240,15 +240,15 @@ $ wallet --config /path/to/wallet/config/file
 The Sui interactive wallet supports the following shell functionality:
 
 * *Command history* -
-  The `history` command can be used to print the interactive shell's command history; 
-  you can also use Up, Down or Ctrl-P, Ctrl-N to navigate previous or next matches from history. 
+  The `history` command can be used to print the interactive shell's command history;
+  you can also use Up, Down or Ctrl-P, Ctrl-N to navigate previous or next matches from history.
   History search is also supported using Ctrl-R.
 * *Tab completion* -
   Tab completion is supported for all commands using Tab and Ctrl-I keys.
 * *Environment variable substitution* -
-  The wallet shell will substitute inputs prefixed with `$` with environment variables, 
-  you can use the `env` command to print out the entire list of variables and 
-  use `echo` to preview the substitution without invoking any commands.  
+  The wallet shell will substitute inputs prefixed with `$` with environment variables,
+  you can use the `env` command to print out the entire list of variables and
+  use `echo` to preview the substitution without invoking any commands.
 
 ### Command line mode
 
@@ -349,26 +349,26 @@ To see how much gas is in an account, use the `gas` command. Note that this comm
 
 ```
 wallet --no-shell gas
-                Object ID                 |  Version   |  Gas Value 
+                Object ID                 |  Version   |  Gas Value
 ----------------------------------------------------------------------
- 0B8A4620426E526FA42995CF26EB610BFE6BF063 |     0      |   100000   
- 3C0763CCDEA4FF5A4557505A62AB5E1DAF91F4A2 |     0      |   100000   
- 45A589A9E760D7F75D399327AC0FCBA21495C22E |     0      |   100000   
- 4C377A3A9D4B1B9C92189DD12BB1DCD0302A954B |     0      |   100000   
- F2961464AC6860A05D21B48C020B7E121399965C |     0      |   100000   
+ 0B8A4620426E526FA42995CF26EB610BFE6BF063 |     0      |   100000
+ 3C0763CCDEA4FF5A4557505A62AB5E1DAF91F4A2 |     0      |   100000
+ 45A589A9E760D7F75D399327AC0FCBA21495C22E |     0      |   100000
+ 4C377A3A9D4B1B9C92189DD12BB1DCD0302A954B |     0      |   100000
+ F2961464AC6860A05D21B48C020B7E121399965C |     0      |   100000
 ```
 
 If one does not want to use the active address, the addresses can be specified:
 
 ```
 /wallet --no-shell gas --address 562F07CF6369E8D22DBF226A5BFEDC6300014837
-                Object ID                 |  Version   |  Gas Value 
+                Object ID                 |  Version   |  Gas Value
 ----------------------------------------------------------------------
- A8DDC2661A19010E5F85CBF6D905DDFBE4DD0320 |     0      |   100000   
- B2683D0B592E5B002D110989A52943BC9DA19158 |     0      |   100000   
- B41BF45B01C9BEFCE3A0A371E2B98E062691438D |     0      |   100000   
- BA9E10F319182F3BD584EDB92C7899CC6D018723 |     0      |   100000   
- F8BFE77A5B21E7ABFA3BC285991F9DA4E5CC2D7B |     0      |   100000  
+ A8DDC2661A19010E5F85CBF6D905DDFBE4DD0320 |     0      |   100000
+ B2683D0B592E5B002D110989A52943BC9DA19158 |     0      |   100000
+ B41BF45B01C9BEFCE3A0A371E2B98E062691438D |     0      |   100000
+ BA9E10F319182F3BD584EDB92C7899CC6D018723 |     0      |   100000
+ F8BFE77A5B21E7ABFA3BC285991F9DA4E5CC2D7B |     0      |   100000
 
 ```
 
@@ -972,3 +972,33 @@ pre-populate two gas objects for four newly generated accounts:
   ]
 }
 ```
+
+If you use any custom accounts in `genesis.conf`, ensure you have a corresponding private key in
+`wallet.key`. Ensure `wallet.key` is in the working directory of the wallet. If you do not have the private key of the addresses specified, you cannot use custom genesis. *Never share your private keys.* For testing the `genesis.conf` example below, you can use the following private key:
+
+`genesis.conf`
+
+```
+{
+  "authorities": [
+    {},{},{},{}
+  ],
+  "accounts": [
+    {
+      "address": "09818AAC3EDF9CF9B006B70C36E7241768B26386",
+      "gas_objects": [
+        {
+          "object_id": "0000000000000000000000000000000000000003",
+          "gas_value": 10000000
+        }
+      ]
+    }
+  ]
+}
+```
+`wallet.key`
+```
+[
+  "WKk4nT2oyPKbFrFAyepT5wEsummWsA6qdhsqzc6CVC9fvTt3J2u6yy5WuW9B6OU3mkcyPC/4Axstn0BpIhzZNg==",
+]
+```

sui/src/sui-move.rs

@@ -12,9 +12,9 @@ pub enum MoveCommands {
     /// Build and verify Move project
     #[clap(name = "build")]
     Build {
-        /// Whether we are printing in hex.
+        /// Whether we are printing in base64.
         #[clap(long)]
-        dump_bytecode_as_hex: bool,
+        dump_bytecode_as_base64: bool,
     },
 
     /// Run all Move unit tests
@@ -26,11 +26,11 @@ impl MoveCommands {
     pub fn execute(&self, path: &Path, is_std_framework: bool) -> Result<(), anyhow::Error> {
         match self {
             Self::Build {
-                dump_bytecode_as_hex,
+                dump_bytecode_as_base64,
             } => {
-                if *dump_bytecode_as_hex {
+                if *dump_bytecode_as_base64 {
                     let compiled_modules =
-                        sui_framework::build_move_package_to_hex(path, is_std_framework)?;
+                        sui_framework::build_move_package_to_base64(path, is_std_framework)?;
                     println!("{:?}", compiled_modules);
                 } else {
                     Self::build(path, is_std_framework)?;

sui_programmability/framework/Cargo.toml

@@ -9,6 +9,7 @@ publish = false
 
 [dependencies]
 anyhow = "1.0.55"
+base64 = "0.13.0"
 smallvec = "1.8.0"
 num_enum = "0.5.6"
 

sui_programmability/framework/src/lib.rs

@@ -7,10 +7,7 @@ use move_package::BuildConfig;
 use move_unit_test::UnitTestingConfig;
 use num_enum::TryFromPrimitive;
 use std::{collections::HashSet, path::Path};
-use sui_types::{
-    base_types::encode_bytes_hex,
-    error::{SuiError, SuiResult},
-};
+use sui_types::error::{SuiError, SuiResult};
 use sui_verifier::verifier as sui_bytecode_verifier;
 
 #[cfg(test)]
@@ -67,13 +64,16 @@ pub fn get_move_stdlib_modules(lib_dir: &Path) -> SuiResult<Vec<CompiledModule>>
     Ok(modules)
 }
 
-/// Given a `path` and a `build_config`, build the package in that path and return the compiled modules as Hex.
+/// Given a `path` and a `build_config`, build the package in that path and return the compiled modules as base64.
 /// This is useful for when publishing via JSON
 /// If we are building the Sui framework, `is_framework` will be true;
 /// Otherwise `is_framework` should be false (e.g. calling from client).
-pub fn build_move_package_to_hex(path: &Path, is_framework: bool) -> Result<Vec<String>, SuiError> {
+pub fn build_move_package_to_base64(
+    path: &Path,
+    is_framework: bool,
+) -> Result<Vec<String>, SuiError> {
     build_move_package_to_bytes(path, is_framework)
-        .map(|mods| mods.iter().map(encode_bytes_hex).collect::<Vec<_>>())
+        .map(|mods| mods.iter().map(base64::encode).collect::<Vec<_>>())
 }
 
 /// Given a `path` and a `build_config`, build the package in that path and return the compiled modules as Vec<Vec<u8>>.