[JSON-RPC] Generate openrpc.json spec and rpc.discover endpoint #1588

patrickkuo · 2022-04-26T10:25:59Z

This PR added a proc macro to generate JSON-RPC spec file sui/open_rpc/spec/openrpc.json and rpc.discover method for the rpc-server.

The Open-RPC spec can be viewed here

Notable changes:

partial implementation of data fields in open rpc spec for the Sui Gateway Apis
new proc macro to automatically generate the open rpc spec
added JsonSchema derive to all data types being used by the gateway api
added JsonSchema impl for types belongs to move, e.g. TypeTag, StructTag, AccountAddress, Identifier etc... ( we shouldn't include these types in our GatewayAPI response, will be refactoring our return types in subsequence PR )
added doc for the rpc server -- The doc is modified from the rest-api.md as they share a lot of similarity.

Todos:
sui_moveCall require Base64 encoded bytes as input, which is not very json-rpc friendly, I will be changing it to use SuiJson instead so we can use strings as arg inputs
Refactor return types, not all data are useful to downstream and can be removed/ transformed to Json value.

Clay-Mysten · 2022-04-26T23:56:50Z

Hi Patrick,

I started my edits in a direct commit here but ran into this error when attempting to update and get the rpc-server binary:

claymurphy@Clays-MacBook-Pro ~ % rpc-server
zsh: command not found: rpc-server
claymurphy@Clays-MacBook-Pro ~ % cargo install --git https://github.com/MystenLabs/sui.git
    Updating git repository `https://github.com/MystenLabs/sui.git`
error: multiple packages with binaries found: sui, sui-faucet. When installing a git repository, cargo will always search the entire repo for any Cargo.toml. Please specify which to install.
claymurphy@Clays-MacBook-Pro ~ %

If you tell me the correct way to update via cargo I will document it on our Install page. Note we will also need to add the rpc-server binary to:
https://github.com/MystenLabs/sui/blob/6573ce9db1370f61ebe8334ab7ca946526c54582/doc/src/build/install.md#binaries

Include this new file in site navigation:
https://github.com/MystenLabs/sui/blob/main/doc/src/navconfig.json

And announce it in [siteconfig.json](https://github.com/MystenLabs/sui/blob/main/doc/src/siteconfig.json)

https://docs.sui.io/learn#see-whats-new - in [learn/index.md](https://github.com/MystenLabs/sui/blob/main/doc/src/learn/index.md)

I can take care of all of this. In fact, I just documented the process:
go/push-mysten-features

Please just help me get this up and running. Thanks!

666lcz · 2022-04-27T06:45:21Z

ran into this error when attempting to update and get the rpc-server binary:

@Clay-Mysten , we should update the command from

cargo install --git https://github.com/MystenLabs/sui.git

to

cargo install --git https://github.com/MystenLabs/sui.git sui

.

However, the latter command will also fail due to another issue, which should be fixed soon.

666lcz

Amazing work!

666lcz · 2022-04-27T07:13:39Z

sui/open_rpc/spec/openrpc.json

@@ -0,0 +1,2087 @@
+{


I was hoping that we can generate ts typing from this schema(so that we can automate the syncing) and tried loading this file with https://github.com/open-rpc/typings, unfortunately I got an out of memory error

npx open-rpc-typings --output-ts . <--- Last few GCs ---> [45334:0x160008000] 40122 ms: Mark-sweep 4042.4 (4128.9) -> 4031.2 (4133.9) MB, 4860.8 / 0.1 ms (average mu = 0.185, current mu = 0.054) allocation failure scavenge might not succeed [45334:0x160008000] 45056 ms: Mark-sweep 4047.5 (4133.9) -> 4034.2 (4136.7) MB, 4902.3 / 0.0 ms (average mu = 0.098, current mu = 0.006) allocation failure scavenge might not succeed <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x102c0a104 node::Abort() [/opt/homebrew/Cellar/node/17.5.0/bin/node] 2: 0x102c0b104 node::OnFatalError(char const*, char const*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 3: 0x102d6243c v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 4: 0x102d623d0 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 5: 0x102eb21d0 v8::internal::Heap::EnsureFillerObjectAtTop() [/opt/homebrew/Cellar/node/17.5.0/bin/node] 6: 0x102eb43a0 v8::internal::Heap::ComputeMutatorUtilization(char const*, double, double) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 7: 0x102eb2a04 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 8: 0x102eb0c74 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 9: 0x102ebdbbc v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 10: 0x102ebdc3c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 11: 0x102e956d4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 12: 0x10314a96c v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 13: 0x102a7d5ec Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/opt/homebrew/Cellar/node/17.5.0/bin/node] 14: 0x1093cbd50 15: 0x1093c9fe8 16: 0x10959b3e4 17: 0x1095b9c1c 18: 0x10935e304 19: 0x102ac56f8 Builtins_PromiseFulfillReactionJob [/opt/homebrew/Cellar/node/17.5.0/bin/node] 20: 0x102a32c54 Builtins_RunMicrotasks [/opt/homebrew/Cellar/node/17.5.0/bin/node] 21: 0x102a0fe84 Builtins_JSRunMicrotasksEntry [/opt/homebrew/Cellar/node/17.5.0/bin/node] 22: 0x102e4e668 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 23: 0x102e4e9d4 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 24: 0x102e6faf4 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 25: 0x102e6f920 v8::internal::MicrotaskQueue::PerformCheckpointInternal(v8::Isolate*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 26: 0x102b41ab4 node::InternalCallbackScope::Close() [/opt/homebrew/Cellar/node/17.5.0/bin/node] 27: 0x102b423fc node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 28: 0x102b5cc9c node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 29: 0x102c0f620 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 30: 0x102c1094c node::fs::AfterNoArgs(uv_fs_s*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 31: 0x104f338c0 uv__work_done [/opt/homebrew/Cellar/libuv/1.43.0/lib/libuv.1.dylib] 32: 0x104f36c38 uv__async_io [/opt/homebrew/Cellar/libuv/1.43.0/lib/libuv.1.dylib] 33: 0x104f46458 uv__io_poll [/opt/homebrew/Cellar/libuv/1.43.0/lib/libuv.1.dylib] 34: 0x104f37058 uv_run [/opt/homebrew/Cellar/libuv/1.43.0/lib/libuv.1.dylib] 35: 0x102b42c60 node::SpinEventLoop(node::Environment*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 36: 0x102c47b58 node::NodeMainInstance::Run(int*, node::Environment*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 37: 0x102c477ac node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 38: 0x102bd7dc0 node::Start(int, char**) [/opt/homebrew/Cellar/node/17.5.0/bin/node] 39: 0x1051cd0f4 [1] 45333 abort npx open-rpc-typings --output-ts .

hmmm the schema works with https://playground.open-rpc.org so it should be valid... I will investigate, we have some cyclic structs in the response type TypeTag maybe that's causing problem...

I will work on simplifying the responses after I finish other devnet tasks, hopefully we can remove TypeTag and make the schema simpler

666lcz · 2022-04-27T07:18:47Z

sui/src/rpc_gateway.rs

+        #[schemars(with = "json_schema::Identifier")] module: Identifier,
+        #[schemars(with = "json_schema::Identifier")] function: Identifier,
+        #[schemars(with = "Option<Vec<json_schema::TypeTag>>")] type_arguments: Option<
+            Vec<TypeTag>,
+        >,
        arguments: Vec<RpcCallArg>,
        gas_object_id: ObjectID,
        gas_budget: u64,


N00b question: why do we use the schemar marco on line 77 but not on line 82?

the schemars tag is used to tell the macro to generate schema with the provided Type instead of the actual declared Type, we need this because Identifier and TypeTag belongs to the move package so we can't implement JsonSchema trait for them, instead we use proxy types in json_schema to generate the schema.

patrickkuo · 2022-04-27T11:15:40Z

Hi Patrick,

I started my edits in a direct commit here but ran into this error when attempting to update and get the rpc-server binary:
claymurphy@Clays-MacBook-Pro ~ % rpc-server
zsh: command not found: rpc-server
claymurphy@Clays-MacBook-Pro ~ % cargo install --git https://github.com/MystenLabs/sui.git
    Updating git repository `https://github.com/MystenLabs/sui.git`
error: multiple packages with binaries found: sui, sui-faucet. When installing a git repository, cargo will always search the entire repo for any Cargo.toml. Please specify which to install.
claymurphy@Clays-MacBook-Pro ~ % 
If you tell me the correct way to update via cargo I will document it on our Install page. Note we will also need to add the rpc-server binary to: https://github.com/MystenLabs/sui/blob/6573ce9db1370f61ebe8334ab7ca946526c54582/doc/src/build/install.md#binaries

Include this new file in site navigation: https://github.com/MystenLabs/sui/blob/main/doc/src/navconfig.json

And announce it in [siteconfig.json](https://github.com/MystenLabs/sui/blob/main/doc/src/siteconfig.json)

https://docs.sui.io/learn#see-whats-new - in [learn/index.md](https://github.com/MystenLabs/sui/blob/main/doc/src/learn/index.md)

I can take care of all of this. In fact, I just documented the process: go/push-mysten-features

Please just help me get this up and running. Thanks!

Thanks! @Clay-Mysten

I will rebase this PR and go thought the cargo install process (didn't know it's broken), I will see if I can fix it, please bear with me :)

patrickkuo · 2022-04-27T13:10:44Z

@Clay-Mysten you can use cargo install --git https://github.com/MystenLabs/sui.git --branch pat/open_rpc sui to install bins from my branch, I have tried just now and it works fine.

Update Cargo install command per input in PR: #1588

Clay-Mysten · 2022-04-27T15:39:57Z

cargo install --git https://github.com/MystenLabs/sui.git sui
OK, I've started this update in:
#1611

But I would like to know why it is needed when the previous command had worked. The new one appears to be working for me now (or is at least fetching the packages without error).

Thanks!

Clay-Mysten · 2022-04-27T15:40:47Z

Also see the TODOs I left inline during my edits.

* more refactoring * updated doc

added JsonSchema for MoveStructLayout

Start edits and add TODOs to RPC server doc

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

* add open rpc rpc.discover support * * generated open rpc spec in sui/open_rpc/spec * more refactoring * updated doc * update doc * update rpc spec * rename package * remove unused deps * more docs added JsonSchema for MoveStructLayout * Update json-rpc.md Start edits and add TODOs to RPC server doc * address comments in doc Co-authored-by: Clay-Mysten <100217682+Clay-Mysten@users.noreply.github.com>

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

* add open rpc rpc.discover support * * generated open rpc spec in sui/open_rpc/spec * more refactoring * updated doc * update doc * update rpc spec * rename package * remove unused deps * more docs added JsonSchema for MoveStructLayout * Update json-rpc.md Start edits and add TODOs to RPC server doc * address comments in doc Co-authored-by: Clay-Mysten <100217682+Clay-Mysten@users.noreply.github.com>

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

patrickkuo force-pushed the pat/open_rpc branch from cf0e66b to 09739f4 Compare April 26, 2022 15:38

patrickkuo marked this pull request as ready for review April 26, 2022 17:05

patrickkuo requested a review from Clay-Mysten as a code owner April 26, 2022 17:05

patrickkuo requested review from huitseeker, lxfind, 666lcz, apburnie, awelc, bmwill, oxade, stella3d and velvia April 26, 2022 17:06

666lcz approved these changes Apr 27, 2022

View reviewed changes

patrickkuo force-pushed the pat/open_rpc branch from 82708c5 to 715a571 Compare April 27, 2022 11:24

Clay-Mysten added a commit that referenced this pull request Apr 27, 2022

Update install.md

dd2621d

Update Cargo install command per input in PR: #1588

Clay-Mysten mentioned this pull request Apr 27, 2022

Update install.md #1611

Merged

patrickkuo and others added 8 commits April 28, 2022 13:37

add open rpc rpc.discover support

1f979af

* generated open rpc spec in sui/open_rpc/spec

6e1e359

* more refactoring * updated doc

update doc

9eda6d3

update rpc spec

e20b495

rename package

54a984d

remove unused deps

a0fff47

more docs

47ee4bf

added JsonSchema for MoveStructLayout

Update json-rpc.md

65ef90c

Start edits and add TODOs to RPC server doc

address comments in doc

a96cc1a

patrickkuo force-pushed the pat/open_rpc branch from 3b3f5ff to a96cc1a Compare April 28, 2022 12:48

patrickkuo linked an issue Apr 28, 2022 that may be closed by this pull request

[RPC gateway] - Investigate open-rpc doc generation from serde for TransactionResponse and ObjectRead #1160

Closed

patrickkuo merged commit a626f6e into main Apr 28, 2022

patrickkuo deleted the pat/open_rpc branch April 28, 2022 13:24

Clay-Mysten added a commit that referenced this pull request Apr 28, 2022

Update install.md (#1611)

d56087d

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

Clay-Mysten mentioned this pull request May 10, 2022

Handle API versioning #1244

Closed

longbowlu pushed a commit that referenced this pull request May 12, 2022

Update install.md (#1611)

a54635a

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

punwai pushed a commit that referenced this pull request Jul 27, 2022

Update install.md (#1611)

02425bc

* Update install.md Update Cargo install command per input in PR: #1588 * Update install.md * Update install.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JSON-RPC] Generate openrpc.json spec and rpc.discover endpoint #1588

[JSON-RPC] Generate openrpc.json spec and rpc.discover endpoint #1588

patrickkuo commented Apr 26, 2022 •

edited

Loading

Clay-Mysten commented Apr 26, 2022

666lcz commented Apr 27, 2022

666lcz left a comment

666lcz Apr 27, 2022

patrickkuo Apr 27, 2022

666lcz Apr 27, 2022

patrickkuo Apr 27, 2022

patrickkuo commented Apr 27, 2022

patrickkuo commented Apr 27, 2022

Clay-Mysten commented Apr 27, 2022

Clay-Mysten commented Apr 27, 2022

[JSON-RPC] Generate openrpc.json spec and rpc.discover endpoint #1588

[JSON-RPC] Generate openrpc.json spec and rpc.discover endpoint #1588

Conversation

patrickkuo commented Apr 26, 2022 • edited Loading

Clay-Mysten commented Apr 26, 2022

666lcz commented Apr 27, 2022

666lcz left a comment

Choose a reason for hiding this comment

666lcz Apr 27, 2022

Choose a reason for hiding this comment

patrickkuo Apr 27, 2022

Choose a reason for hiding this comment

666lcz Apr 27, 2022

Choose a reason for hiding this comment

patrickkuo Apr 27, 2022

Choose a reason for hiding this comment

patrickkuo commented Apr 27, 2022

patrickkuo commented Apr 27, 2022

Clay-Mysten commented Apr 27, 2022

Clay-Mysten commented Apr 27, 2022

patrickkuo commented Apr 26, 2022 •

edited

Loading