-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RPC interface: enclave should sign and encrypt return values to client (or use TLS into enclave) #202
Comments
Log TrustedCall, Error in Execute: [User] bin/substratee-client -p 9994 -P 2094 trusted set-balance //Alice 10 --mrenclave 7WQ7d9zbTFyDNLhikZ46z9mNdBwvm7UNdagPVFS97UHw --direct
[Client] Requesting shielding key
[WorkerApi Direct Client]: Sending request: "{\"jsonrpc\":\"2.0\",\"method\":\"author_getShieldingKey\",\"params\":[],\"id\":1}"
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_getShieldingKey\",\"params\":[],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_getShieldingKey","params":[],"id":1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[181,21,173,...,93,125,0,0],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [181, 21, 173, 21, ..., 125, 0, 0], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [173, 21, 123, ..., 93, 125], do_watch: false, status: Ok }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[181,21,173,...,125,0,0],\"id\":1}"
[WorkerApi Direct Client] Received server response: RpcResponse { jsonrpc: "2.0", result: [181, 21, 173, ..., 125, 0, 0], id: 1 }
[WorkerApi Direct Client] Decoded result: RpcReturnValue { value: [173, 21, 123, 34, ...., 93, 125], do_watch: false, status: Ok }
[WorkerApi Direct Client] Decoded value: "{\"n\":[21,200,94,128,....,249,150],\"e\":[1,0,0,1]}"
[+] Got RSA public key of enclave
[Client] Encrypting TOP: direct_call(TrustedCallSigned {
call: balance_set_balance(d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d (5GrwvaEF...),
d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d (5GrwvaEF...), 10, 10),
nonce: 0,
signature: AnySignature(0xc2c5a4f3c3a5a73116b08c90bb953ea1bff267f13d240478045fea7e42c8a145dcca0a84d7388c33107723ba0124d1e24da179b9a1496a71799885a98ba56d81) })
to [51, 169, 45, 75, ...., 137, 47]
[Client] Composing params to be included in json string: Request { shard: 0x60ae19aef5cccef534eb665c63c55a6cd17242bb1d4efd9235973b2319908632, cyphertext: [51, 169, 45,...., 137, 47] }
[Client] Sending json to server: RpcRequest { jsonrpc: "2.0", method: "author_submitAndWatchExtrinsic", params: [96, 174, 25, 174, ...., 137, 47], id: 1 }
[WorkerApi Direct Client]: Sending request: "{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[96,174,25,174, ...,137,47],\"id\":1}"
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[96,174,25,174, ....,137,47],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,174,.... ,137,47],"id":1}
[Enclave Jsonrpc author_submitAndWatch] Decoded params: Request { shard: 0x60ae19aef5cccef534eb665c63c55a6cd17242bb1d4efd9235973b2319908632, cyphertext: [51, 169, 45, ...., 137, 47] }
[Enclave top pool] Decrypted cyphertext: direct_call(TrustedCallSigned { call: balance_set_balance(, , 10, 10), nonce: 0, signature: <wasm:stripped> }), submitting TOP
[Enclave Jsonrpc author_submitAndWatch] Received response from top pool: Ok(0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f)
[Enclave Jsonrpc author_submitAndWatch] Returning encoded: RpcReturnValue { value: [201, 181, 226,..., 99, 127], do_watch: true, status: TrustedOperationStatus(Submitted) }
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[128,201,181, ....,127,1,1,0],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [128, 201, 181, ..., 127, 1, 1, 0], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [201, 181, 226, ..., 99, 127], do_watch: true, status: TrustedOperationStatus(Submitted) }
[Worker Server] Decoded value: 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f
[Worker Server] Mapping for watching purpose:
Key: 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f
Value: WatchingClient { client: Sender { token: Token(0), channel: mio::channel::SyncSender<Command>, connection_id: 1 }, response: RpcResponse { jsonrpc: "2.0", result: [128, 201, 181, ..., 127, 1, 1, 0], id: 1 } }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[128,201,181....,,127,1,1,0],\"id\":1}"
[Client] Received response from server: RpcResponse { jsonrpc: "2.0", result: [128, 201, 181, ...., 127, 1, 1, 0], id: 1 }
[Client] Decoded response result: RpcReturnValue { value: [201, 181, 226, ...., 99, 127], do_watch: true, status: TrustedOperationStatus(Submitted) }
Trusted call 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f is Submitted
[Enclave] Event!
[Enclave] Watch event: Updated state of hash 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f: Invalid
[Worker Server] Received status event of hash 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f: Invalid
[Worker Server] Retrieved stored response RpcResponse { jsonrpc: "2.0", result: [128, 201, 181, 226, ...., 127, 1, 1, 0], id: 1 }
[Worker Server] Updating old status RpcReturnValue { value: [201, 181, 226, 165, ..., 99, 127], do_watch: false, status: TrustedOperationStatus(Submitted) } to TrustedOperationStatus(Invalid)
[Client] Received response from server: RpcResponse { jsonrpc: "2.0", result: [128, 201, 181,..., 127, 0, 1, 10], id: 1 }
[Client] Decoded response result: RpcReturnValue { value: [201, 181, 226, ...., 99, 127], do_watch: false, status: TrustedOperationStatus(Invalid) }
Trusted call 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f is Invalid
[Worker Server] Removing hash 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f from storage map
[Enclave top pool] Removing hash 0xc9b5e2a5a31981ad52413ec2f338d123a89750d327abdfd92858e2a21609637f from watch list |
Log Trusted Getter: [User] bin/substratee-client -p 9994 -P 2094 trusted balance //Alice --mrenclave 9rRSQUy9Sf833E42qaEKqUWTqRNRjRDANtThCKEAoTjZ --shard 9rRSQUy9Sf833E42qaEKqUWTqRNRjRDANtThCKEAoTjZ
arg_who = "//Alice"
[Client] Requesting shielding key
[WorkerApi Direct Client]: Sending request: "{\"jsonrpc\":\"2.0\",\"method\":\"author_getShieldingKey\",\"params\":[],\"id\":1}"
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_getShieldingKey\",\"params\":[],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_getShieldingKey","params":[],"id":1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[181,21,173,...,,125,0,0],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [181, 21, 173, 21, ..., 125, 0, 0], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [173, 21, 123, ..., 93, 125], do_watch: false, status: Ok }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[181,21,173,21,...,125,0,0],\"id\":1}"
[WorkerApi Direct Client] Received server response: RpcResponse { jsonrpc: "2.0", result: [181, 21, 173, ..., 93, 125, 0, 0], id: 1 }
[WorkerApi Direct Client] Decoded result: RpcReturnValue { value: [173, 21, 123, 34,..., 93, 125], do_watch: false, status: Ok }
[WorkerApi Direct Client] Decoded value: "{\"n\":[21,200,94,128,...,249,150],\"e\":[1,0,0,1]}"
[+] Got RSA public key of enclave
[Client] Composing jsonrpc call "{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[131,134,150...,179,69],\"id\":1}"
[WorkerApi Direct Client]: Sending request: "{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[131,134,150,..,179,69],\"id\":1}"
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[131,134,150, ...,179,69],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[131,134,150,...,179,69],"id":1}
[Enclave Jsonrpc author_submitAndWatch] Decoded params: Request { shard: 0x8386965333c150abf4d5378cc44a2b0578fb1d13085918c65973fd22ba99b52c, cyphertext: [64, 197, 186, 137, ...., 179, 69] }
[Enclave top pool] Decrypted cyphertext: get(trusted(TrustedGetterSigned { getter: free_balance(), signature: <wasm:stripped> })), submitting TOP
[Enclave Jsonrpc author_submitAndWatch] Received response from top pool: Ok(0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f)
[Enclave Jsonrpc author_submitAndWatch] Returning encoded: RpcReturnValue { value: [252, 148, 22, ..., 90, 143], do_watch: true, status: TrustedOperationStatus(Submitted) }
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[128,252,148, ..., 143,1,1,0],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [128, 252, 148, ..., 143, 1, 1, 0], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [252, 148, 22, 56, ..., 90, 143], do_watch: true, status: TrustedOperationStatus(Submitted) }
[Worker Server] Decoded value: 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f
[Worker Server] Mapping for watching purpose:
Key: 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f
Value: WatchingClient { client: Sender { token: Token(0), channel: mio::channel::SyncSender<Command>, connection_id: 1 }, response: RpcResponse { jsonrpc: "2.0", result: [128, 252, 148, 22, ...., 143, 1, 1, 0], id: 1 } }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[128,252,148,...,143,1,1,0],\"id\":1}"
[Client] Received response RpcResponse { jsonrpc: "2.0", result:[128,252,148,...,143,1,1,0],, id: 1 }
[Enclave] Event!
[Enclave] Sending encoded state of hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f
[Worker Server] New state event of hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f
[Worker Server] Retrieving stored response RpcResponse { jsonrpc: "2.0", result: [128,252,148,...,143,1,1,0], id: 1 }
[Worker Server] Updating response result to RpcReturnValue { value: [0], do_watch: false, status: TrustedOperationStatus(Submitted) }
[Worker Server] Removing hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f from storage map
[Client] Received response RpcResponse { jsonrpc: "2.0", result: [4, 0, 0, 1, 0], id: 1 }
0
[Enclave] Event!
[Enclave] Watch event: Updated state of hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f: Invalid
[Worker Server] Received status event of hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f: Invalid
[Worker Server] Removing hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f from storage map
[Enclave top pool] Removing hash 0xfc941638552c82cc0301b605f92552a9f0a95bf4d78f65f9356082bfa86e5a8f from watch list |
Log Pending Extrinsic call {"jsonrpc": "2.0", "method": "author_pendingExtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt"], "id": 1}
[Worker Server] Received request '"{\"jsonrpc\": \"2.0\", \"method\": \"author_pendingExtrinsics\", \"params\": [\"AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt\"], \"id\": 1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc": "2.0", "method": "author_pendingExtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt"], "id": 1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[8,4,0,0,0],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [8, 4, 0, 0, 0], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [4, 0], do_watch: false, status: Ok }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[8,4,0,0,0],\"id\":1}"
{"jsonrpc":"2.0","result":[8,4,0,0,0],"id":1} |
Some invalid calls to log server and enclave error handling: Invalid encryption (error within top pool): {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25, ...., ,185,88],"id":1}
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[96,174,25,...,185,88],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,...,185,88],"id":1}
[Enclave Jsonrpc author_submitAndWatch] Decoded params: Request { shard: 0x60ae19aef5cccef534eb665c63c55a6cd17242bb1d4efd9235973b2319908632, cyphertext: [65, 133, 231, ..., 185, 88] }
[Enclave Jsonrpc author_submitAndWatch] Received response from top pool: Err(Error { code: ServerError(1001), message: "Trusted oprations could not be deciphered", data: None })
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[168,164,84,...,100,0,2],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [168, 164, 84, ...., 100, 0, 2], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [164, 84, 114, ...., 101, 100], do_watch: false, status: Error }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[168,164,84,...,100,0,2],\"id\":1}"
{"jsonrpc":"2.0","result":[168,164,84....,100,0,2],"id":1} Invalid call: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,88],"id":1}
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,88],\"id\":1}\n"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,88],"id":1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":[205,1,197,1,67,....,,46,0,2],\"id\":1}"
[Worker Server] Received: RpcResponse { jsonrpc: "2.0", result: [205, 1, 197,...., 46, 0, 2], id: 1 }
[Worker Server] Decoded result: RpcReturnValue { value: [197, 1, 67, 111, ...., 56, 46], do_watch: false, status: Error }
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":[205,1,197,1, ...,,46,0,2],\"id\":1}"
{"jsonrpc":"2.0","result":[205,1,197,1,....,46,0,2],"id":1} Invalid utf-8 encoded: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,R],"id":1}
[Worker Server] Received request '"{\"jsonrpc\":\"2.0\",\"method\":\"author_submitAndWatchExtrinsic\",\"params\":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,R],\"id\":1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc":"2.0","method":"author_submitAndWatchExtrinsic","params":[96,174,25,174,245,204,206,3000,91,87,97,218,129,126,108,185,R],"id":1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32700,\"message\":\"Parse error\"},\"id\":null}"
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32700,\"message\":\"Parse error\"},\"id\":null}"
{"jsonrpc":"2.0","error":{"code":-32700,"message":"Parse error"},"id":null} Call of non-existent rpc function (error within jsonrpc): {"jsonrpc": "2.0", "method": "author_pendinxtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt"], "id": 1}
[Worker Server] Received request '"{\"jsonrpc\": \"2.0\", \"method\": \"author_pendinxtrinsics\", \"params\": [\"AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt\"], \"id\": 1}"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc": "2.0", "method": "author_pendinxtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxNwNjurjoGRjwurc247nskjgt"], "id": 1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32601,\"message\":\"Method not found\"},\"id\":1}"
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32601,\"message\":\"Method not found\"},\"id\":1}"
{"jsonrpc":"2.0","error":{"code":-32601,"message":"Method not found"},"id":1} Invalid shard: {"jsonrpc": "2.0", "method": "author_pendingExtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxjurjoGRjwurc247nskjgt"], "id": 1}
[Worker Server] Received request '"{\"jsonrpc\": \"2.0\", \"method\": \"author_pendingExtrinsics\", \"params\": [\"AyF3sXQqWdyb8MXGkdHxjurjoGRjwurc247nskjgt\"], \"id\": 1}\n"', forwarding to enclave
[Enclave] Received Json String: {"jsonrpc": "2.0", "method": "author_pendingExtrinsics", "params": ["AyF3sXQqWdyb8MXGkdHxjurjoGRjwurc247nskjgt"], "id": 1}
[Enclave] Returning encoded: "{\"jsonrpc\":\"2.0\",\"result\":\"Shard ID is not of type H256\",\"id\":1}"
[Worker Server] Sending to client: "{\"jsonrpc\":\"2.0\",\"result\":\"Shard ID is not of type H256\",\"id\":1}"
{"jsonrpc":"2.0","result":"Shard ID is not of type H256","id":1} |
Besides the actual topic of this issue: This protocol is very inconsistent in handling errors:
We should either always be fully opaque (encoding everything as ciphertext "result", even if its an error, or let the untrusted worker see that there was an error, but then be opaque about the payload of the error:
|
Now, on the topic: We need to define what an untrusted worker can learn and what misbehaviour has to be prevented. This has to be properly documented in our book. threatsworker operator can (as long as we don't have TLS straight into the enclave. But even in that case, some of these threats basically exist)
categorizationWhat the untrusted worker may see (we have to accept this as the shielding key is not known a-priory, so at least
What the untrusted worker may NOT see:
authenticationWe can add a signature by the enclave signing-key with every response by adding an extra field:
encryptionThe client can look up the signing-key of the enclave on the blockchain. therefore don't need asymmetric encryption for responses.
Ooops, no, this doesn't work! |
For documentation: If we use TLS, the server could be run relatively smoothly directly within the enclave with sgx rustls. There's also an example with the server runnig outside of the enclave from teaclave. In case mio is run within enclave, don't forget to add |
Equal to issue #91 but for all return values from the enclave to the client:
"the client should be able to verify that what a worker sends him is indeed coming from the enclave.
Moreover, the payload should be encrypted such that the worker doesn't learn anything about the query"
The text was updated successfully, but these errors were encountered: