Web3:// protocol engine split #12

nand2 · 2023-10-13T15:04:43Z

Hi!
Ok here is a PR where I extracted the web3:// execution engine, put it on https://github.com/web3-protocol/web3protocol-go and then reuse it on the gateway project.

Here are some things:

The conf is slightly changed, mainly all chain ids which were strings are now integers.
SimpleNameService : I did not find anything about that so I dropped it, but is it still something actively supported, and if yes, can you show me what it is?
Tests: I have worked on making them work all. Some comments:
- I have removed some which were internal to the protocol (e.g. auto mode arg parsing)
- Non-standard ERC-4804, with hosts such as "/concat.w3q->(string)/" : I have disabled
- The package now use a up-to-date go-ethereum dependency, and they recently are now using "input" instead of "data" in the JSON-RPC call (apparently "data" was kind of supported but was not the standard). Cf ethclient: use 'input', not 'data' as field for transaction input ethereum/go-ethereum#28078 As a consequence, several tests to blockchains ids 1666600000, 1666700000, 100001, 1088, 599, 84531 fails because their RPCs does not recognize "data". Commented.
- Tests to chain 110001 are timeout-ing, commented.

Lastly, I haven't supported yet the ?mime.type= and .mime.content= flags, will do so soon after, will be quick.

But first, do you have comments on the PR as it is now?

Regarding /concat.w3q->(string) URLs, if you still want to support them, I can do a bit a code to translate them in proper web3:// URLs before feeding them to the new engine.

…into a separate package, to be published later.

… toml files : work in progress

…. Work in progress.

qzhodl · 2023-10-16T07:58:11Z

cmd/server/main.go

+	}
+
+	// Setup name address cache to 10 minutes, hardocded
+	web3pConfig.NameAddrCacheDurationInMinutes = 10


Why do we want to hardcode the cache duration?

I guess not :) I just copy/pasted the existing hardcoded value from main.go:139
I will add an option to be loaded from the config file.

qzhodl · 2023-10-16T09:26:05Z

cmd/server/protocol.go

 			return "", false, fmt.Errorf("invalid subdomain")
 		}
-		full := strings.Join(pieces[0:2], ":")
+		name := pieces[0]


Similar logic shows multiple times, we may consider to wrap it as an internal function

Ok I'll see if I can cleanup handleSubdomain() a bit

qzhodl · 2023-10-16T09:47:25Z

SimpleNameService : I did not find anything about that so I dropped it, but is it still something actively supported, and if yes, can you show me what it is?

Seems no dApps are using the SimpleNameService right now. SimpleNameService was a simple name service used by chainId 3333. So I think we can remove that from the gateway.

qzhodl · 2023-10-16T09:57:16Z

Regarding /concat.w3q->(string) URLs, if you still want to support them, I can do a bit a code to translate them in proper web3:// URLs before feeding them to the new engine

If what you mean is the old return way for "->" instead of "returns", there is only one dApp "W3Blog" which is using it. We can definitely change the code of W3Blog to deprecate this old calling version.

nand2 · 2023-10-18T06:07:49Z

Regarding /concat.w3q->(string) URLs, if you still want to support them, I can do a bit a code to translate them in proper web3:// URLs before feeding them to the new engine

If what you mean is the old return way for "->" instead of "returns", there is only one dApp "W3Blog" which is using it. We can definitely change the code of W3Blog to deprecate this old calling version.

Ok! And Qi Zhou is also voting on removing support for this old calling version, so I'll remove the commented tests.

…d factorize a bit on hostChangeChainShortNameToId()

…y into 202309-engine-tests

nand2 · 2023-10-18T14:29:28Z

Ok I have used the cacheDurationMinutes flag, removed some commented tests for the old format (e.g. /concat.w3q->(string)), and factorized a bit with hostChangeChainShortNameToId.

It still remains for me to add support for the extra ERC ?mime.type= and mime.content= flags.

syntrust · 2023-10-19T08:45:15Z

Hi Nand,

This is really a considerable amount of change with many details that must have taken a lot of effort, especially since you have to go over the test cases one by one. And I can see a lot of improvements have been made such as making chainId integers, upgrading go-ethereum dependency, and some deprecations, etc., which I believe are the right things to do.
However, I may need some big picture about the split, like the principal about which parts are moved to the engine and which are left. I suppose those rules defined in EIPs are engine parts in general? Do you have any details to add? Any changes of logic made to the engine part in the new repo? Is there any other example that uses the engine?

syntrust · 2023-10-19T08:36:31Z

cmd/server/server_test.go

+// 	{100001, "0xc2f21F8F573Ab93477E23c4aBB363e66AE11Bac5.w3qkc.io", "/greet?returns=(string)", "[\"Hello QKC\"]", http.StatusOK},
+// 	{100001, "w3qkc.io", "/0xc2f21F8F573Ab93477E23c4aBB363e66AE11Bac5/greet?returns=(string)", "[\"Hello QKC\"]", http.StatusOK},
+// }



Should we support the cases of subdomain like in line 267, 269, etc?

You are right, I have commented them temporarily and forgot about them. I just made them work.

nand2 · 2023-10-26T07:17:36Z

Hi!

Thanks, yes sorry I guess I should have added more details!
So, yes the main idea is to separate the code executing the web3:// URL itself as defined in the EIPs, from the rest.
The "engine" interface is (via the function client.FetchUrl) :

Input: URL string
Output: output bytes, HTTP headers, HTTP status code

What remains in web3url-gateway is the http server itself, the gateway URL to web3:// URL translation (mostly done via the handleSubdomain function), the non-web3:// elements (BTC ordinal support), stats gathering and config support.

Coming back to the engine (web3protocol-go), I have taken the code from the gateway, and reworked it so that the 3 main execution steps are separated, in a way that can be easily tested :

URL parsing : determine what needs to be called (that includes resolve mode checking and domain name resolution)
Main contract call
Post processing of the data returned by the main contract
The other implementation in javascript use a very similar approach.

The tests at https://github.com/web3-protocol/web3protocol-tests have been put in toml files, and are made to be used by various implementations : this golang implementation, and very soon https://github.com/web3-protocol/web3protocol-js

Other place using this golang engine : right now only the tests (https://github.com/web3-protocol/web3protocol-go/blob/main/web3protocol_test.go ), I'll add a curl-like app soon. The javascript engine is used by a curl-like app and the EVM browser My hope is that others apps wanting to read web3:// URLs will either use this golang engine, or the JS version.

Changes of logic : there are very slight changes, it aims at supporting ERC-6860, the ERC that Qi Zhou and I are working on which clarifies and formalize ERC-4804 and make the following changes:

On auto mode, integers returned (via ?returns=(int)) are no longer returned as integer string ("200") but as hex strinbg ("0xc8")
It also adds:
On auto mode, support for all solidity types
I have also corrected tiny bugs here and there, and added more checks, otherwise things should work as it used to work.

…se the default one from the golang http server.

…rns args is handled.

nand2 · 2023-10-26T12:55:27Z

Ok I just made all tests pass by adding the ERC-7087 mime.type/mime.config support ; also corrected the way multiple ?returns args is supposed to be handled (the last one should be taken, instead of returning an error).

qzhodl · 2023-11-06T08:24:48Z

Too much changes are involved, I think it will be great to setup a gateway in the server to test the existing websites to see if they are broken, @syntrust

syntrust · 2023-11-06T08:35:20Z

Too much changes are involved, I think it will be great to setup a gateway in the server to test the existing websites to see if they are broken, @syntrust

Yes before merging to the main branch, I may need some time to deploy the branch to a test server for a sanity test in case there are any broken links, especially for the web pages deployed in the early days.
@nand2 @qzhodl

…of which being tuples support in ?return=)

nand2 · 2023-12-15T15:16:03Z

The last commit bump the web3protocol-go to v0.1.0, which add tiny changes (one being support of tuples in ?return=).

syntrust · 2024-01-02T06:32:00Z

This PR has been deployed to a test server for a bunch of sanity tests. The tests cover ethstorage.io (https://eth-store.w3eth.io/#/) and all links and dApps on https://w3url.w3eth.io/ (w3link.io) . All works well except links like

https://web3q.io/file.w3q/0x6d4a199f603b084a2f1761dc9f322f92e68bfd5e/demo.png

where

web3q.io will no longer supported so we need to replace it with w3link.io
the "path" format should be replaced with subdomain format

We may need to make some changes to dApps (qrobot, w3box, etc. ) using this kind of link.

syntrust · 2024-01-02T06:49:10Z

An edge case was reported as #15

https://0x31caefa22959db0581623618d9c554e6436aa4b4.w3q-g.w3link.io/0x6d4a199f603b084a2f1761dc9f322f92e68bfd5e-demo.6d7b6248.png

nand2 and others added 13 commits September 19, 2023 17:45

Work in progress: Spliting the web3 execution code from the gateway, …

bd68db5

…into a separate package, to be published later.

Extracting the web3 engine: Work in progress

82d3065

Splitting of the web3 protocol engine as a separate package, tests in…

25d6b28

… toml files : work in progress

Tests: Still adding them, and transitioning the code to a new package…

9ee82f8

…. Work in progress.

Most of auto mode URL parsing tests done.

a7bcd1b

Testing contract return processing

baa51c3

Fetching code.

7aaf3f4

ERC5219 support on the new separate package.

972601f

Code cleanup, preparing to separate.

cfcf5b0

Engine switched to web3protocol-go

5aca763

Handle most of the tests.

dab25ea

Handle encoding tests.

23c1f6a

Merge branch 'main' into 202309-engine-tests

7a1352a

qizhou requested review from qzhodl and syntrust October 16, 2023 00:49

qzhodl reviewed Oct 16, 2023

View reviewed changes

nand2 added 2 commits October 18, 2023 16:25

Remove tests with old syntax, use the cache duration config entry, an…

9077693

…d factorize a bit on hostChangeChainShortNameToId()

Merge branch '202309-engine-tests' of github.com:nand2/web3url-gatewa…

61f9fd1

…y into 202309-engine-tests

syntrust reviewed Oct 19, 2023

View reviewed changes

Supports TestW3urls tests. If no content-type should be returned, era…

1c2b3c8

…se the default one from the golang http server.

syntrust approved these changes Oct 26, 2023

View reviewed changes

Handle ERC-7087 mime.type/mime.config; correct the way multiple ?retu…

ef16060

…rns args is handled.

qzhodl mentioned this pull request Nov 6, 2023

unknown type: uint8 #8

Closed

qzhodl approved these changes Nov 6, 2023

View reviewed changes

nand2 and others added 2 commits December 14, 2023 08:54

Merge branch 'main' into 202309-engine-tests

b6fbf0a

Bump web3protocol-go to v0.1.0 (tiny changes since last version, one …

fa706df

…of which being tuples support in ?return=)

syntrust mentioned this pull request Jan 2, 2024

EthStorage Devs Meeting #63 Agenda ethstorage/pm#68

Closed

Merge branch 'main' into 202309-engine-tests

6e632a5

syntrust mentioned this pull request Jan 2, 2024

links update ethstorage/qrobot#1

Open

syntrust merged commit b763014 into ethstorage:main Jan 3, 2024

Web3:// protocol engine split #12

Web3:// protocol engine split #12

Uh oh!

Conversation

nand2 commented Oct 13, 2023

Uh oh!

qzhodl Oct 16, 2023

Choose a reason for hiding this comment

Uh oh!

nand2 Oct 18, 2023

Choose a reason for hiding this comment

Uh oh!

qzhodl Oct 16, 2023

Choose a reason for hiding this comment

Uh oh!

nand2 Oct 18, 2023

Choose a reason for hiding this comment

Uh oh!

qzhodl commented Oct 16, 2023

Uh oh!

qzhodl commented Oct 16, 2023

Uh oh!

nand2 commented Oct 18, 2023

Uh oh!

nand2 commented Oct 18, 2023

Uh oh!

syntrust commented Oct 19, 2023

Uh oh!

syntrust Oct 19, 2023

Choose a reason for hiding this comment

Uh oh!

nand2 Oct 26, 2023

Choose a reason for hiding this comment

Uh oh!

nand2 commented Oct 26, 2023

Uh oh!

nand2 commented Oct 26, 2023

Uh oh!

qzhodl commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syntrust commented Nov 6, 2023

Uh oh!

nand2 commented Dec 15, 2023

Uh oh!

syntrust commented Jan 2, 2024

Uh oh!

syntrust commented Jan 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qzhodl commented Nov 6, 2023 •

edited

Loading

syntrust commented Jan 2, 2024 •

edited

Loading