📖 Removal of Bytecode Metadata

[ardislu]

The README currently has this note regarding bytecode metadata:

IMPORTANT
We removed the metadata hash bytecodeHash from the bytecode in order to guarantee a deterministic compilation across all operating systems. This implies that all sourcify.eth verifications have partial verification, as opposed to perfect verification, which requires a matching metadata hash.

I compiled the smart contract using the standard input JSON linked in the README (this file) on both Windows and Windows Subsystem for Linux (WSL) and the bytecode matches across systems. I tested compiling both with and without the bytecode metadata.

See my repo CreateX-compilation-test to reproduce:

Windows (PowerShell):

PS> solc --version
solc, the solidity compiler commandline interface
Version: 0.8.23+commit.f704f362.Windows.msvc

PS> .\build.ps1

Algorithm       Hash                                                                   Path
---------       ----                                                                   ----
SHA256          54A8FA2DF078CB1199A4479FB1277AA1148B107896BE2AE5D900A4E5607A18B1       C:\...\CreateX-powershell.bin
SHA256          318D86DA5FBEA2FE20131FC3C16F4F0F01A4BF86371549546FFCC10A607F17C0       C:\...\CreateX-with-metadata-powershell.bin

WSL (bash):

$ solc --version
solc, the solidity compiler commandline interface
Version: 0.8.23+commit.f704f362.Linux.g++

$ ./build.sh
54a8fa2df078cb1199a4479fb1277aa1148b107896be2ae5d900a4e5607a18b1  ./CreateX-bash.bin
318d86da5fbea2fe20131fc3c16f4f0f01a4bf86371549546ffcc10a607f17c0  ./CreateX-with-metadata-bash.bin

Can you share more details about which parts of the compilation are affected by the bytecodeHash? Am I misunderstanding the note?

If there's differences in other solc outputs besides the bytecode, can these differences be resolved without removing the bytecodeHash? Removing this metadata seems like a drastic step. As described in the README note, lack of bytecode metadata means source code verification is impossible without some reliance on an off-chain mechanism to connect the source code to the smart contract. This affects not just Sourcify, but also any independently-developed smart contract source code verifier in the future.

[pcaversaccio]

If there's differences in other solc outputs besides the bytecode, can these differences be resolved without removing the bytecodeHash? Removing this metadata seems like a drastic step. As described in the README note, lack of bytecode metadata means source code verification is impossible without some reliance on an off-chain mechanism to connect the source code to the smart contract. This affects not just Sourcify, but also any independently-developed smart contract source code verifier in the future.

@ardislu very good question! Let me elaborate on our design decision:

1. Since this factory is/will be deployed on probably hundreds of different EVM chains, having a deterministic compilation across all operating systems is the best way to enable reproducibility (also, see this tweet).
2. Furthermore, one of the general issues with factory contracts is that they can have double metadata hashes (e.g. see here or here). We didn't want to risk to encounter such an issue. The problem is that any double metadata hashes would even remove the possibility of a partial verification match on Sourcify. Thus, we had to balance this trade-off in the best way.
3. Regarding your question about non-deterministic behaviour of the Solidity compiler. As an example, they patched an issue in v0.8.21 about not entirely deterministic order of functions in unoptimized Yul output (see here; the choice of C++ compiler (i.e. Clang / GCC) in some cases would result in different (but equivalent) bytecode (especially from native binaries vs emscripten binaries).
4. This factory has been verified on +55 block explorers so far and the bytecode semantics is verified. Furthermore, as we write in the README, we recommend verifying prior to interacting with CreateX on any chain, that the keccak256 hash of the broadcasted contract creation bytecode is 0x12ec861579b63a3ab9db3b5a23c57d56402ad3061475b088f17054e2f2daf22f or of the deployed runtime bytecode is 0xbd8a7ea8cfca7b4e5f5041d7d4b17bc317c5ce42cfbc42066a00cf26b43eb53f. This will ensure that the bytecode always behaves as intended.
5. Let me address your concern about "As described in the README note, lack of bytecode metadata means source code verification is impossible without some reliance on an off-chain mechanism to connect the source code to the smart contract. This affects not just Sourcify, but also any independently-developed smart contract source code verifier in the future." I don't think this statement is correct: The source code is perfectly verifiable (i.e. all the opcodes used), it's more an issue about changes in the comments, filename etc. If this a major concern for any user, they can still verify the deployed logic via the hashes above to ensure that the logic itself is correct.

I hope this clarifies our choice.

[ardislu]

Thank you for the detailed response!

1. Since this factory is/will be deployed on probably hundreds of different EVM chains, having a deterministic compilation across all operating systems is the best way to enable reproducibility (also, see this tweet).

I appreciate the desire to reduce risk and understand it's probably too late to reverse a decision like this now. I was just curious to understand the problem for my own knowledge.

2. Furthermore, one of the general issues with factory contracts is that they can have double metadata hashes (e.g. see here or here). We didn't want to risk to encounter such an issue. The problem is that any double metadata hashes would even remove the possibility of a partial verification match on Sourcify. Thus, we had to balance this trade-off in the best way.

Thank you for sharing these links, I wasn't aware of this problem with factory contracts. My understanding is that the problems in those issues were from smart contracts referencing other smart contracts, possibly in a circular manner. It also looks like there were issues with recovering the original JSON metadata to recompile the smart contracts. However, since you've already flattened the smart contract into a standalone input JSON, wouldn't these issues be avoided? With this standard input JSON, shouldn't solc be deterministic regardless of the inclusion or exclusion of the bytecode metadata?

3. Regarding your question about non-deterministic behaviour of the Solidity compiler. As an example, they patched an issue in v0.8.21 about not entirely deterministic order of functions in unoptimized Yul output (see here; the choice of C++ compiler (i.e. Clang / GCC) in some cases would result in different (but equivalent) bytecode (especially from native binaries vs emscripten binaries).

I'm still unclear on how the inclusion or exclusion of bytecode metadata affects the compilation; the diff in the issue you linked shows the actual opcodes of the smart contract change between compilers:

@@ -13,11 +13,11 @@
 01a7818361017f565b5050919050565b5f6101b96002610187565b905090565b5f81905091905056
 5b5f7fff000000000000000000000000000000000000000000000000000000000000008216905091
 9050565b5f8160f81b9050919050565b5f61021861021361020e846101be565b6101f2565b6101c7
-565b9050919050565b7f4e487b710000000000000000000000000000000000000000000000000000
-00005f52603260045260245ffd5b5f60029050919050565b5f6102608261024c565b831061026f57
-61026e61021f565b5b6020830280830191505092915050565b5f819050919050565b5f8190509190
-50565b5f6102ab6102a66102a1846101be565b610288565b61027f565b9050919050565b6102bb82
+565b9050919050565b5f819050919050565b5f819050919050565b5f61024b610246610241846101
+be565b610228565b61021f565b9050919050565b7f4e487b71000000000000000000000000000000
+000000000000000000000000005f52603260045260245ffd5b5f60029050919050565b5f61029382
+61027f565b83106102a2576102a1610252565b5b6020830280830191505092915050565b6102bb82
 6101c7565b81525050565b60025f6102cc6101ae565b8091505f6102d9816101fe565b83805f8361
-02f8816102f36102ed85610291565b86610256565b6102b2565b505050505050505050565bfea264
+02f8816102f36102ed85610231565b86610289565b6102b2565b505050505050505050565bfea264
 6970667358221220a48f1dac064e7879796321b58551b9e9892676c3ba4d848e77dd46134e985a74
 64736f6c63430008140033

However, the bytecode metadata is still exactly the same between the two compilers. This part:

                                                                        ... a264
6970667358221220a48f1dac064e7879796321b58551b9e9892676c3ba4d848e77dd46134e985a74
64736f6c63430008140033

So I'm not clear on how excluding the bytecode metadata would have helped with this issue?

4. This factory has been verified on +55 block explorers so far and the bytecode semantics is verified. Furthermore, as we write in the README, we recommend verifying prior to interacting with CreateX on any chain, that the keccak256 hash of the broadcasted contract creation bytecode is 0x12ec861579b63a3ab9db3b5a23c57d56402ad3061475b088f17054e2f2daf22f or of the deployed runtime bytecode is 0xbd8a7ea8cfca7b4e5f5041d7d4b17bc317c5ce42cfbc42066a00cf26b43eb53f. This will ensure that the bytecode always behaves as intended.
5. Let me address your concern about "As described in the README note, lack of bytecode metadata means source code verification is impossible without some reliance on an off-chain mechanism to connect the source code to the smart contract. This affects not just Sourcify, but also any independently-developed smart contract source code verifier in the future." I don't think this statement is correct: The source code is perfectly verifiable (i.e. all the opcodes used), it's more an issue about changes in the comments, filename etc. If this a major concern for any user, they can still verify the deployed logic via the hashes above to ensure that the logic itself is correct.

The main issue is that without the metadata, it's no longer possible to even find the source code for this smart contract without reliance on some centralized repository (whether Etherscan, Sourcify, or GitHub). However, with the metadata, the bytecode is permanently linked on-chain to the IPFS hash of its source code/compilation settings. I could then retrieve the source code using my own IPFS node and then recompile the source code locally to verify the smart contract, completely independent and without knowledge of this GitHub repo. However, I fully acknowledge that this is a pretty specific/contrived scenario and you need to weigh in pragmatic concerns in this decision.

Just taking a step back from the more philosophical side of my comment though, the main thing I wanted to learn was how exactly the inclusion or exclusion of bytecodeHash affected deterministic compilation across systems. My understanding is that, given:

• the standard input JSON, and
• solc compiler version v0.8.23+commit.f704f362 (any platform)

then the same exact bytecode is always produced, regardless of inclusion or exclusion of bytecodeHash. Or, in the event of a compiler issue like in ethereum/solidity#14361 then the bytecode will always be different between systems, again regardless of inclusion or exclusion of bytecodeHash.

Thank you for your patience - not trying to be difficult, just trying to understand the problem fully.

[pcaversaccio]

I appreciate the desire to reduce risk and understand it's probably too late to reverse a decision like this now. I was just curious to understand the problem for my own knowledge.

That's completely fine 😄! And yes, it's too late since the code is frozen and deployed.

Thank you for sharing these links, I wasn't aware of this problem with factory contracts. My understanding is that the problems in those issues were from smart contracts referencing other smart contracts, possibly in a circular manner. It also looks like there were issues with recovering the original JSON metadata to recompile the smart contracts. However, since you've already flattened the smart contract into a standalone input JSON, wouldn't these issues be avoided? With this standard input JSON, shouldn't solc be deterministic regardless of the inclusion or exclusion of the bytecode metadata?

In theory yes, but as a person who has been debugging block explorer verification for over 5 years, I've encountered very weird issues already. As a recent example (Vyper-related): vyperlang/vyper#3370: Etherscan was unable to verify certain Vyper contracts due to a non-deterministic behaviour of set in their Python versions & setup (it took 2 months to debug this issue). As a side note, Etherscan doesn't care about the metadata hash (unfortunately). I want to emphasise here, that the decision taken here is focused on circumventing practical issues and not how in theory everything should work. There are plenty of buggy block explorers and we wanted to ensure the best stability across so many EVM chains. Example: I have not been able to verify the CreateX contract on Harmony test and production networks since they don't allow for standard JSON inputs & I can't configure the EVM version (which is in our case a non-default value: paris).

I'm still unclear on how the inclusion or exclusion of bytecode metadata affects the compilation;

Well, the bytecodeHash is obviously sensitive to any interpretation of spaces, characters etc. So if you have an OS that interprets e.g. a space in the comments differently, the hash will be different and thus the compiled bytecode will differ (i.e. the appended hash will differ). I have already encountered such issues using Gentoo Linux for example. So, removing the bytecodeHash will ensure that all OSes used by block explorers will always match the deployed bytecode. -> I think that's the answer you have been looking for.

Furthermore, I want to point out that we haven't removed all metadata, but just the bytecodeHash. You can see that the Solidity version is CBOR-encoded at the end:

However, I fully acknowledge that this is a pretty specific/contrived scenario and you need to weigh in pragmatic concerns in this decision.

We are aware of these downsides and it's really about having a practical, scalable solution. The same reasoning applies to how we manage the deployment approach using a deployer key (see the discussion here). A keyless approach is in theory the best solution, but doesn't work well in practice...