macOS binaries with correct version strings

[cameel]

PR branch based on #77. Don't merge until that PR is merged! It's on gh-pages now.
Fixes argotorg/solidity#10183.

This PR contains macOS binaries in versions between 0.3.6 and 0.6.0 rebuilt with the change from #75. This fixes the discrepancy in the generated bytecode.

Old binaries are preserved in macosx-amd64/retired-rebuilds-with-version-discrepancy/. I'm not sure if it's worth keeping them. They do let people verify stuff they have already built using them but hopefully not many people were affected by that - macOS binaries were added only very recently.

[cameel]

EDIT 2021-01-24: This comment is now out of date. See updated details further down.

Current status of the bytecode comparison (from #77, which is still WIP): https://github.com/cameel/solc-bin/pull/1/checks?check_run_id=1594500955

Here's the relevant part of the output: bytecode-comparison-2020-12-22.txt

There are still 13 versions with differences (and 0.4.11 which was skipped in this run). From what I've seen so far, it's likely that prepare_report.py just needs more workarounds for the older compilers but we'll see.

I have already detected one binary with bytecode differences, though it's a Linux one: solc-linux-amd64-v0.4.10+commit.f0d539ae actually identifies itself as solc-linux-amd64-v0.4.10+commit.f0d539ae.

[cameel]

I'm done verifying the binaries. Unfortunately there are a few versions with problems that prevent the PR check from #77 from passing. This is, however, all caused by factors other than the macOS rebuild - bytecode differences from binaries on other platforms and macOS differences that can't really be fixed. See below for details.

Report differences

• Emscripten binaries for version 0.3.6 were not built from the exact commit which is tagged as v0.3.6 so they have a different version string and produce different bytecode than the macOS binary. See Emscripten binaries for version 0.3.6 in solc-bin were not built from the commit tagged as v0.3.6 solidity#10846.
• In 0.4.7 and 0.4.8 the bytecode is different on every platform and there's not much that can be done about it. See Binaries for versions 0.4.7 and 0.4.8 in solc-bin produce different bytecode on different platforms solidity#10841.
• On 0.4.10 and 0.4.26 it's the Linux binaries that fail the comparison. See Linux binary for version 0.4.10 in solc-bin was built from a different commit than binaries for other platforms solidity#10839 and Linux binary for version 0.4.26 in solc-bin produces different bytecode than other platforms solidity#10840.
• 0.4.11 seemingly fails comparison on all platforms but that's only because the results get diffed with the wasm report, which is the broken one in this case.
  • The problem is that there are some LLL test cases that crash the emscripten versions of the compiler when passed in as Solidity code. Example of a test case that causes a crash: (returnlll (return (exp (- 0 2) 3))). This does not happen with native binaries.
  • I have added a try/catch block in prepare_report.js to handle the crash but unfortunately the compiler prints an exit code to stdout before it crashes. This throws off the comparison. If you download the reports and manually remove all lines that consist only of the number 5 or 26, the content becomes the same on all platforms. There are no actual differences in the bytecode.
    • I could work around it by rewriting prepare_report.js to write to file (rather than to stdout) but I'm not sure it's worth the extra effort just for this single problematic case.

In all other cases the reports are identical on all platforms (except Windows which I could not check). Here's the most relevant part of the output of the last run of the compare job if you want to see the details: bytecode-comparison-2021-01-25.txt.

To make sure the binaries I did not replace are fine too, I have created a PR in my fork that touched all the binaries to force a CI run covering all of them. Here are the results: bytecode-comparison-all-2021-01-25.txt. There are no extra differences.

[cameel]

Also, I decided not to leave the buggy builds in a subdirectory. They should not be used in normal circumstances and they're still available in repo history in case anyone needs them for verification anyway. I have created a tag to make finding them easier: macosx-binaries-reporting-wrong-versions.

[cameel]

We have decided to close argotorg/solidity#10841 and leave the 0.4.7 and 0.4.8 binaries in the repo.

In that case should I hard-code the PR check to skip these versions? This is an option but it might also be misleading (by not being apparent that they were skipped) and we're not likely to modify these binaries in the future anyway.

[cameel]

Rebased on gh-pages with conflicts resolved.
Let's make sure it passes the checks though.

[axic]

Did we end up releasing the blog post about these?