Getting soljson.js via dedicated solc-bin.ethereum.org domain #479
Conversation
- NOTE: This is only for curl. https.get() used in three other places won't follow redirects.
|
|
||
| var mem = new MemoryStream(null, { readable: false }); | ||
| https.get('https://raw.githubusercontent.com/ethereum/solc-bin/gh-pages/bin/list.json', function (response) { | ||
| https.get('https://solc-bin.ethereum.org/bin/list.json', function (response) { |
There was a problem hiding this comment.
https.get() won't handle HTTP redirects as far as I know. Unfortunately it looks like you need an external library for that. Any suggestions? Is it ok to add a new dependency just for that?
There was a problem hiding this comment.
Oh really? Yeah, please either add the most lightweight library or write a simple function that follows at most 3 redirect or something.
There was a problem hiding this comment.
I'll do it in a separate PR not to block this one. Right now we don't have any redirects in solc-bin anyway.
|
Oh this is really annoying. Can we maybe find a solution where we first try one url and then fall back to the other if it fails? We could even handle the "non-working symlinks" in such a way. |
|
@chriseth I just tried to reproduce the rate limiting issue to be able to test the fallback and manged to make a ton of requests without actually running into the limit. I did 1k requests to So I'm not sure if the fallback is really needed given that we're switching to S3 anyway. I think that using the |
|
I just noticed that it was still a draft. I'm marking it as ready now. |
Currently
solc-jsis not using solc-bin.ethereum.org domain to getsoljson.jsbinaries fromsolc-binrepository. Instead it fetches the files from raw.githubusercontent.com URLs. This prevents us from freely reorganizing the structure ofsolc-binsince, unlike Github Pages, it does not provide the original file content when you access it via a symlink. Instead you get a text file with a path andsojc-jscan't handle that currently (argotorg/solc-bin#43, argotorg/solc-bin#44).Not using a domain under our control is also an obstacle to transparently moving files to a different hosting and this is exactly what we're facing now (see argotorg/solidity#9258). We can fix it in newer versions of
solc-jsbut the old ones will keep trying to get them from the old location.This PR fixes the immediate issue by switching the URLs to solc-bin.ethereum.org. I originally assumed this to be just an oversight but after browsing through issues and PRs in this repo I see that the switch to raw.githubusercontent.com was actually intentional (#446, #445). It seems that the reason was that GH pages are rate-limited. This means that switching to the official domain would bring the problem back as it's currently pointing at GH pages too. I'm in the process of preparing an Amazon S3 bucket to mirror
solc-bincontent and I think I'll have to keep this PR as a draft until that's ready and the official domain is pointing at it.