Make AppImage update tools aware of GitHub's rate limit

[simoniz0r]

Continued from https://github.com/AppImage/AppImageKit/issues/653#issuecomment-380662974

... the AppImage update tools should be aware of GitHub's rate limit and possibly even allow users to login for checks made to GitHub's API if they are making use of GitHub's API. Only 60 checks per hour are allowed without authentication (this goes up to 5000 per hour with authentication); after you hit the limit, you just get an error response. A user could very easily hit that limit and even possibly raise some flags on GitHub's end if they continue to try to connect after hitting that limit.

https://developer.github.com/v3/rate_limit/

[TheAssassin]

@simoniz0r you're right, this might be an issue, I recently hit the rate limit myself a few times with an other application.

I'd rather consider using web scraping to work around these restrictions, though. GitHub doesn't need to track update checks, IMO. Also, people shouldn't have to login with GitHub just to check for updates...

[antony-jr]

May be we could use the developers github token ? But keeping it safe is another big deal... Webscraping is a good idea but it can put a lot of development overhead.

[TheAssassin]

But keeping it safe is another big deal

How would you keep it "safe"? And, more importantly, imagine if 2500 users use that token to perform 1 update check, the rate limit is triggered...

[simoniz0r]

I've personally found web scraping to be much, much less reliable than using GitHub's API. Authentication is pretty easy to do and completely mitigates this issue; no average user is going to hit the rate limit at 5000 checks per hour.

May be we could use the developers github token ? But keeping it safe is another big deal... Webscraping is a good idea but it can put a lot of development overhead.

That's not gonna work. The token will keep the rate limit with it, so all users who use AppImage update tools would only be able to do 5000 checks per hour total and not 5000 checks per hour individually as they would if they authenticated with their own credentials.

[TheAssassin]

The first thing I'd do here is to make AppImageUpdate "aware" of the rate limit by evaluating API responses and checking whether the rate limit has been hit (showing an error with a message).

We could think about using some environment variable like GITHUB_TOKEN to allow users to pass the token to the tool. Then we can think about further measures.

[antony-jr]

We could think about using some environment variable like GITHUB_TOKEN to allow users to pass the token to the tool. Then we can think about further measures.

This seriously makes the end-user suffer a lot. I guess web-scraping is the best choice here because github does not update the site very often.

[antony-jr]

For a end-user , He/She must create a github account and get a token just for checking updates ? I would rather download the new version directly from the github releases myself.

[TheAssassin]

@antony-jr you got me wrong there, the requests are of course continued to be made anonymously first. The token will only be used once the rate limit is hit.

Anything else would break with my privacy philosophies. I am not convinced that a token that does indeed solve the issue is a good idea privacy wise, but I would agree to optionally offer such a solution for now, to help users affected by the bug right now.

Of course it doesn't make sense to always require a token. And even if that was a necessity, the UX of setting an environment variable is really... well, you can imagine. As said, we will need to think of alternative ways of asking users to log into GitHub when making update checks.

I'm sure @simoniz0r didn't suggest to require a GitHub token as well. I just imagine @simoniz0r has a lot of AppImages, and after a while, the rate limit is being hit, making future update checks impossible. That might happen even more often in the future, since more and more AppImages are being published every day. Therefore, it's good to discuss the issue now before the majority of users will hit this issue.

[antony-jr]

@TheAssassin Got it 👍

Note : Even jspm uses the environmental variable to solve this issue.

[simoniz0r]

the UX of setting an environment variable is really... well, you can imagine.

Some Desktop Environments actually allow you to set environment variables for the whole session in an easy to use GUI. LXQt, for example, has a very easy to use option to do this in their session manager tool.

It's really not the greatest solution, but scraping the raw HTML has been much less reliable for me compared to using their API.

[TheAssassin]

It's not like I was keen on writing some screen scraping in C++... But on the other hand, imagine a user had 60 AppImages of which >= 50% uses GitHub (not that unlikely, really), being a non-dev, I don't really want to ask them to register with a specific service for update checks.

[simoniz0r]

A much nicer alternative would be for AppImageHub to keep track of AppImage versions so that the user doesn't ever have to.

[antony-jr]

A much nicer alternative would be for AppImageHub to keep track of AppImage versions so that the user doesn't ever have to.

@simoniz0r Using version numbers is not a good idea and not all developers wants to register their appimage in the AppImageHub and keep on updating the version info.