Move to *.scryfall.io for card sources

[Elarnon]

Scryfall changed its image URIs: https://scryfall.com/blog/upcoming-api-changes-to-scryfall-image-uris-and-download-uris-224

Magic-Token still uses old c1.scryfall.com URIs instead of the new cards.scryfall.io URIs.

[SlightlyCircuitous]

Is this as straightforward as changing all instances of "c1.scryfall.com/file/scryfall-cards" to "cards.scryfall.io" and leaving the rest of the url the same?

[Elarnon]

I believe that should work, at least as a quick fix, although ideally it'd be better to use an api.scryfall.com url (possibly with an accompanying script to resolve the redirects) in case the urls change again.

(More specifically, cards.scryfall.io URLs are aggressively cached and api.scryfall.com URLs do appropriate cache busting)

[SlightlyCircuitous]

yeah, that would definitely be more ideal. I'm just not super familiar with how the code that reads the token.xml works atm so I don't know what (if anything) would need to change to use the API instead.

[ebbit1q]

to have the token.xml use the api all that needs to change is the picurl in the xml, cockatrice will resolve redirects as normal

[tooomm]

cockatrice will resolve redirects as normal

I'm wondering if the API rate limits could be an issue for Cockatrice as well?

Like if somebody loads a new deck and all card pictures are queried...

Rate Limits and Good Citizenship (Scryfall API Docs)

We kindly ask that you insert 50 – 100 milliseconds of delay between the requests you send to the server at api.scryfall.com. (i.e., 10 requests per second on average).

Submitting excessive requests to the server may result in a HTTP 429 Too Many Requests status code. Continuing to overload the API after this point may result in a temporary or permanent ban of your IP address.

The file origins used by the API, located at *.scryfall.io do not have these rate limits.

[ebbit1q]

they are, it's just rather hard to diagnose, also cockatrice's picture loader is pretty slow and will never do requests in parallel. I often ask for logs when I suspect this has occurred to a user in the support channel on discord yet never see a 429 code, truncating often gets in the way here though so I don't actually have that many instances where I can check this. not to mention cockatrice will fall back to using gatherer when such an error would occur.

I've got a growing suspicion that this issue is more severe than it seems, yet reproducing this is pretty hard. in many cases users with failing scryfall requests have high bandwidth network and are from the US which would presumably have the lowest latency (I suspect latency to be the primary limiter to cockatrice's single threaded image downloader)

[ebbit1q]

it'd be better if the pictureloader was ratelimited in my opinion, it'd remove such doubts, though I'm not sure how effective this would actually be. as a comparison I did a test and for my connection it takes about half a second already to download a single picture, I'd never hit the api rate limit. doing a request to the api without resolving the redirect already takes 200ms so I'm clearly limited by my connection's latency (not just my abysmal bandwidth), this is roughly what is expected for a web request to the US from europe. I'd need someone with a lower latency of like sub 20ms to break the api rate limit.

there is another concern though, someone could break the rate limit even though their latency is high by doing them in parallel, while the client never does this scryfall's ban and tracking are per IP address so if multiple users on the same IP address were to install cockatrice and load a thousand card deck at the same time the rate limit could be triggered. sharing IP addresses for many users happens often on eg company and school networks. installing a rate limit on the client wouldn't prevent this. increasing the rate limit to something substantially above their limit could mitigate this, at a certain point we'd have to consider what the largest amount of people on one IP address using cockatrice could be.

as an alternative we could have users not go through the api and give them direct links to the image cdn, this is something that's discussed on this pr for example: Cockatrice/Cockatrice#4762

[Elarnon]

I had written a Python script last week as a proof-of-concept to make requests to the API and generate up-to-date URLs without having to go through the API (to avoid rate limiting). I got sidetracked by other things and forgot to make it public — my apologies.

I have opened #209 with the script. It might be a better solution that using API URLs due to the rate limiting while still being somewhat future-proof.