Links to sites that have DDoS protections are marked as failed

[NicolasMassart]

More and more sites have DDoS or anti spam protections like Cloudflare DDoS Attack Protection or Godaddy that makes link checker fail on these links (server returns 502 code for instance if you are not an actual browser)

Only way for the moment to me is to exclude the sites.

But in the future we will have more and more site using these tools and link check will be only able to check internal links.
We have to figure out if we have any technical solution to this issue and see how to implement it.

All suggestions are welcome in the comments.
Thanks.

[NicolasMassart]

I've been suggested to try using a real browser in these specific failure cases and https://github.com/puppeteer/puppeteer may help. To investigate.

[NicolasMassart]

Cloudflare for instance displays a captcha (click to say you are not a robot) to solve as a security before the page is reached.

So even with a puppeteer controlled browser it would ne help solving it. I would be too easy ;)

And we can't implement all the security methods of all web providers. Here it's a CloudFlare one but it will evolve and others will create new ways to secure, it's an endless race.

[cadeef]

It might be advantageous to extend MLC enable/disable via annotation to include outcomes that mark as success/fail. I could see an annotation along the lines of markdown-link-check-status: ddos as a means by which to describe an expected behavior; it would permit all other statuses to be checked. This extends to abnormal status codes too-- it would be nice to expect a link to return say a 403 since it isn't authenticated.