Is custom CloudFlare configuration needed for readthedocs.io?

[dimaqq]

Details

• Read the Docs project URL: https://ops.readthedocs.io

Expected Result

Either being able to use social previews, or clean error if previews are broken

Actual Result

Example solution: https://community.cloudflare.com/t/attention-required-message-when-sharing-link/88999/3

[humitos]

Hi. I'm not sure to understand the issue. Can you expand on what you are trying to do and what's the error you are getting?

[dimaqq]

I imagine that the fact that rtd is proxies by cloudflare is configured in rtd, something that I don’t have access to.

cloudflare, in turn shows a “I’m not a robot” page when Facebook tries to load a “social image”. Because that’s server-to-server communication, human check can never pass.

instead, rtd’s cloudflare account should be configured to allow machine/direct/non-browser access from a white list of ip addresses.

[humitos]

Thanks for the explanation. I'm pinging @ericholscher here because I think he will have more context about this.

[ericholscher]

Seems like the site is working to me?

-> curl -IL https://ops.readthedocs.io/en/latest/
HTTP/2 200 

[dimaqq]

The site is working fine, it's facebook loading the "social image" or preview that doesn't work.

[dimaqq]

You can use this tool from facebook to debug:

https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Fops.readthedocs.io%2F

[dimaqq]

Btw, this seems to affect all projects on .io, heres pypi:aapns

https://developers.facebook.com/tools/debug/?q=https%3A%2F%2Faapns.readthedocs.io%2Fen%2Flatest%2F

[dimaqq]

CloudFlare advisees to "check your security events in your Cloudflare dashboard"

ref: https://community.cloudflare.com/t/link-preview-is-not-working/643722/2

[ericholscher]

Oh... I'm guessing this is an issue from the locked down configuration we have for Facebook's AI crawler which is spamming our site: https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/ -- is there a specific user agent that they are using for these requests, we can. perhaps unblock it, but their abusive behavior is pretty awful, so might not be possible.