Use <slug>.readthedocs.io for suggested custom domain endpoint

[agjohnson]

Our custom domain directions tell people to CNAME to readthedocs.io. One idea that might be nice is to have them CNAME to <slug>.readthedocs.io. This would let us configure DNS for a single domain rather than making large changes.

In order for this to work, we need to tweak our Cloudflare settings to have the "orange cloud" for *.readthedocs.io. Projects that already proxy through cloudflare and have their DNS set to <slug>.readthedocs.io will have issues that can be solved with the directions here. I expect there's no more than a handful of these.

[davidfischer]

There is a ticket with solutions for users experiencing issues with Cloudflare #4395.

[davidfischer]

This is a bit of a tricky situation since it's not really easy for us to tell which domains will have problems once we make this change. We don't really know which projects have their custom domains proxying through Cloudflare. Perhaps the right solution here is to make the switch and have a test script where we can uncover these issues quickly and then either put workarounds in place or contact the project authors proactively.

[agjohnson]

@davidfischer I forgot where we landed on this. Does the above point still stand? Should we script up something that resolves the custom project domains to detect cloudflare proxying first?

[davidfischer]

I think we should script something and try to detect it. There's probably not too many projects that it will affect. I'd estimate ~5-10 but it's worth figuring it out first.

[agjohnson]

Roger. In terms of priorities, this is pretty low and we're a few versions out from making this change. Perhaps this is a v3 change in fact.

[agjohnson]

@rtfd/core, how do we feel about making this a v3.0 release item EOY?

[davidfischer]

I'm +1.

[davidfischer]

I think we should script something and try to detect it. There's probably not too many projects that it will affect. I'd estimate ~5-10 but it's worth figuring it out first.

There's actually more like 40. I ran a script over our ~3k domains which checked if the custom domain pointed to a CNAME matching *.readthedocs.io (but didn't match "readthedocs.io" exactly). For each matching domain, I checked if the root domain had its nameservers on Cloudflare.

Interestingly, I also found ~7 projects that were pointed to cloudflare-to-cloudflare.readthedocs.io -- meaning they were going to use their own SSL cert per the docs -- but they hadn't setup an SSL cert. I don't know why they would do this but I emailed a couple of them to try to figure this out.

[ericholscher]

I do wonder if we should CNAME to <slug>.cname.readthedocs.org or similar. That way it will give us even more flexibility over DNS, without having to change how we do DNS for *.readthedocs.io itself. I don't know if that will break the Cloudflare setup, but I think being explicit here for a magic DNS/SSL CNAME is better than trying to piggyback on top of the existing DNS entry used in production.

[davidfischer]

From a Cloudflare perspective, it only has to resolve to a domain we control. We could probably do something like <slug>.cname.readthedocs.io. I think it's best to keep this off .org.

[ericholscher]

I think it's best to keep this off .org.

Makes sense, but curious what the reasons are. Just to keep it consistent with the domain we're hosting docs on?

[davidfischer]

It is conceivable that in the future .org and .io are not even served by the same servers.

[davidfischer]

I added the relevant DNS entry for *.cname.readthedocs.io. If we want that to be our recommendation, we don't need to notify anyone in advance and we can just change our docs. If we are all on board with that, I'll just submit a PR updating them.

[davidfischer]

I should also mention I verified it by adding a custom domain that resolves to slug.cname.readthedocs.io and the certificate issued successfully.

[davidfischer]

Now that I think about this a little further, there is at least one drawback to using slug.cname.readthedocs.io instead of slug.readthedocs.io and that is that ~250 domains already point to some variation of *.readthedocs.io. By changing our DNS to support that directly, we could issue SSL certs for those domains without requiring 250 users to update DNS. However, the drawback to supporting slug.readthedocs.io is the problems we will have with the ~40 domains.

[ericholscher]

One of my goal is being able to seperate the operational aspects of DNS around *.readthedocs.io from the actual user DNS entries. It gives us a bit more flexibility around doing something custom for a user, particularly around CDN's for example, without effecting the origin server.

So for example:

pip.cname.readthedocs.io -> Fastly
Fastly -> pip.readthedocs.io
pip.readthedocs.io -> Azure load balancer

This would be really convoluted if we wanted to repoint pip.readthedocs.io to fastly, and then have an alternative origin server. We could always do slug.origin.readthedocs.io as a backing origin for a CDN, but I think having the CNAME DNS record explicit gives us more flexibility.

[ericholscher]

We should consider this again now that we're fully on CloudFlare for DNS. Do we want to point users to a slug-specific domain again? I see benefits of each approach:

• Having users point at slug.readthedocs.io got us into this mess, where we can't control the actual serving of docs at *.readthedocs.io without effecting custom domains. I'm -1 on this approach
• Pointing users at readthedocs.io is useful, because it lets us redirect all custom domains at one time, but doesn't let us resolve issues on a case-by-case basis, as we did with our latest migration. This gives us less flexibility in the future
• I think a project-specific DNS entry is best, giving us flexibility on what to do, but it shouldn't be the same as our doc-serving DNS entry. I think something like slug.cname.readthedocs.io as I mentioned is the best. It lets us customize DNS on a per-project level, but also leaves us free to adjust our docserving at slug.readthedocs.io as needed.

[humitos]

@ericholscher @stsewd is this issue still relevant after all the migration you have done regarding custom domains?

[stsewd]

On .com we are using a hash for the CNAME to protect users from subdomain hijacking.