Recommend a redirect strategy for docs

[dedemorton]

We will be refactoring a lot of content in the coming months (beats, cloud, etc). Right now, our strategy for handling moved content (changed URLs) is not clear.

In the past, we've requested that the website team create redirects. Here's a random example of a request.

We also have a legacy redirects page that we planned to use in the future to manage redirects, but I don't think that page is being updated, and I don't think it's actually used by the build.

Some teams maintain a Deleted pages appendix and use that to redirect users manually to a page that's moved.

We need a clear strategy going forward, and I'm not sure whether redirects are the right way to go.

According to our internal wiki (copied from there):

• Web team says it's best practice to update links on the site that reference the updated URL - search engines will see the old link if we don't update it, and it's not best practice to rely on the redirects.
• From an SEO perspective, it's best to replace URLs where possible / worth the effort since the positive impact to SEO is diminished if the crawlers see the old URL first. When crawlers hit the old URL, this will pass along some information about the new URL but is read as coming from a second-hand source. It's better overall for SEO if the crawler hits the live page first instead of coming to the live page from a redirect.
• Another consideration for the Web team is the monitoring and maintenance for redirects - after too many redirects (more than 4?), the page will not load due to too many redirects (the redirect gets stuck in a loop) - at this point, the page will stop being indexed. If we replace the URLs instead of redirecting, we're saving the team from having to monitor / remember some of these.

[alaudazzi]

@dedemorton thank you for raising this issue. This is going to impact a major doc refactoring that is currently ongoing on the Cloud ECE docs.

[dedemorton]

Also see the related issue: #1357

[debadair]

Yes, this is a major issue for restructuring content. The current process (request redirects from the web team) has led to pain because we have no insight into what redirects already exist. We've ended up with circular redirects and assorted broken-ness. The redirects file in the docs repo was Nik's attempt at bringing some order to the chaos, but was never adopted by the web team & the folks at RAW who handle the actual infra/deployment of the site.

Website redirects also only work at the page level, so if the chunking of content changes, they don't really solve the problem.

The redirects appendices feel like a very hacky solution--but it's one we can control pretty easily. On the ES side, @jrodewig and I discussed a strategy where we would clean up old entries when a new version was released, rather than keeping them around indefinitely. (Knowing that there are likely to be a number of necessary redirects between the last minor and a new major, so it's not just a matter of deleting all of them and starting over.)

One motivation for the redirect appendices was to minimize surprise cross doc links that caused chaos on release days. Now that we have the CI checks in place and have improved the process around releases, it's probably worth enforcing that if you move/remove a topic and break a link from somewhere else in the docs, you need to fix it, not just add an entry to the redirects appendix. The redirects appendix should be used to keep external links (like Google search results) from 404-ing.

Ideally, it's best to mark pages that are being removed with a noindex tag and request a reindex from Google before they disappear. I think that we could basically accomplish that by adding a noindex tag to the redirect appendices and requesting a reindex when we roll out big changes or before we clean up old entries.

@AnneB-SEO might have other insight into how to minimize the SEO disruption as we reorg the docs.

[debadair]

Also, simply keeping track of everything that has moved around is a chore. It would be really helpful to have a new & deleted anchor report generated for each PR.

[gtback]

Also, simply keeping track of everything that has moved around is a chore. It would be really helpful to have a new & deleted anchor report generated for each PR.

This is exactly the type of feedback which helps me prioritize what I'm working on!

[bmorelli25]

I hate to link to a private Slack conversation in a public repository, but I think it's necessary to illustrate how big of a problem this is: Slack 🧵. We should remember to clean up the broken links that already exist with whatever solution we choose to move forward with.

[jrodewig]

The redirects appendix should be used to keep external links (like Google search results) from 404-ing.

From an SEO perspective, a manual redirect done using a redirect appendix is inferior to a server-side 301 or 302 redirect.

Those manual redirect pages still respond with a 200 HTTP status code, which indicates to search engines that the old page is still alive. This means our new page is competing with the old (redirect) page. As older pages typically have more link juice, the redirect pages may be returned in SERPs before actual content pages.

The best case is a 301/302 that passes that link juice on to the new page. However, even a 404 would at least let the old page die. Right now, the redirect appendices are keeping old, zombie pages alive.

I also don't think the redirect appendix is the best experience for users.

I would love to abolish redirect appendices entirely, except maybe in cases where there is no good redirect. Better control and visibility of server-side redirects would be my preferred path forward.

[debadair]

Agreed. The redirect appendices are a patch for a broken process. Beyond the issue of ending up with zombie pages that never go away, the manual process simply doesn't scale for major reorganization of existing content. We need to be able to automatically detect changes that require redirects, and manage the redirects in a way that doesn't require multiple spreadsheets and teams.