Suggestion: embed Vitess documentation inside this repo

[shlomi-noach]

Today, the vitess code and the vitess documentation are on different repos. This means one can make code changes, create a PR, merge that PR, and then consider the case for making a documentation PR.

I'm suggesting to import the documentation into this repo. This will be a cultural/flow change: much like we're expected to add tests to validate our changes, we will also update the documentation in the same PR that changes the code, and we will culturally expect documentation updates when reviewing a PR.

We can otherwise still keep that other documentation repo, https://github.com/vitessio/website as is: we can choose to use CI/Actions cycle to push documentation updates from this repo to the website repo upon PR merge, or just a few times a day, etc.

Thoughts?

[harshit-gangal]

Only thing is we would need documentation versions. We should not update the website from master. It should be updated from release branches only.

[deepthi]

docs used to live in vitessio/vitess/doc and were moved to website at some point. @sougou might know why.
There's a request to provide versioned docs. so we could have LATEST, 7.0, 8.0 etc.
+1 on requiring documentation with PRs.

[shlomi-noach]

Thanks for the info. Noting that we can work with versioned documents within vitess repo, coupled with the vitess releases, and push them accordingly to website repo, from where they can deploy.

[sougou]

CNCF recommended that we use this approach because it's how kubernetes did it and it worked well for them. They also did the move for us, and continue to contribute to structural maintenance like translation into chinese, etc.

But 💯 on updating documentation as part of making changes. So, we can look at the CI approach to push a mirror, which may give us the best of both worlds.

[raunaqgupta]

Chipping in for what I was told earlier as to why a separate repo was created (part of the reason) was to discourage documentation pages in github (markdown files) to be indexed on top of the built pages available on the website while searching on Google. Also as part of this reasoning, the vitess/website main branch is called prod and not master since by default, Github indexing on Google is applied to the master branch content only.

Although this can still be enabled (I think) by renaming the main branch in this repo to something other than master

[shlomi-noach]

to discourage documentation pages in github (markdown files) to be indexed on top of the built pages available on the website while searching on Google

Ahhh, that's interesting indeed. Looking at GitHub's robots.txt it seems like there's no more mention of master. However, if we place our docs under, say, wiki/, than they should no be indexed, thanks to:

Disallow: */wiki/

[shlomi-noach]

Actually, GitHub's robots.txt now disallows any blobs. Google no longer indexes anything in a GitHub repo that is not on main repo page. I double checked, and all my orchestrator documentation, for example, that I know used to be indexed by Google, is no longer indexed.

This means we can put the docs wherever we like.