Protect preview environment from Search Index Crawlers #295
Description
Is your feature request related to a problem? Please describe.
Preview deployments generate multiple URLs that can be found and indexed by search engines which is likely undesired.
As of now, I haven't found any hub website indexed but giving a preview URL to any checker tool it would say it is indexable.
Describe the solution you'd like
There are multiple ways to avoid this but since it's probably a common issue, could be nice to have a built-in solution and let the user enable/disable easily.
- Some solutions mention that canonical URLs pointing to the production urls could fix it (unsure).
- Adding (maybe overwriting) a robots.txt.
- Adding noindex meta tag.
X-Robots-Tag: noindexheader.
For reference:
https://community.cloudflare.com/t/pages-preview-protect-from-search-index-crawlers/384545
https://community.cloudflare.com/t/how-to-non-index-domain-its-pages/407579
I open this issue to discuss what could be the best approach.