Remove custom robots.txt in favor of RTD default (#4055)

* Remove custom robots.txt

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Remove link to robots.txt from conf.py

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>

* Update docs/source/conf.py

Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>

---------

Signed-off-by: Dmitry Sorokin <dmd40in@gmail.com>
Signed-off-by: Dmitry Sorokin <40151847+DimedS@users.noreply.github.com>
Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
Co-authored-by: Nok Lam Chan <nok.lam.chan@quantumblack.com>

docs/source/conf.py

@@ -214,7 +214,9 @@
 
 # html_extra_path used to define a path to robots.txt which is used by webcrawlers
 # to ignore or allow certain links.
-html_extra_path = [str(here / "robots.txt")]
+# We've decided to rely on RTD's default robots.txt, so we no longer need to specify a custom one,
+# see discussion at https://github.com/kedro-org/kedro/issues/3741
+# html_extra_path = [str(here / "robots.txt")]
 
 # Removes, from all docs, the copyright footer.
 html_show_copyright = False