robots.txt should be served as content-type text/plain

[t83714]

robots.txt should be served as content-type text/plain

We currently serve a robots.txt with search engine bots instructions to:

• supply sitemap entry point
• avoid certain pages (e.g. search result page) being indexed

The serving response currently carries content-type header text/html, which might see here prevent certain search bots from recognising this file.

We will also add additional instructions to prevent URLs with q query parameter from being indexed --- although some existing instructions already cover this by URL path, the additional rule might help to secure this logic further.

[t83714]

closed via PR: #3561