The Robots.txt template by Jonas Jacek, has been moved to ditig.com.
This repository will be closed shortly.
This repository contains 2 robots.txt file templates to help webmasters keep unwanted web robots (e.g. scraper bots, people search engines, seo tools, marketing tools, etc.) away from their websites but allow legitimate robots (e.g. search engine crawlers).
To be legitimate and get listed, robots must fully obey the Robots Exclusion Standard. The robots.txt file templates contain a white list. Unlisted robots (User-agents) are, by the conventions of the Robots Exclusion Standard, not allowed to access.
The robots.txt template files contain an alphabetically ordered white list of legitimate web robots. In the commented version, each bot is shortly described in a comment above the (list of) user-agent(s). Uncomment or delete bots (User-agents) you do not wish to allow to access your website.
There are two robots.txt file versions:
- The regular file (with comments)
/robots.txt - The minified file (no comments)
/robots.min.txt
If you use the minified version, do not forget to rename it to robots.txt to be effective.