robots.txt

The software ships with a stock robots.txt file. It is very basic, and more advanced blog administrators may want to edit it. For those who don't know, the robots.txt file exists to tell webpage scraper bots what pages they should and shouldn't visit. I've set it up to allow them to visit all pages by default, and disallow them from visiting the login and register pages as those pages don't need to be indexed by search engines. For context these scraper bots visit websites and gather information for search engines. At least the legitimate ones do.

It's important to note that the robots.txt file will not work if the software is installed under a subdirectory. E.g. http://localhost/blog/. In this case, a robots.txt file would need to be placed at http://localhost/robots.txt and the rules pertaining to AtomicBlog would have to be modified and added to it. E.g. Disallow: /blog/login/ instead of Disallow: /login/.