Skip to content

Commit

Permalink
Update robots.txt (#2632)
Browse files Browse the repository at this point in the history 
I have found that Bing/Yahoo/DuckDuckGo, Yandex and Google report crawl errors when using the default robots.txt. Specifically their bots will not crawl the the path '/' or any sub-paths. I agree that the current robots.txt should work and properly implements the specification. However it still does not work.

In my experience explicitly permitting the path '/' by adding the directive Allow: / resolves the issue.

More details can be found in a blog post about the issue here: https://www.dfoley.ie/blog/starting-with-the-indieweb
  • Loading branch information
@dsofeir @rhukster
dsofeir authored and rhukster committed Aug 18, 2019
1 parent 20b9ca5 commit ed87faa
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions robots.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ Disallow: /user/
Allow: /user/pages/
Allow: /user/themes/
Allow: /user/images/
Allow: /

1 comment on commit ed87faa

@hughbris
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reasons behind this issue made me sad :(

Reading about "indieweb" made me happy :)

WTH is with these companies. DDG: I'm disappointed!

Please sign in to comment.