robots.txt missing Disallow

[NJAldwin]

Looking more closely at the robots.txt standard, it states that "At least one Disallow field needs to be present in a record."

The current robots.txt creates a record with User-agent: * but omits the Disallow field. As far as I can understand, the rationale behind this is either:

• Disallow nothing by default, in which case an empty Disallow: line should be added to make the file valid.
• Encourage the boilerplate user to implement his or her own robots.txt, in which case I'd think that some text pointing out that the boilerplate robots.txt is invalid would be useful.

Thoughts?

[alrra]

the boilerplate robots.txt is invalid

@NJAldwin the robots.txt "standard" is really outdated (see: "The /robots.txt standard is not actively developed"). Different crawlers (if they even decide to respect the /robots.txt file at all) behave differently, and even have nonstandard extensions that are not valid according to some "standards", but are valid according to others. That said, validity is a relative thing, different from what is usually in the real world!

I've tested using several validators, and some don't complain about the lack of Disallow:, while others do. I also tested using the Google and Yandex tools and they don't seem to complain, although their documentations (see: 1, 2, 3) suggest using Disallow:.