Plugins CSS and JS excluded from robots.txt

[gmpreussner]

Some search engine crawlers, such as Google throw low severity warnings, because plug-in .css and .js files are excluded from robots.txt. I added the following to resolve the warnings:

Allow: /user/plugins/*.css$
Allow: /user/plugins/*.js$

[w00fz]

This would probably belong best in the learn documentation. I don't think it should be Grav Core providing this very specific and opinionated behavior.
It really is a theme developer choice very much driven by the context of the site.

Do you think the best behavior in general is to let google crawl CSS and JS?

[gmpreussner]

I don't have any strong opinions about this, and I haven't used Grav long enough to comment on the philosophical choices. That being said, I only use official Team Grav plug-ins at the moment, some of which ship with Grav out of the box. As a user, I would expect there not to be any issues. I think it should just work out of the box (like it does for pretty much everything else). I could see this being a user configurable option in the Admin panel for those who want to hide CSS and JS from web crawlers for whatever reason.

[w00fz]

What are the warnings you are seeing? Also how are you checking these?

[gmpreussner]

In my particular case, search engines fail to fetch the following files:

/user/plugins/form/assets/form-styles.css
/user/plugins/highlight/css/github.css
/user/plugins/login/css/login.css
/user/plugins/highlight/js/highlight.pack.js

Regardless of whether or not they should be trying to fetch these in the first place, it seems to me a bit of an inconsistency in Grav, since CSS and JS files in Themes are not excluded by default.

[w00fz]

That's fair @gmpreussner , I personally wasn't even aware of these warnings for assets.

We'll look into it, thanks!

[gmpreussner]

Awesome, thanks!

[mahagr]

Just to add, there are also compiled css files in user/data/gantry5 which cause similar issues with Google:

gantry/gantry5#2020

[mahagr]

See also #3067

[rhukster]

Fixed by bc22c8d