Send Accept header (or add an option to send it)

[vokal-isaac]

I have encountered at least one URL that yields a 403 error when htmltest checks it, but if I add an Accept: */* header to the request, it succeeds. It might be useful to have an Accept header by default, but at least having an option to add one would help.

[wjdp]

Unsure what it should be set to, what do normal browser send?

Config option would be valuable.

[vokal-isaac]

I'm not entirely sure with normal browsers. My point of comparison was using curl, which sends Accept: */* by default. Conceptually, I think */* may make sense for checking link validity, since it should matter more that there's something there than that the something falls into a particular set of MIME types.

[vokal-isaac]

For reference, this yields a 403:

curl -Ls \
    -H "User-Agent: Go-http-client/1.1" \
    -H "Range: bytes=0-0" \
    -H "Accept:" \
    -o /dev/null \
    -w "%{http_code}\n" \
    'http://html5doctor.com/'

whereas this yields a 200:

curl -Ls \
    -H "User-Agent: Go-http-client/1.1" \
    -H "Range: bytes=0-0" \
    -H "Accept: */*" \
    -o /dev/null \
    -w "%{http_code}\n" \
    'http://html5doctor.com/'

[wjdp]

Just done a bit of reading into what browsers send. Seems to be a preference list that varies in levels of sensibility between vendors and all end with */*. I will look into making that the default here.