External checking of links to large files/resources

[hellt]

Hi @wjdp , thank you very much for this awesome project. It is a life saver.

I am curious to know how the checks to external links are handled? Once in a while I have external_timeout issues for links that point to an application/pdf content (quite large).
This makes me wonder if you check the availability of a linked resource by using HEAD method or getting the entire document with GET?

If it is the latter, I wonder if it makes sense to allow for a lightweight HEAD-based testing?

[wjdp]

Hey @hellt, thanks!

I tried to mitigate this by sending Range: bytes=0-0. This is still a GET but some servers will respond with a 200 and usual either nothing or a single byte of the file. I seem to remember that worked ok and is handled by most servers but can't remember specifics.

(See

htmltest/htmltest/options.go

Line 118 in 3c05eff

"Range": "bytes=0-0", // If server supports prevents body being sent

)

I think the reason for doing this rather than doing a HEAD was to force the external server to do a little bit more than a GET would (a bit fuzzy I know). Though I think adding support for HEAD and actually testing this is a very good idea.

[hellt]

It seems that in my case (and to put specific context here I talk about a link like this one) the server either doesn't support bytes=0-0 or, if it supports it, it genuinely has intermittent issues with responding in 15s interval.

Since the links I provide are 99% in markdown, which later are generated to html, it seems I can't put a flag in a markdown to skip verification for a specific link?
So I suppose my best remediation approach would be to create a wildcard for a domain from which I link these documentation articles. Unfortunately with this I loose any checks against these links...
With HEAD support I'd have had at least a guarantee that "something" sits behind the link.