Request: Rate Limit

[pBrambi]

I love your CLI tool! I am truly missing just one thing - rate liming.

I use --concurrency 1 and even than, on a larger website, I get rate limited (our websites) usually after 10-15 seconds.

Some of them are on shared hosting so I have no way to whitelist/ignore my IP.

**It would be lovely to have an option to set "pause" inbetween the requests (in ms).

Ex.: --pause 500ms**

Thank you for this wonderful tool!

[JustinBeckwith]

So this is a really tough one - ultimately a lot of websites are going to be very aggressive about this kind of thing. One of the ways I try to get around this is properly supporting 429 responses. For the websites where you're seeing this problem - do you know what status code they're returning? What's the behavior like when you get rate limited? An example site would be incredibly helpful!

[pBrambi]

Hello,

as for my shared hosting provider, it returns:

403 Forbidden

They use FailToBan I think, which rate limits me for about 60 seconds if requests in a set timeframe exceed some set ammount of requests.

After 60 seconds about, I get 200 Ok again right away - and the timeframe starts again.

It does not return 429 Too many requests before - it goes from 200 Ok to straight 403 Forbidden.