Follow redirects feature

[firecodergithub]

If the server responds with redirect (301 or 302), the crawler should follow these redirects (like curl is able for example), otherwise it's missing a bunch of crawlable content

[floodpants]

It looks like this project hasn't been touched in a couple of years, so I'm not sure if it is still maintained or not. If so, I suspect this could be an easy fix by simply pulling another setting from app.settings and using that in LinkCrawler.cs where the RestClient is initiated, currently:

var restClient = new RestClient(new Uri(crawlUrl)) { FollowRedirects = false };

Setting FollowRedirects to true seems to have the desired effect in testing, though the original link still shows in the output with a 200 status rather than say

301 https://example.com
200 https://www.example.com

I could also be way off base here! Happy to do a PR if I'm not.

[hiteshbhavsar]

Can I work on this feature if it is still open?