Enhancement proposal: Better way to not accept redirecting links

[Ravlen]

We're using this great tool to look for redirecting links in our docs, because we want to update them to the latest destinations. it's also a good way to prevent broken links in the long run, as redirecting links might suddenly become dead links.

The way we do this is with max_redirects = 0, then we get errors in the list like:

• [301] https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.4 | Rejected status code (this depends on your "accept" configuration): Moved Permanently

This does work well for us, as it generates a list of links to review and update. The downside is that this does not show the final destination. If I set max_redirects = 5 and also set it to verbose with -v, I can get the information I seek:

• [200] https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.4 | Redirect: Followed 1 redirect resolving to the final status of: OK. Redirects: https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.4 --> https://learn.microsoft.com/en-us/powershell/scripting/install/install-powershell-on-windows?view=powershell-7.4

So my proposal is to find a way to improve how this works to find redirecting links.

• How about a setting like always_follow_redirects = true/false so that it will always follow redirects even when not in the accept array (would still use max_redirects to prevent loops). When true it displays the final destination in the output?
• Alternatively, how about a reject array, then we could add [301, 308] or similar, which would mark redirects as rejected without needing to drop max_redirects to 0? This would rely on it knowing to output the final destination even when in the reject array.

[thomas-zahner]

Thanks, glad to hear that you like lychee. This is a good question indeed. We also try to encourage users to update redirecting links with the destination links since very recently, though this change hasn't been released yet.

So your use-case does make sense and might be adapted by more users over time.

• How about a setting like always_follow_redirects = true/false so that it will always follow redirects even when not in the accept array (would still use max_redirects to prevent loops). When true it displays the final destination in the output?

This is quite contradictory. This would only have an effect when max_redirects is reached and thus have the potential to infinitely follow redirect loops. Also I feel like such an option would be really confusing. (to myself and other users)

• Alternatively, how about a reject array, then we could add [301, 308] or similar, which would mark redirects as rejected without needing to drop max_redirects to 0? This would rely on it knowing to output the final destination even when in the reject array.

Currently, status codes in between redirects have no effect on whether the result is accepted or not. We only use the resulting end status code to decide how the result is categorised. So rejecting 3XX only has an effect if max_redirects < actual redirects.

I think what we are actually looking for is a way to reject redirects without changing the behaviour of max_redirects. We could think of adding a simple flag to achieve this, e.g. reject-redirects.

At the moment we would have to manually handle the output of lychee. For example using JSON for a more robust solution

echo http://example.com | lychee - -f json | jq '.redirects' | awk '{exit ($1 > 0)}'

But you could of course also use the non-JSON output. It would be even so simple as to look for the word Redirect in stdout (progress is logged to stderr) and fail if the word is contained. But of course this is not very user-friendly which is why it would make sense to add a flag to reject redirects.

[Ravlen]

@thomas-zahner Yes, it's certainly doable right now, for example if you just go fully verbose, you can get the information you need, for example in our docs:

[200] https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html |
  Redirect: Followed 1 redirect resolving to the final status of: OK.
  Redirects: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html -->
    https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html

But going verbose, for example in a CI/CD job, has drawbacks because a project like ours has over 20k links which rapidly fills up the log when verbose mode is turned on and can go over the max log length. We could do it in JSON and save it as a large report, but again it's quite unwieldy.

I like the idea of a reject-redirects option. Because basically all I think we need is to have a way to get all the information in that output in my example above, but just have it go "red" rather than "Green", so it's output in non-verbose mode. Then we can use that redirect information in automation to point to the new locations.

Alternatively (and i know this isn't the purpose of Lychee), but if there was an --update-redirects auto-fix option, I would use it in production tomorrow 😅

[thomas-zahner]

Thanks for your insights, this reinforces the argument in favour of adding a dedicated flag.

Alternatively (and i know this isn't the purpose of Lychee), but if there was an --update-redirects auto-fix option, I would use it in production tomorrow 😅

It should be possible to script this behaviour by now. But it also has the drawback of having to use JSON output again. Also see: #1433 (comment)

By now I could actually imagine adding this feature to lychee, given that we make sure that the changes are applied to a directory which is tracked by git and given that the git directory is in a clean state. You can create an issue for it if you would like to see this feature. This feature would probably be blocked by #1304.

[Ravlen]

@thomas-zahner Looking at the other issues, perhaps naming it --suggest-redirects would make sense?

Edit: And perhaps --update-redirects would be the other feature request (#2044)