add: link checker job

[ArthurFlag]

No description provided.

[cloudflare-workers-and-pages]

Deploying aiven-docs with    Cloudflare Pages

Latest commit:	be1c838
Status:	⚡️  Build in progress...

View logs

[harshini-rangaswamy]

LGTM

[rriski]

Running bash scripts/check_links.sh locally I get lots of hits to other aiven.io subroutes, e.g. /developer, /cluster, /privacy. We should probably only check /docs from this repo?

�[33mhttps://aiven.io/developer/valkey�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/developer/flink�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/developer/alloydb-omni�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/resources?contentType=podcast�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/developer/postgresql�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/events/aws-immersion-day�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/solutions/aiven-for-ai�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/recruitment-and-candidate-sourcing-privacy-policy�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/blog/category/leader-insight�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io
�[33mhttps://aiven.io/blog/category/announcements�[0m
	�[31mtoo many redirections�[0m	https://twitter.com/aiven_io

Negative lookahead regex seems to do the trick:

#!/bin/bash

excludeList=(
	"aiven.io/?!docs"
	"aiven.io/docs/assets"
	"aiven.io/docs/images"
)

rateLimit="--rate-limit=30"
limitConnections="--max-connections=9"
bufferSize="--buffer-size=8192"
timeout="--timeout=3600" # 1 hour
address="https://aiven.io/docs/"
redirectionLimit="--max-redirections=5"
acceptedResponses="--accepted-status-codes=200..404"

muffet ${bufferSize} \
	   ${timeout} \
	   ${excludeList[@]/#/--exclude } \
	   ${limitConnections} \
	   ${rateLimit} \
	   ${redirectionLimit} \
	   ${acceptedResponses} \
	   "--color=always" \
	   ${address};

This should also makes the job quicker, which means we could probably run it more often (maybe once a day?) and the timeout could be reduced from 11hours to 1 hour perhaps?