Make Apache block some self-identified bots.

While the proxy servers can block by user-agent for plain traffic, they
cannot for TLS connections.

Author: amigdal-mit

server/fedora/config/etc/httpd/conf/httpd.conf

@@ -110,6 +110,13 @@ UserDir disabled
 <Directory />
     AllowOverride None
     Options FollowSymLinks IncludesNoExec
+
+    # Block some (self-identifying) bots, by giving them a 403.
+    # The proxy servers should catch these (/etc/haproxy/blacklist-agent.txt),
+    # but it can only look at HTTP traffic.  This was added primarily for HTTPS
+    # traffic.
+    Require expr %{HTTP_USER_AGENT} !~ /Bytespider|Bytedance|ClaudeBot/
+
     # The new syntax wasn't added until 2.4,
     # so there's simply no way any deployed sites
     # are already using the new syntax.