Good bots white-list #27
Description
Typically, good bots, like web search spiders, behave normally and should not be blocked by WebShield. However, it still could make sense to white list them explicitly, especially to work with advanced protectors like URL access patterns.
We can port and use utils/check_googlebot_ip.py from our website repo as an utility to produce the white list for Google.
TBD: we need to intersect the white list with detector query results, so it seems we should keep in in ClickHouse for faster operation