site-crawler

Open-source SEO monitoring. GSC + Site Crawler + Core Web Vitals in one dashboard.

Generate clean, AI-ready llms.txt files for your website or docs. Supports crawling, sitemaps, static builds, and framework-aware adapters (Next.js, Vite, Nuxt, Astro, Remix). Includes Markdown/MDX docs mode and robots.txt generator for LLM and search crawlers.

Free Chrome extension for comprehensive Shopify SEO analysis. 50+ checks, real Core Web Vitals, site crawler, Google rank tracker, competitor comparison, internal link map, and PDF reports. No API keys, no signup, no cost.

SnapShot — Web Capture API by WOLVAREX Capture websites as screenshots, recordings, and more through a simple, powerful API.

Omnipotent Master SEO & Security Auditor (Cyber Fusion Engine v4.0)

grandstage catalog data scraper

Crawl an entire website and surface every broken link — self-contained PHP + cURL, dark-themed HTML dashboard + CSV export.

Site crawler used in Ethora platform as an option to import your specific business data into your AI agent chat bot.

Crawlcat public REST API: OpenAPI contract, multi-language code examples, and docs for site SEO audits — crawl sites, export Copy for AI markdown, and integrate with Cursor, Claude, or ChatGPT via MCP.