Free Local AEO & SEO Spider and a Markdown content extractor
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Crawler.sh converts any website into RAG-ready Markdown with automatic content extraction, word counts, author bylines, and excerpts. Pro plan enables bulk Content Archive export as ZIP with individual Markdown files per page, eliminating manual content preparation for LLM fine-tuning and agent context.
Crawler.sh runs locally on your machine with no cloud bill or API quota. Pro plan ($99/year) allows up to 10,000 pages per crawl with concurrent requests, automatic retry, and robots.txt compliance. Built in Rust for speed; typical crawls complete in seconds.
Crawler.sh runs 24 automated SEO checks (missing titles, duplicate meta descriptions, noindex directives, thin content, broken links, content freshness signals, etc.) across every crawled page in a single pass. Results export as CSV or TXT for team review and batch remediation.
Crawler.sh automatically generates W3C-compliant Sitemap XML from live crawls, keeping sitemaps accurate without manual maintenance. Available in both free and Pro tiers; export happens locally with no external dependencies.
Crawler.sh can be run regularly to detect broken links, missing pages, and status code changes across your site. Results export as JSON or CSV for integration into monitoring dashboards or CI/CD pipelines; runs entirely locally for privacy and speed.