OpenCrawl
A web crawler built for AI agents.
It crawls politely, shows what each AI bot can actually read, and lets your assistant query the results over MCP.
The open crawler built for agents, with a live crawl log on the home page.
One crawler from the first rule to the last report.
Drive it from anything, and send results anywhere.
The product page leads with the MCP server: a crawler your agents can drive.
Crawling as a tool call, with plugin hooks at every step.
Every URL is accounted for, from discovery to the report.
Published benchmarks across the Go, Rust and TypeScript engines.
The Problem
Site owners now have two audiences: people, and the AI crawlers and agents that read pages on their behalf. GPTBot, ClaudeBot, PerplexityBot and a dozen others each follow their own rules, and a firewall or CDN can quietly block one while letting another through.
Most crawlers were built for search engines and spreadsheets, not for answering"which of my pages can ClaudeBot actually read?"And the ones that exist are usually closed tools that decide for you what and how you can crawl.
The Approach
I designed OpenCrawl around three steps, so it works whether a person, a pipeline or an agent is driving:
- Point it at a site from the CLI, from code, or from an AI assistant. It reads robots.txt, llms.txt and sitemaps before anything else.
- Add your logic with plugin hooks before the crawl, before and after each page, and inside the renderer.
- Hand results on by streaming pages as they're captured, or letting an agent query the finished crawl over MCP.
Every page is checked against19 AI crawlers, fetchers and robots.txt tokens, from a registry that records each agent's user agent, operator, purpose and documented stance on robots rules.
The Solution
A single Go binary with a built-in MCP server. Run opencrawl audit for an HTML report, or opencrawl mcp install and ask Claude Code, Claude Desktop or Cursor about your site directly.
Key Features
Per-Agent Access
For every page and every agent: allowed, blocked by robots.txt, blocked by meta robots orX-Robots-Tag, or turned away by the server. An agent can ask "which pricing pages can't GPTBot read?" and get an answer from real crawl data.
Render Diff
Raw HTML by default, headless Chrome on demand. A side-by-side diff shows which blocks only appear with JavaScript, and the script responsible, since most AI crawlers never run it.
MCP Tools
Start, monitor and cancel crawls, query saved audits by agent, state or path, and compare two runs. Long crawls run as background jobs, so the assistant never sits waiting.
Real Agent Logs
opencrawl logs reads nginx, Apache or Cloudflare logs and checks each AI bot request against the operator's published IP ranges, so spoofed bots are reported separately instead of skewing the numbers.
Saved Runs
Every audit is stored in SQLite. Each new crawl of a host prints what changed since the last one.
Design Decisions
- Label how far to trust it.Every finding is marked exact (robots rules, raw HTML), simulated (a request sent with an agent's user agent) or heuristic (an inference). The MCP server tells the model to pass that on, and never to claim anything about training, ranking or citations.
- Polite by default.robots.txt is read first and matched per RFC 9309. At most four requests are in flight, 250ms apart, and an agent's user agent is never sent to a URL robots.txt disallows for it.
- Runs on your machine.One self-contained binary with no account and no cloud. Chrome is only needed for render diffs.
- Measure, don't guess.Before settling on Go, I built the same crawler in Go, Rust and TypeScript and benchmarked all three.
Benchmarks
Same crawl, three runtimes: a breadth-first, same-domain crawl against a local synthetic site, where the crawler itself is the bottleneck, and against a live site, averaged over 20 passes.
| Runtime | Synthetic req/s | Live req/s |
|---|---|---|
| Go | 33,450 | 285.7 |
| Rust | 32,405 | 277.8 |
| TypeScript | 14,230 | 283.8 |
Go and Rust were neck and neck, and on a real network all three land within 3% of each other. Go won on speed, a single static binary and an official MCP SDK. Raw HTML crawling also came out2,587× faster than rendering every page in Chrome, which is why rendering is opt-in.
The Site
I designed open-crawl.com around one image:a crawler finding its way through a maze.
- The hero is an infinite maze zooming toward its centre in WebGL. A build-time SVG of the same maze renders first, and stays for visitors without WebGL or who prefer reduced motion. Three.js only downloads when the animation will actually run.
- Product features are shown as working UI: live crawl stats, a drop-off funnel from discovered to delivered URLs, and a render diff with a draggable divider.
- The benchmarks page races the three runtimes through the maze using the published results.
- The invite form is styled as a terminal command and asks for just a work email.
Technologies Used
| Stack Area | Tools & Frameworks |
|---|---|
| Engine | Go, SQLite, chromedp (headless Chrome) |
| Agents | MCP Go SDK, stdio server |
| Benchmarks | Go, Rust and TypeScript crawlers; Bubble Tea TUI |
| Site | Astro, Tailwind CSS, Three.js, Storybook |
| Hosting | Cloudflare Workers + D1 for invites |
Outcome
OpenCrawl is now invite-only. I took it from an idea to a working product: the crawler and its MCP server, the benchmarks behind the choice of language, and the site that explains it. WhereSitebulb Labs checks one page in the browser, OpenCrawl covers a whole site and lets an agent ask the questions.