What is Crawlbase MCP?

Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.

It handles the complexity of scraping for you:

JavaScript rendering for modern web apps
Proxy rotation & anti-bot evasion
Structured outputs (HTML, Markdown, screenshots)

How It Works

Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.

Setup & Integration

Claude Desktop

Open Claude Desktop → File → Settings → Developer → Edit Config
Add to claude_desktop_config.json:
Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Claude Code

Add to your claude.json configuration:

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Cursor IDE

Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
Add to mcp.json:
Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

Windsurf IDE

Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
Add to mcp_config.json:
Replace your_token_here and your_js_token_here with the tokens from your dashboard.

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

🔑 Get your free tokens at Crawlbase ↗️.

Usage

Once configured, use these commands inside Claude, Cursor, or Windsurf:

crawl → Fetch raw HTML
crawl_markdown → Extract clean Markdown
crawl_screenshot → Capture screenshots

Example prompts:

“Crawl Hacker News and return top stories in markdown.”
“Take a screenshot of TechCrunch homepage.”
“Fetch Tesla investor relations page as HTML.”

Use Cases

Market research → Pull live data from competitors, news, and reports
E-commerce monitoring → Track products, reviews, and prices in real time
News & finance feeds → Keep AI agents up-to-date with live events
Autonomous AI agents → Give them vision to act on fresh web data

Resources & Next Steps

Looking to supercharge your AI agents with live web data? Get started here:

✍️ Learn More – See how MCP powers AI agents with real-time web data ↗️
🌐 Crawlbase Website – Get free tokens & start crawling today ↗️