<p align="center" <img src="src/assets/images/icon.png" alt="Scrape-LE Logo" width="96" height="96"/ </p <h1 align="center"Scrape-LE: Zero Hassle Scrapeability Checks</h1 <p align="center" <bLoad a URL in headless Chromium and see what will block your scraper — before you write it</b<br/ <iAnti-bot vendors, rate limits, robots.txt rules, login walls, console errors, screenshots</i </p
<p align="center" <a href="https://marketplace.visualstudio.com/items?itemName=nolindnaidoo.scrape-le" <img src="https://img.shields.io/badge/Install%20from-VS%20Code-blue?style=for-the-badge&logo=visualstudiocode" alt="Install from VS Code Marketplace" / </a <a href="https://open-vsx.org/extension/OffensiveEdge/scrape-le" <img src="https://img.shields.io/open-vsx/dt/OffensiveEdge/scrape-le?style=for-the-badge&label=Open%20VSX&color=blue" alt="Open VSX downloads" / </a <a href="https://www.npmjs.com/package/scrape-le-mcp" <img src="https://img.shields.io/npm/v/scrape-le-mcp?style=for-the-badge&label=MCP%20server&color=blue&logo=npm" alt="scrape-le-mcp on npm" / </a <a href="https://crates.io/crates/scrape-le" <img src="https://img.shields.io/crates/v/scrape-le?style=for-the-badge&label=Rust%20CLI&color=blue&logo=rust" alt="scrape-le on crates.io" / </a <a href="https://letools.dev/tools/scrape-le" <img src="https://img.shields.io/badge/LE%20Tools-letools.dev-blue?style=for-the-badge" alt="LE Tools" / </a </p
---
<p align="center" <img src="src/assets/images/demo.gif" alt="Scrapeability Check Demo" style="max-width: 100%; height: auto;" / </p
Useful? A star or rating is how other developers find it — ★ GitHub · ★ Open VSX · ★ Marketplace
What it does
Run Scrape-LE: Check URL Scrapeability (Ctrl+Alt+S / Cmd+Alt+S), enter a URL, and the page loads in a real headless Chromium. The report lands in the output channel: HTTP status, page title, load time, console errors, a full-page screenshot, and four detections. Works in VS Code and VS Code–based editors like Cursor and VSCodium (installable from Open VSX).
One-time setup: run Scrape-LE: Setup Browser to install Chromium (~130MB, into Playwright's browser cache).
Install
| Where | What you get | Install | |---|---|---| | VS Code | The same check, in your editor, on a keystroke | Marketplace | | Cursor, VSCodium, Windsurf | The same extension | Open VSX | | A terminal or a CI step | The same run over a whole tree, with exit codes | cargo install scrape-le · crates.io | | Any MCP agent, via Node | analyzerobotstxt over stdio | npx scrape-le-mcp · npm | | Zed | The MCP server as a context server | add it by hand (no listing yet) |
Use it from an AI agent
The same engine runs as an MCP server, so an agent can call it directly instead of you running a command.
| Editor | How | |---|---| | VS Code 1.101+ | Nothing to install — the extension registers analyzerobotstxt with agent mode | | Zed | No listing yet — add the MCP server by hand | | Claude Code | claude mcp add scrape-le -- npx -y scrape-le-mcp | | Cursor, Windsurf, anything else | point it at npx scrape-le-mcp |
analyze_robots_txt(content, path, maxResults?)Given robots.txt contents and a path, reports whether the generic (User-agent: ) rules permit crawling it, plus the crawl delay, disallowed patterns and any sitemaps.
The server takes content and returns data — it reads no files and makes no network requests of its own. Published as scrape-le-mcp on npm and as io.github.nolindnaidoo/scrape-le in the MCP registry.
<details <summary<bConfiguring it by hand</b — any host with an MCP config file</summary
Most hosts read a JSON config. Add one entry: