Crawl4AI

What it does

Crawl4AI crawls websites and returns content in formats that models handle well: clean Markdown, or structured data that you define. It drives a real browser, so it copes with JavaScript-heavy pages.

Why it is useful

• Open source and self-hosted, so there is no per-page fee.
• Built for RAG and agent pipelines, not only for dumping HTML.

How to use it

pip install -U crawl4ai

Then run the setup step from the README to install the browser it uses.

Watch out for

• Respect robots.txt and the terms of the sites you crawl.
• Large crawls need rate limits. Start small.