Crawl4AI v0.6.0 Release Notes

We're excited to announce the release of Crawl4AI v0.6.0, our biggest and most feature-rich update yet. This version introduces major architectural upgrades, brand-new capabilities for geo-aware crawling, high-efficiency scraping, and real-time streaming support for scalable deployments.

Highlights

1. World-Aware Crawlers

Crawl as if you’re anywhere in the world. With v0.6.0, each crawl can simulate: - Specific GPS coordinates - Browser locale - Timezone

Example: 

CrawlerRunConfig(
    url="https://browserleaks.com/geo",
    locale="en-US",
    timezone_id="America/Los_Angeles",
    geolocation=GeolocationConfig(
        latitude=34.0522,
        longitude=-118.2437,
        accuracy=10.0
    )
)
Great for accessing region-specific content or testing global behavior.

2. Native Table Extraction

Extract HTML tables directly into usable formats like Pandas DataFrames or CSV with zero parsing hassle. All table data is available under result.media["tables"].

Example: 

raw_df = pd.DataFrame(
    result.media["tables"][0]["rows"],
    columns=result.media["tables"][0]["headers"]
)
This makes it ideal for scraping financial data, pricing pages, or anything tabular.

3. Browser Pooling & Pre-Warming

We've overhauled browser management. Now, multiple browser instances can be pooled and pages pre-warmed for ultra-fast launches: - Reduces cold-start latency - Lowers memory spikes - Enhances parallel crawling stability

This powers the new Docker Playground experience and streamlines heavy-load crawling.

4. Traffic & Snapshot Capture

Need full visibility? You can now capture: - Full network traffic logs - Console output - MHTML page snapshots for post-crawl audits and debugging

No more guesswork on what happened during your crawl.

5. MCP API and Streaming Support

We’re exposing MCP socket and SSE endpoints, allowing: - Live streaming of crawl results - Real-time integration with agents or frontends - A new Playground UI for interactive crawling

This is a major step towards making Crawl4AI real-time ready.

6. Stress-Test Framework

Want to test performance under heavy load? v0.6.0 includes a new memory stress-test suite that supports 1,000+ URL workloads. Ideal for: - Load testing - Performance benchmarking - Validating memory efficiency

Core Improvements

  • Robots.txt compliance
  • Proxy rotation support
  • Improved URL normalization and session reuse
  • Shared data across crawler hooks
  • New page routing logic

Breaking Changes & Deprecations

  • Legacy crawl4ai/browser/* modules are removed. Update imports accordingly.
  • AsyncPlaywrightCrawlerStrategy.get_page now uses a new function signature.
  • Deprecated markdown generator aliases now point to DefaultMarkdownGenerator with warning.

Miscellaneous Updates

  • FastAPI validators replaced custom validation logic
  • Docker build now based on a Chromium layer
  • Repo-wide cleanup: ~36,000 insertions, ~5,000 deletions

New Examples Included

  • Geo-location crawling
  • Network + console log capture
  • Docker MCP API usage
  • Markdown selector usage
  • Crypto project data extraction

Watch the Release Video

Want a visual walkthrough of all these updates? Watch the video: 🔗 https://youtu.be/9x7nVcjOZks

If you're new to Crawl4AI, start here: 🔗 https://www.youtube.com/watch?v=xo3qK6Hg9AA&t=15s

Join the Community

We’ve just opened up our Discord for the public. Join us to: - Ask questions - Share your projects - Get help or contribute

💬 https://discord.gg/wpYFACrHR4

Install or Upgrade

pip install -U crawl4ai

Live long and import crawl4ai. 🖖