🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update

November 14, 2025 • 10 min read

---

Today I'm releasing Crawl4AI v0.7.7—the Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.

🎯 What's New at a Glance

• 📊 Real-time Monitoring Dashboard: Interactive web UI with live system metrics and browser pool status
• 🔌 Comprehensive Monitor API: Complete REST API for programmatic access to all monitoring data
• ⚡ WebSocket Streaming: Real-time updates every 2 seconds for custom dashboards
• 🎮 Control Actions: Manual browser management (kill, restart, cleanup)
• 🔥 Smart Browser Pool: 3-tier architecture (permanent/hot/cold) with automatic promotion
• 🧹 Janitor Cleanup System: Automatic resource management with event logging
• 📈 Production Metrics: 6 critical metrics for operational excellence
• 🏭 Integration Ready: Prometheus, alerting, and log aggregation examples
• 🐛 Critical Bug Fixes: Async LLM extraction, DFS crawling, viewport config, and more

📊 Real-time Monitoring Dashboard: Complete Visibility

The Problem: Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the container—memory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.

My Solution: I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.

The Self-Hosting Value Proposition

Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:

• 🔒 Data Privacy: Your data never leaves your infrastructure
• 💰 Cost Control: No per-request pricing or rate limits
• 🎯 Full Customization: Complete control over configurations and strategies
• 📊 Complete Transparency: Real-time visibility into every aspect
• ⚡ Performance: Direct access without network overhead
• 🛡️ Enterprise Security: Keep workflows behind your firewall

Interactive Monitoring Dashboard

Access the dashboard at http://localhost:11235/dashboard to see:

• System Health Overview: CPU, memory, network, and uptime in real-time
• Live Request Tracking: Active and completed requests with full details
• Browser Pool Management: Interactive table with permanent/hot/cold browsers
• Janitor Events Log: Automatic cleanup activities
• Error Monitoring: Full context error logs

The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.

🔌 Monitor API: Programmatic Access

The Problem: Monitoring dashboards are great for humans, but automation and integration require programmatic access.

My Solution: A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.

System Health Endpoint

import httpx
import asyncio

async def monitor_system_health():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/health")
        health = response.json()

        print(f"Container Metrics:")
        print(f"  CPU: {health['container']['cpu_percent']:.1f}%")
        print(f"  Memory: {health['container']['memory_percent']:.1f}%")
        print(f"  Uptime: {health['container']['uptime_seconds']}s")

        print(f"\nBrowser Pool:")
        print(f"  Permanent: {health['pool']['permanent']['active']} active")
        print(f"  Hot Pool: {health['pool']['hot']['count']} browsers")
        print(f"  Cold Pool: {health['pool']['cold']['count']} browsers")

        print(f"\nStatistics:")
        print(f"  Total Requests: {health['stats']['total_requests']}")
        print(f"  Success Rate: {health['stats']['success_rate_percent']:.1f}%")
        print(f"  Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")

asyncio.run(monitor_system_health())

Request Tracking

async def track_requests():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/requests")
        requests_data = response.json()

        print(f"Active Requests: {len(requests_data['active'])}")
        print(f"Completed Requests: {len(requests_data['completed'])}")

        # See details of recent requests
        for req in requests_data['completed'][:5]:
            status_icon = "✅" if req['success'] else "❌"
            print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")

Browser Pool Management

async def monitor_browser_pool():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/browsers")
        browsers = response.json()

        print(f"Pool Summary:")
        print(f"  Total Browsers: {browsers['summary']['total_count']}")
        print(f"  Total Memory: {browsers['summary']['total_memory_mb']} MB")
        print(f"  Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")

        # List all browsers
        for browser in browsers['permanent']:
            print(f"🔥 Permanent: {browser['browser_id'][:8]}... | "
                  f"Requests: {browser['request_count']} | "
                  f"Memory: {browser['memory_mb']:.0f} MB")

Endpoint Performance Statistics

async def get_endpoint_stats():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/endpoints/stats")
        stats = response.json()

        print("Endpoint Analytics:")
        for endpoint, data in stats.items():
            print(f"  {endpoint}:")
            print(f"    Requests: {data['count']}")
            print(f"    Avg Latency: {data['avg_latency_ms']:.0f}ms")
            print(f"    Success Rate: {data['success_rate_percent']:.1f}%")

Complete API Reference

The Monitor API includes these endpoints:

• GET /monitor/health - System health with pool statistics
• GET /monitor/requests - Active and completed request tracking
• GET /monitor/browsers - Browser pool details and efficiency
• GET /monitor/endpoints/stats - Per-endpoint performance analytics
• GET /monitor/timeline?minutes=5 - Time-series data for charts
• GET /monitor/logs/janitor?limit=10 - Cleanup activity logs
• GET /monitor/logs/errors?limit=10 - Error logs with context
• POST /monitor/actions/cleanup - Force immediate cleanup
• POST /monitor/actions/kill_browser - Kill specific browser
• POST /monitor/actions/restart_browser - Restart browser
• POST /monitor/stats/reset - Reset accumulated statistics

⚡ WebSocket Streaming: Real-time Updates

The Problem: Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.

My Solution: WebSocket streaming with 2-second update intervals for building custom real-time dashboards.

WebSocket Integration Example

import websockets
import json
import asyncio

async def monitor_realtime():
    uri = "ws://localhost:11235/monitor/ws"

    async with websockets.connect(uri) as websocket:
        print("Connected to real-time monitoring stream")

        while True:
            # Receive update every 2 seconds
            data = await websocket.recv()
            update = json.loads(data)

            # Access all monitoring data
            print(f"\n--- Update at {update['timestamp']} ---")
            print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
            print(f"Active Requests: {len(update['requests']['active'])}")
            print(f"Total Browsers: {update['browsers']['summary']['total_count']}")

            if update['errors']:
                print(f"⚠️  Recent Errors: {len(update['errors'])}")

asyncio.run(monitor_realtime())

Expected Real-World Impact: - Custom Dashboards: Build tailored monitoring UIs for your team - Real-time Alerting: Trigger alerts instantly when metrics exceed thresholds - Integration: Feed live data into monitoring tools like Grafana - Automation: React to events in real-time without polling

🔥 Smart Browser Pool: 3-Tier Architecture

The Problem: Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.

My Solution: A smart 3-tier browser pool that automatically adapts to usage patterns.

How It Works

import httpx

async def demonstrate_browser_pool():
    async with httpx.AsyncClient() as client:
        # Request 1-3: Default config → Uses permanent browser
        print("Phase 1: Using permanent browser")
        for i in range(3):
            await client.post(
                "http://localhost:11235/crawl",
                json={"urls": [f"https://httpbin.org/html?req={i}"]}
            )
            print(f"  Request {i+1}: Reused permanent browser")

        # Request 4-6: Custom viewport → Cold pool (first use)
        print("\nPhase 2: Custom config creates cold pool browser")
        viewport_config = {"viewport": {"width": 1280, "height": 720}}
        for i in range(4):
            await client.post(
                "http://localhost:11235/crawl",
                json={
                    "urls": [f"https://httpbin.org/json?v={i}"],
                    "browser_config": viewport_config
                }
            )
            if i < 2:
                print(f"  Request {i+1}: Cold pool browser")
            else:
                print(f"  Request {i+1}: Promoted to hot pool! (after 3 uses)")

        # Check pool status
        response = await client.get("http://localhost:11235/monitor/browsers")
        browsers = response.json()

        print(f"\nPool Status:")
        print(f"  Permanent: {len(browsers['permanent'])} (always active)")
        print(f"  Hot: {len(browsers['hot'])} (frequently used configs)")
        print(f"  Cold: {len(browsers['cold'])} (on-demand)")
        print(f"  Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")

asyncio.run(demonstrate_browser_pool())

Pool Tiers:

• 🔥 Permanent Browser: Always-on, default configuration, instant response
• ♨️ Hot Pool: Browsers promoted after 3+ uses, kept warm for quick access
• ❄️ Cold Pool: On-demand browsers for variant configs, cleaned up when idle

Expected Real-World Impact: - Memory Efficiency: 10x reduction in memory usage vs creating browsers per request - Performance: Instant access to frequently-used configurations - Automatic Optimization: Pool adapts to your usage patterns - Resource Management: Janitor automatically cleans up idle browsers

🧹 Janitor System: Automatic Cleanup

The Problem: Long-running crawlers accumulate idle browsers and consume memory over time.

My Solution: An automatic janitor system that monitors and cleans up idle resources.

async def monitor_janitor_activity():
    async with httpx.AsyncClient() as client:
        response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
        logs = response.json()

        print("Recent Cleanup Activities:")
        for log in logs:
            print(f"  {log['timestamp']}: {log['message']}")

# Example output:
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)

🎮 Control Actions: Manual Management

The Problem: Sometimes you need to manually intervene—kill a stuck browser, force cleanup, or restart resources.

My Solution: Manual control actions via the API for operational troubleshooting.

Force Cleanup

async def force_cleanup():
    async with httpx.AsyncClient() as client:
        response = await client.post("http://localhost:11235/monitor/actions/cleanup")
        result = response.json()

        print(f"Cleanup completed:")
        print(f"  Browsers cleaned: {result.get('cleaned_count', 0)}")
        print(f"  Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")

Kill Specific Browser

async def kill_stuck_browser(browser_id: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:11235/monitor/actions/kill_browser",
            json={"browser_id": browser_id}
        )

        if response.status_code == 200:
            print(f"✅ Browser {browser_id} killed successfully")

Reset Statistics

async def reset_stats():
    async with httpx.AsyncClient() as client:
        response = await client.post("http://localhost:11235/monitor/stats/reset")
        print("📊 Statistics reset for fresh monitoring")

📈 Production Integration Patterns

Prometheus Integration

# Export metrics for Prometheus scraping
async def export_prometheus_metrics():
    async with httpx.AsyncClient() as client:
        health = await client.get("http://localhost:11235/monitor/health")
        data = health.json()

        # Export in Prometheus format
        metrics = f"""
# HELP crawl4ai_memory_usage_percent Memory usage percentage
# TYPE crawl4ai_memory_usage_percent gauge
crawl4ai_memory_usage_percent {data['container']['memory_percent']}

# HELP crawl4ai_request_success_rate Request success rate
# TYPE crawl4ai_request_success_rate gauge
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}

# HELP crawl4ai_browser_pool_count Total browsers in pool
# TYPE crawl4ai_browser_pool_count gauge
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
"""
        return metrics

Alerting Example

async def check_alerts():
    async with httpx.AsyncClient() as client:
        health = await client.get("http://localhost:11235/monitor/health")
        data = health.json()

        # Memory alert
        if data['container']['memory_percent'] > 80:
            print("🚨 ALERT: Memory usage above 80%")
            # Trigger cleanup
            await client.post("http://localhost:11235/monitor/actions/cleanup")

        # Success rate alert
        if data['stats']['success_rate_percent'] < 90:
            print("🚨 ALERT: Success rate below 90%")
            # Check error logs
            errors = await client.get("http://localhost:11235/monitor/logs/errors")
            print(f"Recent errors: {len(errors.json())}")

        # Latency alert
        if data['stats']['avg_latency_ms'] > 5000:
            print("🚨 ALERT: Average latency above 5s")

Key Metrics to Track

CRITICAL_METRICS = {
    "memory_usage": {
        "current": "container.memory_percent",
        "target": "<80%",
        "alert_threshold": ">80%",
        "action": "Force cleanup or scale"
    },
    "success_rate": {
        "current": "stats.success_rate_percent",
        "target": ">95%",
        "alert_threshold": "<90%",
        "action": "Check error logs"
    },
    "avg_latency": {
        "current": "stats.avg_latency_ms",
        "target": "<2000ms",
        "alert_threshold": ">5000ms",
        "action": "Investigate slow requests"
    },
    "browser_reuse_rate": {
        "current": "browsers.summary.reuse_rate_percent",
        "target": ">80%",
        "alert_threshold": "<60%",
        "action": "Check pool configuration"
    },
    "total_browsers": {
        "current": "browsers.summary.total_count",
        "target": "<15",
        "alert_threshold": ">20",
        "action": "Check for browser leaks"
    },
    "error_frequency": {
        "current": "len(errors)",
        "target": "<5/hour",
        "alert_threshold": ">10/hour",
        "action": "Review error patterns"
    }
}

🐛 Critical Bug Fixes

This release includes significant bug fixes that improve stability and performance:

Async LLM Extraction (#1590)

The Problem: LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).

The Fix: Resolved the blocking issue to enable true parallel processing for LLM extraction.

# Before v0.7.7: Sequential processing
# After v0.7.7: True parallel processing

async with AsyncWebCrawler() as crawler:
    urls = ["url1", "url2", "url3", "url4"]

    # Now processes truly in parallel with LLM extraction
    results = await crawler.arun_many(
        urls,
        config=CrawlerRunConfig(
            extraction_strategy=LLMExtractionStrategy(...)
        )
    )
    # 4x faster for parallel LLM extraction!

Expected Impact: Major performance improvement for batch LLM extraction workflows.

DFS Deep Crawling (#1607)

The Problem: DFS (Depth-First Search) deep crawl strategy had implementation issues.

The Fix: Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.

Browser & Crawler Config Documentation (#1609)

The Problem: Documentation didn't match the actual async_configs.py implementation.

The Fix: Updated all configuration documentation to accurately reflect the current implementation.

Sitemap Seeder (#1598)

The Problem: Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).

The Fix: Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.

Remove Overlay Elements (#1529)

The Problem: The remove_overlay_elements functionality wasn't working (issue #1396).

The Fix: Fixed by properly calling the injected JavaScript function.

Viewport Configuration (#1495)

The Problem: Viewport configuration wasn't working in managed browsers (issue #1490).

The Fix: Added proper viewport size configuration support for browser launch.

Managed Browser CDP Timing (#1528)

The Problem: CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).

The Fix: Added exponential backoff for CDP endpoint verification to handle timing variations.

Security Updates

• pyOpenSSL: Updated from >=24.3.0 to >=25.3.0 to address security vulnerability
• Added verification tests for the security update

Docker Fixes

• Port Standardization: Fixed inconsistent port usage (11234 vs 11235) - now standardized to 11235
• LLM Environment: Fixed LLM API key handling for multi-provider support (PR #1537)
• Error Handling: Improved Docker API error messages with comprehensive status codes
• Serialization: Fixed fit_html property serialization in /crawl and /crawl/stream endpoints

Other Important Fixes

• arun_many Returns: Fixed function to always return a list, even on exception (PR #1530)
• Webhook Serialization: Properly serialize Pydantic HttpUrl in webhook config
• LLMConfig Documentation: Fixed casing and variable name consistency (issue #1551)
• Python Version: Dropped Python 3.9 support, now requires Python >=3.10

📊 Expected Real-World Impact

For DevOps & Infrastructure Teams

• Full Visibility: Know exactly what's happening inside your crawling infrastructure
• Proactive Monitoring: Catch issues before they become problems
• Resource Optimization: Identify memory leaks and performance bottlenecks
• Operational Control: Manual intervention when automated systems need help

For Production Deployments

• Enterprise Observability: Prometheus, Grafana, and alerting integration
• Debugging: Real-time logs and error tracking
• Capacity Planning: Historical metrics for scaling decisions
• SLA Monitoring: Track success rates and latency against targets

For Development Teams

• Local Monitoring: Understand crawler behavior during development
• Performance Testing: Measure impact of configuration changes
• Troubleshooting: Quickly identify and fix issues
• Learning: See exactly how the browser pool works

🔄 Breaking Changes

None! This release is fully backward compatible.

• All existing Docker configurations continue to work
• No API changes to existing endpoints
• Monitoring is additive functionality
• No migration required

🚀 Upgrade Instructions

Docker

# Pull the latest version
docker pull unclecode/crawl4ai:0.7.7

# Or use the latest tag
docker pull unclecode/crawl4ai:latest

# Run with monitoring enabled (default)
docker run -d \
  -p 11235:11235 \
  --shm-size=1g \
  --name crawl4ai \
  unclecode/crawl4ai:0.7.7

# Access the monitoring dashboard
open http://localhost:11235/dashboard

Python Package

# Upgrade to latest version
pip install --upgrade crawl4ai

# Or install specific version
pip install crawl4ai==0.7.7

🎬 Try the Demo

Run the comprehensive demo that showcases all monitoring features:

python docs/releases_review/demo_v0.7.7.py

The demo includes: 1. System health overview with live metrics 2. Request tracking with active/completed monitoring 3. Browser pool management (permanent/hot/cold) 4. Complete Monitor API endpoint examples 5. WebSocket streaming demonstration 6. Control actions (cleanup, kill, restart) 7. Production metrics and alerting patterns 8. Self-hosting value proposition

📚 Documentation

New Documentation

• Self-Hosting Guide - Complete self-hosting documentation with monitoring
• Demo Script: docs/releases_review/demo_v0.7.7.py - Working examples

Updated Documentation

• Docker Deployment → Self-Hosting (renamed for better positioning)
• Added comprehensive monitoring sections
• Production integration patterns
• WebSocket streaming examples

💡 Pro Tips

1. Start with the dashboard - Visit /dashboard to get familiar with the monitoring system
2. Track the 6 key metrics - Memory, success rate, latency, reuse rate, browser count, errors
3. Set up alerting early - Use the Monitor API to build alerts before issues occur
4. Monitor browser pool efficiency - Aim for >80% reuse rate for optimal performance
5. Use WebSocket for custom dashboards - Build tailored monitoring UIs for your team
6. Leverage Prometheus integration - Export metrics for long-term storage and analysis
7. Check janitor logs - Understand automatic cleanup patterns
8. Use control actions judiciously - Manual interventions are for exceptional cases

🙏 Acknowledgments

Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.

The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.

📞 Support & Resources

• 📖 Documentation: docs.crawl4ai.com
• 🐙 GitHub: github.com/unclecode/crawl4ai
• 💬 Discord: discord.gg/crawl4ai
• 🐦 Twitter: @unclecode
• 📊 Dashboard: http://localhost:11235/dashboard (when running)

---

Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platform—it's a game changer for operational excellence!

Happy crawling with full visibility! 🕷️📊

- unclecode