Crawl4AI CLI Guide

Table of Contents

Basic Usage

The Crawl4AI CLI (crwl) provides a simple interface to the Crawl4AI library:

# Basic crawling
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# Verbose JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache

# See usage examples
crwl --example

Quick Example of Advanced Usage

If you clone the repository and run the following command, you will receive the content of the page in JSON format according to a JSON-CSS schema:

crwl "https://www.infoq.com/ai-ml-data-eng/" -e docs/examples/cli/extract_css.yml -s docs/examples/cli/css_schema.json -o json;

Configuration

Browser Configuration

Browser settings can be configured via YAML file or command line parameters:

# browser.yml
headless: true
viewport_width: 1280
user_agent_mode: "random"
verbose: true
ignore_https_errors: true

# Using config file
crwl https://example.com -B browser.yml

# Using direct parameters
crwl https://example.com -b "headless=true,viewport_width=1280,user_agent_mode=random"

Crawler Configuration

Control crawling behavior:

# crawler.yml
cache_mode: "bypass"
wait_until: "networkidle"
page_timeout: 30000
delay_before_return_html: 0.5
word_count_threshold: 100
scan_full_page: true
scroll_delay: 0.3
process_iframes: false
remove_overlay_elements: true
magic: true
verbose: true

# Using config file
crwl https://example.com -C crawler.yml

# Using direct parameters
crwl https://example.com -c "css_selector=#main,delay_before_return_html=2,scan_full_page=true"

Extraction Configuration

Two types of extraction are supported:

  1. CSS/XPath-based extraction: 
    # extract_css.yml
type: "json-css"
params:
  verbose: true

// css_schema.json
{
  "name": "ArticleExtractor",
  "baseSelector": ".article",
  "fields": [
    {
      "name": "title",
      "selector": "h1.title",
      "type": "text"
    },
    {
      "name": "link",
      "selector": "a.read-more",
      "type": "attribute",
      "attribute": "href"
    }
  ]
}
  1. LLM-based extraction: 
    # extract_llm.yml
type: "llm"
provider: "openai/gpt-4"
instruction: "Extract all articles with their titles and links"
api_token: "your-token"
params:
  temperature: 0.3
  max_tokens: 1000

// llm_schema.json
{
  "title": "Article",
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "The title of the article"
    },
    "link": {
      "type": "string",
      "description": "URL to the full article"
    }
  }
}

Advanced Features

LLM Q&A

Ask questions about crawled content:

# Simple question
crwl https://example.com -q "What is the main topic discussed?"

# View content then ask questions
crwl https://example.com -o markdown  # See content first
crwl https://example.com -q "Summarize the key points"
crwl https://example.com -q "What are the conclusions?"

# Combined with advanced crawling
crwl https://example.com \
    -B browser.yml \
    -c "css_selector=article,scan_full_page=true" \
    -q "What are the pros and cons mentioned?"

First-time setup: - Prompts for LLM provider and API token - Saves configuration in ~/.crawl4ai/global.yml - Supports various providers (openai/gpt-4, anthropic/claude-3-sonnet, etc.) - For case of ollama you do not need to provide API token. - See LiteLLM Providers for full list

Structured Data Extraction

Extract structured data using CSS selectors:

crwl https://example.com \
    -e extract_css.yml \
    -s css_schema.json \
    -o json

Or using LLM-based extraction:

crwl https://example.com \
    -e extract_llm.yml \
    -s llm_schema.json \
    -o json

Content Filtering

Filter content for relevance:

# filter_bm25.yml
type: "bm25"
query: "target content"
threshold: 1.0

# filter_pruning.yml
type: "pruning"
query: "focus topic"
threshold: 0.48

crwl https://example.com -f filter_bm25.yml -o markdown-fit

Output Formats

  • all - Full crawl result including metadata
  • json - Extracted structured data (when using extraction)
  • markdown / md - Raw markdown output
  • markdown-fit / md-fit - Filtered markdown for better readability

Complete Examples

  1. Basic Extraction: 

    crwl https://example.com \
    -B browser.yml \
    -C crawler.yml \
    -o json

  2. Structured Data Extraction: 

    crwl https://example.com \
    -e extract_css.yml \
    -s css_schema.json \
    -o json \
    -v

  3. LLM Extraction with Filtering: 

    crwl https://example.com \
    -B browser.yml \
    -e extract_llm.yml \
    -s llm_schema.json \
    -f filter_bm25.yml \
    -o json

  4. Interactive Q&A: 

    # First crawl and view
crwl https://example.com -o markdown

# Then ask questions
crwl https://example.com -q "What are the main points?"
crwl https://example.com -q "Summarize the conclusions"

Best Practices & Tips

  1. Configuration Management:
  2. Keep common configurations in YAML files
  3. Use CLI parameters for quick overrides

  4. Store sensitive data (API tokens) in ~/.crawl4ai/global.yml

  5. Performance Optimization:

  6. Use --bypass-cache for fresh content
  7. Enable scan_full_page for infinite scroll pages

  8. Adjust delay_before_return_html for dynamic content

  9. Content Extraction:

  10. Use CSS extraction for structured content
  11. Use LLM extraction for unstructured content

  12. Combine with filters for focused results

  13. Q&A Workflow:

  14. View content first with -o markdown
  15. Ask specific questions
  16. Use broader context with appropriate selectors

Recap

The Crawl4AI CLI provides: - Flexible configuration via files and parameters - Multiple extraction strategies (CSS, XPath, LLM) - Content filtering and optimization - Interactive Q&A capabilities - Various output formats