🎯 PCrawler v2.0

Production web crawler với N8N + API orchestration

---

⚡ Quick Start

# Start core services (API + N8N + Celery)
docker compose up -d

# Access services
# API Docs: http://localhost:8000/docs
# N8N: http://localhost:5678
# Flower: http://localhost:5555

# Test Dynamic Crawl API
curl -X POST http://localhost:8000/n8n/crawl/dynamic \
  -H "Content-Type: application/json" \
  -d '{"table_name": "test", "url_pattern": "https://example.com/{id}", "ids": ["1"], "selectors": {"title": ["h1"]}}'

Core Services:

• ✅ API: http://localhost:8000
• ✅ API Docs: http://localhost:8000/docs
• ✅ N8N: http://localhost:5678
• ✅ Flower (Celery): http://localhost:5555

Optional: Add Airflow (heavy workflows)

docker compose -f docker-compose.yml -f docker-compose.airflow.yml up -d
# Airflow: http://localhost:8080 (admin/admin)

---

🏗️ Architecture

N8N (Visual workflows) ─┐
                        ├─→ PCrawler API ─→ Celery Workers ─→ Database
API (Direct access) ────┘    (FastAPI)       (Async crawl)

Optional: Airflow (Heavy batch workflows)

Extractors:

• lxml - Fast CSS/XPath (production)
• Crawl4AI - AI-powered extraction
• Regex - Emails, phones, URLs
• SmartExtractor - Auto-combine all (strategy: "auto")

---

🎯 Common Use Cases

1. Two-Phase Crawl (List → Detail) ⭐

Perfect for:

• Company directories (1900.com.vn)
• E-commerce (product listings → details)
• Job boards (listings → job details)
• Real estate (listings → property details)

Setup:

# Edit: airflow/dags/pcrawler_two_phase_crawl.py

LIST_CRAWL_CONFIG = {
    "base_url": "https://your-site.com/list",
    "pages": range(1, 11),  # Pages 1-10
    "selectors": {
        "links": {
            "selector": "a.item",  # ← Your CSS selector
            "attr": "href",
            "multiple": True,
        }
    },
    "pagination": {"param": "page", "type": "query_param"}
}

DETAIL_CRAWL_CONFIG = {
    "selectors": {
        "title": "h1.title",      # ← Your selectors
        "price": "span.price",
        "description": "div.desc",
    },
    "extract_emails": True,
    "extract_phones": True,
    "strategy": "auto",
}

Run:

1. Airflow UI → pcrawler_two_phase_crawl
2. Click Play
3. Monitor in Graph View
4. Check database for results

See: AIRFLOW.md for complete guide

---

2. Simple Single URL

Use: Test selectors, quick crawl

DAG: pcrawler_simple_crawl

---

3. Batch Crawl

Use: Daily scheduled crawls

DAG: pcrawler_batch_crawl

Schedule: Edit schedule_interval='0 2 * * *' for daily at 2 AM

---

📖 Documentation

File	What's Inside
AIRFLOW.md	Airflow guide, two-phase tutorial, config templates
N8N.md	N8N setup, workflows, API integration
REFERENCE.md	Cheatsheet, troubleshooting, test report

---

🚀 5-Minute Tutorial: Crawl 1900.com.vn

Step 1: Inspect HTML (2 min)

# Open browser → DevTools (F12)
# List page: https://1900.com.vn/danh-sach-cong-ty?page=1
#   Find: selector for company links (e.g., a.company-link)

# Detail page: https://1900.com.vn/cong-ty/example.html
#   Find: selectors for name, address, phone, etc.

Step 2: Edit Config (2 min)

File: airflow/dags/pcrawler_two_phase_crawl.py

LIST_CRAWL_CONFIG = {
    "base_url": "https://1900.com.vn/danh-sach-cong-ty",
    "pages": range(1, 4),  # Test 3 pages first!
    "selectors": {
        "company_links": {
            "selector": "a.YOUR-SELECTOR",  # ← Replace with actual
            "attr": "href",
            "multiple": True,
        }
    },
    "pagination": {"param": "page", "type": "query_param"}
}

DETAIL_CRAWL_CONFIG = {
    "selectors": {
        "company_name": "h1.name",     # ← Replace
        "address": ".address",
        "phone": ".phone",
    },
    "extract_emails": True,
    "extract_phones": True,
    "strategy": "auto",
}

Step 3: Run (1 min)

# Airflow auto-detects DAG in ~30s
# UI: http://localhost:8080
# Trigger: pcrawler_two_phase_crawl
# Watch logs in Graph View

Step 4: Check Results

# Expected logs:
# Phase 1: Found 60 URLs from 3 pages
# Phase 2: Success 58, Errors 2

# Query database:
docker compose exec api sqlite3 data/crawler_orchestrator.db \
  "SELECT * FROM data_items WHERE execution_id LIKE 'detail_%' LIMIT 5;"

Complete guide: AIRFLOW.md

---

💡 Key Points

❌ API Not Needed for Airflow

Airflow imports Python directly:

from app.engine.workflow import WorkflowEngine
# No HTTP calls needed!

API /n8n/* is only for N8N (external HTTP service).

✅ Test Selectors First

Browser console:

document.querySelectorAll('a.your-selector')

✅ Start Small

"pages": range(1, 3),  # Only 2 pages for testing

Scale up after confirming selectors work.

---

🆘 Common Issues

Issue	Fix
Phase 1 finds 0 URLs	Selector wrong → inspect HTML again
Phase 2 extracts wrong data	Field selector wrong → test in console
DAG not appearing	Wait 30s for auto-detect, check syntax errors
Scheduler unhealthy	Ignore - service running correctly

Full troubleshooting: REFERENCE.md

---

🎓 Learning Path

Day 1: Basic Setup

• docker compose up -d
• Access Airflow UI
• Trigger pcrawler_simple_crawl
• Check logs

Day 2: Two-Phase Crawl

• Read AIRFLOW.md
• Inspect target website
• Edit pcrawler_two_phase_crawl.py
• Test with 2-3 pages
• Scale up if successful

Day 3: Production

• Set up scheduling
• Add error handling
• Export results
• Explore N8N for visual workflows

---

📊 File Structure

pcrawler/
├── README.md              ← You are here
├── AIRFLOW.md             ← Complete Airflow guide
├── N8N.md                 ← N8N setup & usage
├── REFERENCE.md           ← Cheatsheet & troubleshooting
│
├── airflow/dags/
│   ├── pcrawler_two_phase_crawl.py  ← ⭐ Main DAG
│   ├── pcrawler_simple_crawl.py
│   └── pcrawler_batch_crawl.py
│
├── app/engine/
│   ├── extractors.py      ← SmartExtractor, lxml, Crawl4AI, Regex
│   ├── workflow.py
│   └── nodes/
│
└── docker-compose.yml

---

🔗 Documentation

Quick Start Guides

• Dynamic Crawl: docs/QUICK-START-DYNAMIC-CRAWL.md - Crawl 20k+ URLs với dynamic workflow
• Full Guide: docs/DYNAMIC-CRAWL-WORKFLOW.md - Complete documentation
• Scripts: scripts/README.md - Python CLI tools

Advanced Guides

• N8N Integration: N8N.md - Visual workflow automation
• Airflow (Optional): AIRFLOW.md - Heavy batch workflows
• API Reference: REFERENCE.md - Full API documentation

---

🚀 Common Commands

# Start/stop services
docker compose up -d
docker compose down

# View logs
docker compose logs -f api
docker compose logs -f worker

# Restart services
docker compose restart api worker

# Monitor tasks
# Flower: http://localhost:5555

# With Airflow (optional)
docker compose -f docker-compose.yml -f docker-compose.airflow.yml up -d

---

🎉 Ready to crawl! Start with Quick Start Guide