gocrawler — Concurrent Web Crawler with Redis

A concurrent web crawler built from scratch in Go as a learning project for a distributed databases course.

Architecture

Fetcher  → raw HTML bytes
Parser   → extracts links (href + text)
Store    → Redis backend (visited set + frontier queue)
Crawler  → orchestrates: worker pool + depth control

What it does

• Fetches pages with configurable timeout and User-Agent
• Parses HTML and extracts all links
• Normalizes URLs (resolves relative paths, strips fragments)
• Deduplicates via Redis SET (persistent across crashes)
• Frontier queue via Redis LIST (survives restarts)
• Concurrent worker pool with clean shutdown (context + WaitGroup)

Run locally

# Start Redis
docker run -d --name gocrawler-redis -p 6379:6379 redis:alpine

# Run the crawler
go run cmd/gocrawler/main.go https://example.com

Run with Docker Compose

docker compose run --rm crawler https://example.com

Configuration

Env var	Default	Purpose
REDIS_ADDR	localhost:6379	Redis server address

Depth and worker count are set in main.go via NewCrawler(fetcher, parser, store, maxDepth, numWorkers).

What was learned

• Go concurrency: goroutines, channels, sync.WaitGroup, context cancellation
• Worker pool pattern for controlled parallelism
• HTML parsing with golang.org/x/net/html
• Redis integration: SET for dedup, LIST for queue, BRPOP blocking pop
• Docker multi-stage builds and Docker Compose for multi-service apps