WebDiver is a versatile Python script for crawling websites, extracting internal and external links, titles, and descriptions. It's useful for tasks such as web analysis, OSINT (Open Source Intelligence) gathering, and competitive analysis.

A tool for search engine based competitive analysis. AWS Amplify project with many web crawlers for different search engines, and a simple web frontend for displaying historical results in Next.js

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap.txt or XML file for your website!

A simple WebCrawler for exploring and downloading content from web pages within a given domain/url.

.net-based Crawlers for news of Taiwan (.net 台灣新聞爬蟲，數據物件化，方便使用)

A high-performance, concurrent web crawler in Go that extracts URLs, downloads content, and converts tens of thousands of web pages to Markdown in minutes

Simple WebCrawler

Web Crawler written in C# that parses all urls from a specific page then recursively visits them while parsing all links available on that webpage

Various webcrawler programs made to gain a deeper understandings of webcrawling concepts.

🕷️ Crawl websites efficiently with this Bash script, producing a clean list of URLs while respecting `robots.txt` and staying within specified domains.

🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.