Skip to content

README_SCRAPER.md

Latest commit

 

History

History
72 lines (55 loc) · 2.07 KB

README_SCRAPER.md

File metadata and controls

72 lines (55 loc) · 2.07 KB
Raw

Wiktionary Cognate Scraper

This script scrapes Wiktionary to find true cognate pairs by:

  1. Finding pages for reconstructed ancestor words (Proto-languages)
  2. Extracting descendant words from those pages
  3. Creating pairs of descendants as true cognates

Installation

Install the required dependencies:

pip install -r requirements_scraper.txt

Or install individually:

pip install requests beautifulsoup4 lxml

Usage

Run the scraper:

python scrape_cognates.py

The script will:

  • Search for reconstructed words in Proto-Indo-European, Proto-Germanic, Proto-Slavic, Proto-Celtic, and Proto-Italic
  • Extract descendant words from each reconstructed word page
  • Create cognate pairs from descendants of the same ancestor
  • Save results to scraped_cognates.csv (in the format used by eval.py) and scraped_cognates.json (with detailed information)

Customization

You can modify the main() function to:

  • Add more proto-languages to search
  • Change the number of reconstructed words to check per language
  • Adjust the minimum number of descendants required

Example:

cognate_pairs = scraper.scrape_cognate_pairs(
    proto_languages=["Proto-Indo-European", "Proto-Germanic"],
    num_words_per_language=20,  # Check 20 reconstructed words per language
    min_descendants=2  # Need at least 2 descendants to create pairs
)

Output Format

CSV Output (scraped_cognates.csv)

Compatible with your existing eval.py script:

input,target
English "brother" and German "Bruder",Yes
Latin "frater" and English "brother",Yes

JSON Output (scraped_cognates.json)

Detailed information including:

  • Proto-word and proto-language
  • Both descendant words with their languages
  • Original source information

Notes

  • The script includes polite rate limiting (0.5-1 second delays between requests)
  • Wiktionary's structure can vary, so parsing may not catch all descendants
  • Some reconstructed words may have many descendants, others very few
  • The script uses Wiktionary's API for searching and regular HTTP requests for page content