Skip to content
BranchesTags

Repository files navigation

Data Centers Are Eating The World 🌍

An open-source, automated mapping project that makes the invisible infrastructure powering our digital lives visible—starting with Kenya, expanding globally.

Project Status License Data License

🎯 What is This?

An interactive, real-time mapping platform that:

  • Makes Infrastructure Visible - Interactive maps showing every data center
  • Tracks Growth Over Time - Timeline showing infrastructure expansion
  • Reveals Impact - Visualize power consumption and economic investment
  • Stays Current - Automated scrapers pull data from multiple sources with high-precision Google Geocoding
  • Frictionless Submissions - Google Places Autocomplete enables public submitters to easily suggest data center facilities

🚀 Quick Start

Prerequisites

  • Node.js 18+ and npm
  • Python 3.9+
  • PostgreSQL 14+ with PostGIS extension

Installation

# Clone the repository
git clone [your-repo-url]
cd data_centers_mapping

# Install all dependencies
npm run install:all

# Install Python dependencies
cd scraper && pip install -r requirements.txt

Setup Database

# Create PostgreSQL database with PostGIS
createdb datacenter_map
psql datacenter_map -c "CREATE EXTENSION postgis;"

# Run migrations
npm run db:setup
npm run db:migrate

Configuration

Create .env files in backend/, frontend/, and scraper/:

backend/.env (see backend/.env.example)

DATABASE_URL=postgresql://localhost:5432/datacenter_map
PORT=3001
NODE_ENV=development
JWT_SECRET=your-32plus-char-random-secret
# Optional Google admin sign-in (same OAuth Client ID as frontend VITE_GOOGLE_CLIENT_ID):
# GOOGLE_CLIENT_ID=...
# ADMIN_GOOGLE_EMAILS=you@gmail.com
# FRONTEND_ORIGIN=http://localhost:5173

frontend/.env

VITE_API_URL=http://localhost:3001
VITE_MAPBOX_TOKEN=your_mapbox_token_here
# Unified Google Maps Platform API Key (used for Places Autocomplete):
GOOGLE_MAPS_API_KEY=your_google_maps_api_key_here
# Same OAuth Client ID as backend GOOGLE_CLIENT_ID if using Google admin sign-in:
# VITE_GOOGLE_CLIENT_ID=...

scraper/.env

DATABASE_URL=postgresql://localhost:5432/datacenter_map
# Unified Google Maps Platform API Key (used for Geocoding & Places Text Search):
GOOGLE_MAPS_API_KEY=your_google_maps_api_key_here

Google Cloud Setup: Enable Places API, Geocoding API, and Maps JavaScript API on your Google Cloud Project. A single GOOGLE_MAPS_API_KEY cuts across both frontend and scraper pipelines.

Run the Application

# Start both frontend and backend
npm run dev

# Or start separately:
npm run dev:frontend  # Frontend at http://localhost:5173
npm run dev:backend   # Backend at http://localhost:3001

Run Data Scrapers

npm run scrape

News Monitor 🎉

Automatically monitors news sources for data center announcements and flags them for manual review:

cd scraper
python news_monitor.py

Or integrate into main scraper:

python main.py --include-news  # Run news monitor + normal scraping
python main.py --news-only      # Only check news (no DB updates)

🗺️ Google Maps Platform (GMP) Integration

Google Maps Platform is integrated across the system:

  1. Public Facility Suggestions (frontend/src/pages/SuggestFacility.tsx)

    • Integrated Google Places Autocomplete widget.
    • Typing a facility name or commercial building auto-populates Address, City, Country, Latitude, and Longitude, removing decimal coordinate typing friction for public submitters.

  2. Scraper Data Pipeline (scraper/processors/geocoder.py)

    • Replaced raw Nominatim queries with Google Geocoding API & Google Places Text Search REST API.
    • Accurately resolves named commercial data center facilities across African cities where standard open geocoders fall back to city centroids.

  3. Persistent Disk Caching (scraper/data/geocoding_cache.json)

    • All resolved geocoding results are cached to a persistent JSON disk cache across scraper executions.
    • Prevents redundant API calls, saves quota, and speeds up pipeline runs.

  4. Graceful Fallback Mode

    • If no Google API key is configured, the frontend gracefully preserves standard manual input fields, and the scraper automatically falls back to Nominatim.

🏗️ Project Structure

data_centers_mapping/
├── frontend/          # React + TypeScript UI
│   ├── src/
│   │   ├── components/   # UI components (Mapbox, ProtectedRoute, Navbar, etc.)
│   │   ├── pages/        # Page components (Map, SuggestFacility, AdminDashboard, Login)
│   │   ├── hooks/        # Custom React hooks
│   │   ├── services/     # API integration services
│   │   └── types/        # TypeScript type definitions
│   └── public/
├── backend/           # Node.js/Express API
│   ├── src/
│   │   ├── routes/       # Express API routes
│   │   ├── controllers/  # Ingestion & Admin controllers
│   │   ├── models/       # Database access models
│   │   ├── middleware/   # Auth & error handling middleware
│   │   └── utils/        # Utility modules
│   └── db/
│       └── migrations/   # PostgreSQL database schema & migrations
├── scraper/           # Python web scrapers & geocoding pipeline
│   ├── scrapers/         # Source scrapers (datacentermap, datacenterscom, manual_data)
│   ├── processors/       # Geocoder (GMP + Nominatim fallback), Deduplicator
│   ├── db/               # Scraper PostgreSQL database connection & sync
│   └── data/             # Curated datasets & persistent geocoding cache (geocoding_cache.json)
└── docs/              # Architectural, setup, deployment, and API guides

🛠️ Tech Stack

Frontend

  • React 18 + TypeScript
  • Vite - Fast build tool
  • Google Maps Platform - Places Autocomplete API
  • Mapbox GL JS - Interactive vector maps & spatial layers
  • D3.js - Data visualizations & analytics
  • Tailwind CSS - Styling & responsive UI
  • React Query - Data fetching & caching state

Backend

  • Node.js + Express
  • PostgreSQL + PostGIS - Geospatial database
  • pg (node-postgres) - Database client
  • Node-cron - Scheduled background tasks

Data Pipeline

  • Python 3.9+
  • Google Maps Geocoding & Places Text Search API - High-precision facility location resolution
  • Persistent JSON Disk Cache - Zero-redundancy geocoding cache
  • Geopy / Nominatim - Geocoding fallback
  • BeautifulSoup4 & Selenium - Dynamic scraping
  • FuzzyWuzzy - Record deduplication

📊 Features

Live Features

  • ✅ Interactive map with data center locations & clustering
  • ✅ Filter by status, capacity, ownership type
  • ✅ Public facility suggestion form with Google Places Autocomplete
  • ✅ Detail panel for each facility
  • ✅ Automated scrapers with persistent Google Geocoding
  • ✅ News monitor for infrastructure announcements
  • ✅ Export data (JSON, CSV, GeoJSON)
  • ✅ Admin dashboard for reviewing & approving public/scraper suggestions

Planned Features

  • Timeline slider (2010-2025)
  • Layer system (power grid, fiber, submarine cables)
  • Impact calculator
  • Public API endpoints
  • Automated email alerts on ingestion

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

📄 License

  • Code: MIT License
  • Data: CC BY 4.0 (attribution required)

👨‍💻 Author & Attribution

Created by: Victor Jotham Ashioya

How to Attribute This Project

If you use this project or its data, please provide attribution:

For Code (MIT License):

Data Centers Mapping Platform by Victor Jotham Ashioya
https://github.com/ashioyajotham/data_centers_mapping

For Data (CC BY 4.0):

Data sourced from "Data Centers Are Eating The World" by Victor Jotham Ashioya
Licensed under CC BY 4.0
https://github.com/ashioyajotham/data_centers_mapping

🙏 Acknowledgments

Inspired by:

About

This idea is mostly how supercomputers (AI data centers) are coming up fast so it is an attempt to map them like Semi-Analysis does

ashioyajotham.github.io/projects.html

Topics

data-centerdata-visualizationmappingneon-postgresneondbpostgis-databaserender-deploymentsupercomputerstypescriptvercel-deployment

Resources

Readme
License

Contributing

Contributing
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages