Skip to content

Commit 7b1608e

Browse files
bralbralbralbral
committed
added mmdb exporter
1 parent 2e314dd commit 7b1608e

4 files changed

Lines changed: 293 additions & 1 deletion

File tree

‎Readme.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -260,3 +260,80 @@ Examples:
260260
# Export custom table with pipe delimiter
261261
./bin/export_to_gzip -t users -d '|' -o users_dump
262262
```
263+
264+
# Export to MMDB format
265+
266+
You can also export the database to MaxMind DB (MMDB) format for use with GeoIP2-compatible tools:
267+
```
268+
./bin/export_to_mmdb
269+
```
270+
An MMDB file will be created with the current date in the filename.
271+
272+
**⚠️ HIGH MEMORY USAGE WARNING**
273+
The MMDB export process loads all data into memory before writing the final file. For large databases (millions of records), this can require several GB of RAM. Ensure you have sufficient memory available before running the export.
274+
275+
The script supports various options:
276+
```
277+
./bin/export_to_mmdb [OPTIONS]
278+
279+
Options:
280+
-t, --table TABLE Table name to export (default: block)
281+
-o, --output PREFIX Output file prefix (default: block_dump)
282+
-h, --host HOST Database host (default: db)
283+
-u, --user USER Database user (default: network_info)
284+
-n, --database NAME Database name (default: network_info)
285+
-p, --password PASSWORD Database password (default: network_info)
286+
--help Show this help message
287+
```
288+
289+
Examples:
290+
```
291+
# Default export
292+
./bin/export_to_mmdb
293+
294+
# Export custom table
295+
./bin/export_to_mmdb -t users -o users_dump
296+
297+
# Export with custom database connection
298+
./bin/export_to_mmdb -h localhost -u myuser -n mydb -p mypass
299+
```
300+
301+
## Using the MMDB file
302+
303+
The MMDB file contains the following network information fields for each IP range:
304+
305+
- **netname** - Network name identifier
306+
- **country** - Country code (e.g., US, RU, DE)
307+
- **description** - Network description
308+
- **maintained_by** - Maintainer information
309+
- **created** - Creation date (YYYY-MM-DD format)
310+
- **last_modified** - Last modification date (YYYY-MM-DD format)
311+
- **source** - Source RIR (ARIN, RIPE, APNIC, LACNIC, AfriNIC)
312+
- **status** - Network status
313+
314+
Once you have the MMDB file, you can use it with various tools:
315+
316+
### Python examples
317+
318+
Simple lookup:
319+
```python
320+
import maxminddb
321+
322+
# Open the MMDB file
323+
with maxminddb.open_database('block_dump_2026-05-01.mmdb') as reader:
324+
# Lookup an IP address
325+
result = reader.get('8.8.8.8')
326+
if result:
327+
print(f"Network: {result['netname']}")
328+
print(f"Country: {result['country']}")
329+
print(f"Source: {result['source']}")
330+
```
331+
332+
### Command line example
333+
```bash
334+
# Install mmdblookup tool
335+
sudo apt install mmdb-bin
336+
337+
# Lookup an IP address
338+
mmdblookup --file block_dump_2026-05-01.mmdb --ip 8.8.8.8
339+
```

‎bin/export_to_mmdb‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
#!/bin/bash
2+
3+
set -euf -o pipefail
4+
5+
# Detect docker compose command
6+
if command -v docker-compose &> /dev/null; then
7+
DOCKER_COMPOSE="docker-compose"
8+
else
9+
DOCKER_COMPOSE="docker compose"
10+
fi
11+
12+
# Default values
13+
TABLE="block"
14+
OUTPUT_PREFIX="block_dump"
15+
DB_HOST="db"
16+
DB_USER="network_info"
17+
DB_NAME="network_info"
18+
DB_PASSWORD="network_info"
19+
20+
# Parse arguments
21+
while [[ $# -gt 0 ]]; do
22+
case $1 in
23+
-t|--table)
24+
TABLE="$2"
25+
shift 2
26+
;;
27+
-o|--output)
28+
OUTPUT_PREFIX="$2"
29+
shift 2
30+
;;
31+
-h|--host)
32+
DB_HOST="$2"
33+
shift 2
34+
;;
35+
-u|--user)
36+
DB_USER="$2"
37+
shift 2
38+
;;
39+
-n|--database)
40+
DB_NAME="$2"
41+
shift 2
42+
;;
43+
-p|--password)
44+
DB_PASSWORD="$2"
45+
shift 2
46+
;;
47+
--help)
48+
echo "Usage: $0 [OPTIONS]"
49+
echo ""
50+
echo "Options:"
51+
echo " -t, --table TABLE Table name to export (default: block)"
52+
echo " -o, --output PREFIX Output file prefix (default: block_dump)"
53+
echo " -h, --host HOST Database host (default: db)"
54+
echo " -u, --user USER Database user (default: network_info)"
55+
echo " -n, --database NAME Database name (default: network_info)"
56+
echo " -p, --password PASSWORD Database password (default: network_info)"
57+
echo " --help Show this help message"
58+
echo ""
59+
echo "Examples:"
60+
echo " # Default export"
61+
echo " $0"
62+
echo ""
63+
echo " # Export custom table"
64+
echo " $0 -t users -o users_dump"
65+
echo ""
66+
echo " # Export with custom database connection"
67+
echo " $0 -h localhost -u myuser -n mydb -p mypass"
68+
exit 0
69+
;;
70+
*)
71+
echo "Unknown option: $1"
72+
echo "Use --help for usage information"
73+
exit 1
74+
;;
75+
esac
76+
done
77+
78+
# Generate output filename with date
79+
OUTPUT_FILE="${OUTPUT_PREFIX}_$(date +'%Y-%m-%d').mmdb"
80+
81+
# Build connection string
82+
CONNECTION_STRING="postgresql+psycopg://${DB_USER}:${DB_PASSWORD}@${DB_HOST}:5432/${DB_NAME}"
83+
84+
echo "Exporting table '$TABLE' from database '$DB_NAME'..."
85+
echo "Output: $OUTPUT_FILE"
86+
87+
$DOCKER_COMPOSE run --rm \
88+
-e PGPASSWORD="$DB_PASSWORD" \
89+
-v "$(pwd):/app/output" \
90+
--name mmdb_exporter \
91+
--entrypoint=python3 \
92+
network_info \
93+
export_to_mmdb.py \
94+
-c "$CONNECTION_STRING" \
95+
-o "/app/output/$OUTPUT_FILE" \
96+
-t "$TABLE"
97+
98+
echo "Export completed successfully: $OUTPUT_FILE"

‎export_to_mmdb.py‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
#!/usr/bin/env python3
2+
# -*- coding: utf-8 -*-
3+
4+
import argparse
5+
import sys
6+
import gc
7+
from datetime import datetime
8+
import netaddr
9+
from sqlalchemy import select
10+
from sqlalchemy.orm import sessionmaker
11+
from db.model import Block
12+
from db.helper import create_postgres_pool
13+
14+
15+
def export_to_mmdb(connection_string, output_file, table="block"):
16+
"""Export PostgreSQL table to MMDB format"""
17+
18+
# Create database connection
19+
engine = create_postgres_pool(connection_string)
20+
Session = sessionmaker(bind=engine)
21+
session = Session()
22+
23+
# Query all blocks with streaming to avoid memory issues
24+
print(f"Exporting table '{table}' to MMDB format...")
25+
query = select(Block)
26+
result = session.execute(query)
27+
28+
# Use yield_per to stream blocks instead of loading all at once
29+
blocks = result.scalars().yield_per(1000)
30+
31+
print(f"Starting export (streaming blocks)...")
32+
33+
try:
34+
from mmdb_writer import MMDBWriter
35+
except ImportError:
36+
print("Error: mmdb_writer library not found. Please install it with: pip install mmdb-writer")
37+
sys.exit(1)
38+
39+
writer = MMDBWriter(
40+
ip_version=6,
41+
ipv4_compatible=True,
42+
database_type="Network Info Block",
43+
languages=["en"],
44+
description={
45+
"en": "Network block information from RIR databases"
46+
},
47+
)
48+
49+
count = 0
50+
51+
for block in blocks:
52+
try:
53+
# Skip default route blocks that aren't real networks
54+
cidr = str(block.inetnum)
55+
if cidr in ['::/0', '0.0.0.0/0']:
56+
continue
57+
58+
# Create the data structure for this network - minimize memory usage
59+
data = {
60+
"netname": block.netname or "",
61+
"country": block.country or "",
62+
"description": block.description or "",
63+
"maintained_by": block.maintained_by or "",
64+
"created": block.created.strftime('%Y-%m-%d') if block.created else "",
65+
"last_modified": block.last_modified.strftime('%Y-%m-%d') if block.last_modified else "",
66+
"source": block.source or "",
67+
"status": block.status or ""
68+
}
69+
70+
# Insert network using IPSet with CIDR string list
71+
ipset = netaddr.IPSet([cidr])
72+
writer.insert_network(ipset, data)
73+
count += 1
74+
75+
if count % 10000 == 0:
76+
print(f" Processed {count} blocks...")
77+
# Force garbage collection periodically
78+
gc.collect()
79+
80+
except Exception as e:
81+
print(f"Warning: Failed to process block {block.inetnum}: {e}")
82+
continue
83+
84+
# Write to file
85+
writer.to_db_file(output_file)
86+
print(f"Export completed successfully: {count} blocks -> {output_file}")
87+
88+
session.close()
89+
90+
91+
def main():
92+
parser = argparse.ArgumentParser(description="Export PostgreSQL table to MMDB format")
93+
parser.add_argument("-c", "--connection",
94+
default="postgresql://network_info:network_info@db:5432/network_info",
95+
help="Database connection string")
96+
parser.add_argument("-o", "--output",
97+
default="block_dump.mmdb",
98+
help="Output MMDB file path")
99+
parser.add_argument("-t", "--table",
100+
default="block",
101+
help="Table name to export")
102+
103+
args = parser.parse_args()
104+
105+
# Add date to output filename if not already present
106+
if "_$(date" not in args.output and not args.output.endswith(".mmdb"):
107+
args.output = f"{args.output}_{datetime.now().strftime('%Y-%m-%d')}.mmdb"
108+
elif not args.output.endswith(".mmdb"):
109+
args_output = args.output.rsplit(".", 1)[0]
110+
args.output = f"{args_output}_{datetime.now().strftime('%Y-%m-%d')}.mmdb"
111+
112+
export_to_mmdb(args.connection, args.output, args.table)
113+
114+
115+
if __name__ == "__main__":
116+
main()

‎requirements.txt‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
netaddr==1.3.0
1+
netaddr>=0.8.0
22
psycopg==3.3.3
33
psycopg-c==3.3.3
44
psycopg-pool==3.3.0
55
SQLAlchemy==2.0.49
6+
mmdb-writer==0.2.6

0 commit comments

Comments
 (0)