A multilingual parallel corpus from South African government publications in 11 official languages
The Vuk'uzenzele South African Multilingual Corpus contains text data extracted from the South African government magazine Vuk'uzenzele, published by the Government Communication and Information System (GCIS). This corpus provides valuable parallel text data across 11 South African languages, making it an important resource for multilingual NLP research in African languages.
- Raw data: 192 editions from November 2012 to December 2022
- Processed & aligned data: 84 editions from January 2018 to December 2022
- Languages: 11 South African official languages
- Total aligned pairs: 60,000+ sentence pairs across 55 language pair combinations
📦 HuggingFace Datasets:
- Monolingual dataset - Individual language data with train/test/eval splits
- Sentence-aligned dataset - Parallel sentence pairs across language combinations
📚 Zenodo Archive: https://doi.org/10.5281/zenodo.7598539
📄 Research Paper: ACL Anthology | arXiv
💬 Give Feedback: DSFSI Resource Feedback Form
The corpus includes all 11 South African official languages:
| Language | Code | Language | Code |
|---|---|---|---|
| English | eng | Sepedi | sep |
| Afrikaans | afr | Setswana | tsn |
| isiNdebele | nbl | Siswati | ssw |
| isiXhosa | xho | Tshivenda | ven |
| isiZulu | zul | Xitstonga | tso |
| Sesotho | nso |
| src_lang | trg_lang | num_aligned_pairs |
|---|---|---|
| ssw | xho | 2202 |
| ssw | zul | 2183 |
| xho | zul | 2102 |
| nso | xho | 2081 |
| nso | tso | 2071 |
| ssw | tso | 2034 |
| nso | ssw | 2021 |
| tsn | tso | 2020 |
| tsn | xho | 2009 |
| tso | xho | 2009 |
| nso | tsn | 2002 |
| ssw | tsn | 1987 |
| tso | zul | 1957 |
| nso | zul | 1953 |
| tsn | zul | 1933 |
| eng | zul | 1923 |
| eng | tso | 1923 |
| eng | nso | 1867 |
| eng | ssw | 1821 |
| afr | xho | 1816 |
| eng | xho | 1801 |
| nbl | sep | 1795 |
| sep | ven | 1794 |
| afr | ssw | 1783 |
| eng | tsn | 1772 |
| afr | zul | 1769 |
| afr | nso | 1746 |
| nbl | ven | 1699 |
| afr | eng | 1661 |
| afr | tsn | 1631 |
| afr | tso | 1617 |
| afr | sep | 551 |
| afr | ven | 498 |
| afr | nbl | 491 |
| nso | sep | 410 |
| nso | ven | 352 |
| sep | tso | 326 |
| sep | tsn | 319 |
| tso | ven | 307 |
| sep | ssw | 305 |
| sep | xho | 300 |
| ssw | ven | 290 |
| tsn | ven | 285 |
| nbl | ssw | 282 |
| nbl | nso | 266 |
| ven | xho | 260 |
| eng | sep | 258 |
| nbl | xho | 250 |
| sep | zul | 249 |
| nbl | tso | 238 |
| eng | ven | 234 |
| nbl | tsn | 230 |
| nbl | zul | 226 |
| ven | zul | 225 |
| eng | nbl | 184 |
from datasets import load_dataset
# Load monolingual data for a specific language
dataset = load_dataset("dsfsi/vukuzenzele-monolingual", "zul") # isiZulu
print(dataset['train'][0])
# Load sentence-aligned parallel data
aligned_dataset = load_dataset("dsfsi/vukuzenzele-sentence-aligned", "eng-zul")The repository contains data in multiple stages of processing:
./data
├── raw/ # Raw PDFs of Vuk'uzenzele magazine (2012-2022, 192 editions)
├── interim/ # Extracted text requiring manual refinement
├── processed/ # Manually cleaned text files (2018-2022, 84 editions)
├── opt_aligned_out/ # LASER-aligned sentence pairs (CSV format)
├── sentence_align_output/ # Sentence alignment outputs
├── simple_align_output/ # Simple 1:1 sentence alignment (CSV)
├── editions_json/ # JSON format outputs
├── monolingual_jsonl/ # JSONL format for HuggingFace
└── huggingface/ # Prepared datasets for HuggingFace Hub
Each edition follows the naming format: YYYY-MM-edN (e.g., 2020-01-ed1).
We welcome contributions! There are several ways you can help expand this corpus:
The Vuk'uzenzele magazine has been published since 2003, but our corpus currently only covers 2012-2022. We need help expanding coverage to:
- Earlier years: 2003-2011 editions
- Recent years: 2023 onwards
- Download PDFs: Get Vuk'uzenzele PDFs from the official archive
- Extract text: Follow the extraction workflow in CLAUDE.md or src/data/ReadME.md
- Refine data: Manually clean extracted text to ensure quality
- Submit PR: Create a pull request with your additions
- Improve alignment: Enhance sentence alignment algorithms
- Add metadata: Extract publication dates, authors, topics
- Quality control: Review and correct existing alignments
- Documentation: Improve guides and examples
- Applications: Build NLP models and share results
See our GitHub Issues for specific tasks or create a new issue to discuss your ideas!
# Clone the repository
git clone https://github.com/dsfsi/vukuzenzele-nlp.git
cd vukuzenzele-nlp
# Install dependencies
pip install -r requirements.txt
# See CLAUDE.md for detailed development commandsThis corpus has been used for:
- Machine Translation: Training translation models for South African language pairs
- Language Models: Pretraining multilingual models like PuoBERTa
- Speech Recognition: Creating the Vukuzenzele isiXhosa Speech Dataset
- Cross-lingual Transfer: Evaluating multilingual NLP systems
- Low-resource Language Research: Advancing NLP for under-resourced African languages
If you use this corpus in your research, please cite:
@inproceedings{lastrucci-etal-2023-preparing,
title = "Preparing the Vuk{'}uzenzele and {ZA}-gov-multilingual {S}outh {A}frican multilingual corpora",
author = "Richard Lastrucci and Isheanesu Dzingirai and Jenalea Rajab and Andani Madodonga and Matimba Shingange and Daniel Njini and Vukosi Marivate",
booktitle = "Proceedings of the Fourth workshop on Resources for African Indigenous Languages (RAIL 2023)",
month = may,
year = "2023",
address = "Dubrovnik, Croatia",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2023.rail-1.3",
pages = "18--25"
}@dataset{marivate_vukosi_2023_7598540,
author = {Marivate, Vukosi and
Njini, Daniel and
Madodonga, Andani and
Lastrucci, Richard and
Dzingirai, Isheanesu and
Rajab, Jenalea},
title = {The Vuk'uzenzele South African Multilingual Corpus},
month = feb,
year = 2023,
publisher = {Zenodo},
doi = {10.5281/zenodo.7598539},
url = {https://doi.org/10.5281/zenodo.7598539}
}Core Team:
- Vukosi Marivate - Project Lead
- Andani Madodonga
- Daniel Njini
- Richard Lastrucci
- Isheanesu Dzingirai
- Jenalea Rajab
Affiliated with the Data Science for Societal Impact Research Group
This dataset contains machine-readable data extracted from PDF documents from https://www.vukuzenzele.gov.za/, provided by the Government Communication Information System (GCIS). While efforts were made to ensure accuracy and completeness, there may be errors or discrepancies between the original publications and this dataset.
No warranties, guarantees, or representations are given regarding the information in this dataset. Neither the Data Science for Societal Impact Research Group nor the Government Communication Information System (GCIS) bear responsibility for any errors or discrepancies. Users should verify all information before making decisions based on this data.
Maintained by: Data Science for Societal Impact Research Group (DSFSI)
Contact: Create an issue or use our feedback form