| Yes, (V1, V2, V3) |
Wikipedia |
wikipedia, wili2018, leipzigwiki |
Yes, however! |
Articles |
HF_Wikipedia dumps.wikipedia, wili_2018, Leipzig |
CC BY-SA 4.0 |
| Yes, (V1, V2, V3) |
SETIMES |
setimes |
Yes |
News |
Opus_SETIMES |
CC-BY-SA 3.0 |
| Yes, (V1, V2, V3) |
Tatoeba |
tatoeba |
Yes, mostly |
Crowdsourcing |
Tatoeba |
CC-BY |
| Yes, (V1, V2, V3) |
Global Voices |
globalvoices |
Yes |
News stories |
Opus_GlobalVoices |
CC BY 3.0 |
| Yes, (V1, V2, V3) |
XL-Sum |
xlsum |
Yes |
BBC News |
HF_xlsum, Github_xlsum |
CC BY-NC-SA 4.0 |
| Yes, (V1, V2, V3) |
Leipzig - News and Newscrawl |
leipzignews |
Yes |
News |
Leipzig |
CC BY-NC-SA 3.0 |
| Yes, (V3) |
FLORES dev |
floresdev |
Yes, mostly |
Human Translation |
FLORES |
CC-BY-SA 4.0 |
| Yes, (V1, V2, metadata check for V3) |
NLLB_seed |
nllbseed |
Yes |
Professionally-translated sentences (Wikipedia domain) |
NLLB_Seed |
CC-BY-SA 4.0 |
| Yes, (V1, V2, clean for V3) |
MT-560v1 |
mt560 |
?, openlid version is much cleaner. |
Multiple domain |
OpenLID compilation |
Apache License 2.0 |
| Yes, (V1, V2, V3) |
Autshumato |
autshumato |
Yes |
Government domain |
HF_autshumato |
CC Attribution 2.5 South Africa License |
| Yes, (V1, V2, metadata check for V3) |
Open Bibles |
pbc, cbc, bibleslti |
Yes, but some closely related languages (or similar name) might be confused. |
Bible versions |
1000Langs, PBC, CorpusCrawler, PNG, Open-Bibles, Bible.is, ebible, biblenlp-corpus, JHUBC, bible.com |
Mixed, CC family mostly CC BY-NC-ND |
| Yes (partly), (V1, V2, more clean for V3) |
JW |
jw, jw300, jw300data |
Yes, except sign language codes |
New World Bible |
Masakhane-Mt, JW, JW300 |
Usage for your own personal and non-commercial purposes is permitted. Copyrighted and distribution is not allowed. |
| Yes (partly), (V1, V2, V3) |
LTI |
lti |
? |
Multiple domain |
whatlang |
Custom License/ Open partly |
| Yes (partly), (V1, V2, delete some languages for V3) |
Arabic (DART, SHAMI, TSAC, PADIC, AOC, Arabic Dialects Dataset, MADAR) |
iadd, madar, arabdialects |
? |
Multiple domain |
IADD, MADAR, Arabic Dialects |
Multiple open licenses |
| Yes, (V1, V2, V3) |
Persian (TEP, MIZAN) |
tep, mizan |
Yes |
literature, subtitle |
TEP, MIZAN |
TEP: GNU General Public License, MIZAN: CC BY 4.0 |
| Yes, (V1, V2, V3) |
TIL Corpus |
til |
? |
Multiple domain |
TIL-MT |
CC BY-NC-SA 4.0 |
| Yes, (V1, V2, V3) |
bho-resources |
bhojpuri |
Yes |
News |
bho-resources |
CC BY-NC-SA 4.0 |
| Yes, (V1, V2, V3) |
Guaraní Parallel Set |
guarani |
Yes |
News |
Guaraní Parallel Set |
No explicit license |
| Yes, (V1, V2, V3) |
HKCanCor |
hkcancor |
Yes |
Transcribed conversations |
hkcancor |
CC BY 4.0 |
| Yes, (V2, V3) |
ai4d challenge (Nyanja) |
ai4d |
Yes |
News |
ai4d-malawi-news |
No explicit license |
| Yes, (V2, V3) |
Wanca 2016 |
wanca2016 |
? |
Web |
wanca2016 |
CC - BY |
| No, (V2, delete for V3) |
smugri |
|
No, after training V2 model we find this data is not clean |
News |
smugri-data |
CC BY 4.0 |
| No, (V2, delete for V3) |
finno-ugric |
|
? |
? |
finno-ugric-train |
CC BY 4.0 |
| Yes, (V2, V3) |
smugri-flores |
smugriflores |
Yes |
Human Translation |
smugri-flores-testset |
CC BY 4.0 |
| Yes, (V2, V3) |
Abkhaz National Corpus |
abnc |
Yes |
grammatically annotated text (linguistics, literary studies, history, political and social sciences) |
Abkhaz National Corpus, abkhaz_text |
Public domain (cc0-1.0) |
| Yes, (V2, V3) |
Luo News (Radio Ramogi) |
radioramogi |
Yes |
News |
Luo |
CC BY 4.0 |
| Yes, (V2, more metadata check for V3) |
Lyrics |
lyrics |
Yes, but there might be some issues with the translation part. |
Song lyrics |
lyricstranslate |
Copyright issues prevent distributing the original lyrics. Lyrics on Lyricstranslate.com are licensed through Musixmatch. Translations on Lyricstranslate.com belong to their authors. |
| Yes, (V2, V3) |
GlotSparse |
glotsparse |
Yes |
News and Articles |
HF_GlotSparse, Github_GlotSparse |
Web crawl data; copyright retained by original content owners. |
| Yes, (V2, more metadata check for V3) |
GlotStoryBook |
glotstorybook |
Yes |
StoryBooks |
HF_GlotStoryBook, Github_GlotStoryBook |
mostly CC family |
| Yes (partly), (V2, more clean for V3) |
Universal Dependencies v2.12 |
ud |
Yes, mostly |
Multiple domain |
UD |
CC family |
| Yes, (V2, V3) |
CommonVoice v11 |
commonvoice |
Yes, mostly |
crowdsourcing |
CommonVoice v11 |
Public domain (cc0-1.0) |
| Yes, (V2, V3) |
GOUD.MA |
goud |
?, only the headlines are definitely written in Moroccan Darija, but some noises exist. |
News |
Goud-sum |
No explicit license |
| Yes, (V2, V3) |
Vuk'uzenzele |
vukuzenzele |
Yes |
government domain |
vukuzenzele-monolingual |
License for Data - CC BY 4.0 |
| Yes, (V2, V3) |
Masakhanews |
masakhanews |
Yes |
News |
masakhane/masakhanews |
CC BY-NC 4.0 |
| Yes, (V2, V3) |
AfriQA |
afriqa |
Yes |
Human Translation |
masakhane/afriqa |
CC BY-NC 4.0 |
| Todo |
African News Corpus |
|
Yes |
News |
African News Corpus |
Non-Commercial Government Licence |
| Todo |
AfriSenti |
|
?, location tags converted to language tags followed by annotation |
Tweets |
AfriSenti |
CC BY 4.0 |
| Todo |
Bambara Dataset - Sentiment |
|
?, followed by annotation |
CommonCrawl |
Bambara Dataset |
No explicit license |
| Todo |
TUNIZI Dataset - Sentiment |
|
?, followed by annotation |
YouTube videos comments |
TUNIZI Dataset, HF_TUNIZI |
No explicit license |
| Todo |
CTAB |
|
?, followed by manual verification |
Facebook Public Pages |
Zenodo_CTAB |
CC BY 4.0 |
| Todo |
Open Subtitles |
|
? |
Movie subtitles |
OpenSubtitles |
|
| Todo |
GNOME |
|
Yes |
Human Translation |
opus_GNOME, HF_opus_GNOME, gnome/releases |
|
| Todo |
KDE4 |
|
Yes |
Human Translation |
opus_kde4 HF_kde4 kde4 |
|
| Todo |
Ubuntu |
|
Yes |
Human Translation |
HF_opus_ubuntu |
|
| Todo |
Web Inventory of Transcribed & Translated (WIT) Ted Talks |
|
? |
Ted talks |
ted_talks_iwslt |
|
| Todo |
igbo_monolingual |
|
Yes |
News, Radio, Books |
igbo_monolingual |
|
| Todo |
QADI |
|
?, location tags can be converted to language tags, but not very reliable for LID. |
Tweets |
QADI Tweet IDs |
Apache License 2.0 |
| Todo |
Mot |
|
Yes |
VOA News |
mot |
MIT License |
| Todo |
gov-za |
|
Yes |
government domain |
gov-za-monolingual |
License for Data - CC BY 4.0 |
| Yes, (V3) |
NusaT, NusaP, NusaX |
nusa |
Yes |
Human Translation |
indonlp, nusa-writes, NusaX |
CC-BY-SA 4.0 or Apache License 2.0 |
| Yes, (V3) |
MT Shared Task American NLP |
americasnlp2024 |
|
|
americasnlp2024 |
|
| Yes, (V3) |
ShaShiYaYi |
shashiyayi |
|
|
multilingual-data-peru |
|
| No, (delete for v3, it's code-switch) |
Hinglish collection |
|
|
|
english-to-hinglish |
|
| Yes, (V3) |
Indic corpora |
in22conv, in22gen |
|
|
In22Conv, In22Gen |
CC-BY-SA 4.0 |
| Yes, (V3) |
Yue parallel text |
yuecmnengparallel |
|
|
yue-cmn-eng |
|
| Todo |
Dakshina |
|
|
|
dakshina |
|
| Yes, (V3) |
Bhasha-Abhijnaanam |
bhashaabhijnaanam |
|
|
Bhasha-Abhijnaanam |
|
| Yes, (V3) |
Bloom Library |
bloombooks |
|
|
bloom-lm |
CC family 4.0 |
| Yes, (V4) |
FineWeb2 |
fineweb2 |
only und labels after recycling with GlotScript |
Web crawl / CommonCrawl derived web text |
FineWeb2 |
Web crawl data; copyright retained by original content owners. |
| Yes, (V4) |
Unicode/Google Fonts language samples |
googlefonts, unicodefonts |
only very low resource scripts |
Font samples / multilingual text examples |
rolandwalker/unicode-fonts , languagegeek.com |
|
| Yes, (V4) |
MINEDU Repository (Ministerio de Educación del Perú) |
minedu |
Yes (government datasets) |
Government / Education / Indigenous language resources |
repositorio.minedu , https://datos.minedu.gob.pe/ |
Peruvian Government Open Data License |
| Yes, (V4) |
GlotWeb / GlotCC corpus |
glotweb |
Yes, only confimred one |
Web pages for low-resource scripts and languages |
GlotWeb , GlotCC-V1 |
Web crawl data; copyright retained by original content owners. |
| Yes, (V4) |
Wiktionary wordlists |
wiktionary |
Yes |
Dictionary / lexical entries |
wiktionary_extract |
CC BY-SA 3.0 (Wiktionary content) |
| Yes, (V4) |
UDHR (only for rare scripts), the rest is for LID benchmarking |
udhr |
Yes |
Universal Declaration of Human Rights |
Omniglot, UDHR-LID |
Public Domain (United Nations) |