This repository accompanies our study on cross-lingual word embeddings for mutually intelligible African languages. We investigate how cross-lingual projection methods, including VecMap and MUSE, affect embedding quality and introduce a transfer gain framework for quantifying positive and negative transfer relative to monolingual baselines. The repository contains code for generating monolingual and cross-lingual embeddings, conducting bilingual lexicon ablations, and evaluating semantic similarity using the WordSim-353 and SimLex-999 benchmarks.
The fasttext_ablation.py script is used to train monolingual embeddings using FastText across different training data sizes. The data is stored in 'temp_corpora'. It performs dataset ablation experiments by varying the amount of input text used to train each embedding model, allowing us to study how embedding quality changes with data scale.
The script generates multiple FastText models per language, each trained on a different subset of the dataset, which are later used for evaluation on semantic similarity benchmarks such as WordSim-353 and SimLex-999.
The Monolingual_Evaluation.py script is used to evaluate the quality of the trained monolingual FastText embeddings. It computes embedding performance using Spearman’s rank correlation coefficient to measure alignment with human-annotated semantic similarity and relatedness judgments.
The evaluation is performed on standard benchmarks, including SimLex-999 and WordSim-353, which are stored in the BenchMarks/ directory. This script provides a consistent framework for assessing how well the learned embeddings capture semantic structure in a monolingual setting, and serves as the baseline for subsequent cross-lingual transfer experiments. This is ran through fasttext_eval_script.sh
The VecMap/ and MUSE/ directories contain implementations for generating cross-lingual embeddings using the pre-trained monolingual FastText embeddings. These methods are used to learn linear mappings between source and target embedding spaces, enabling cross-lingual representation alignment.
Both approaches are evaluated under multiple experimental conditions, including varying sizes of bilingual lexicons and different ablations of training data, in order to study their impact on alignment quality and downstream performance.
The bilingual lexicons used for training and evaluation are stored in the DLI_paired_lexicons_only_words/ directory, which contains multiple lexicon subsets of different sizes. These subsets are used to systematically analyse how the amount of bilingual supervision affects cross-lingual transfer performance.
The evaluate_muse_embeddings.py and evaluate_vecmap_embeddings.py scripts are used to evaluate cross-lingual embeddings produced by MUSE and VecMap, respectively. These evaluation pipelines apply the same experimental ablations used during training, including varying bilingual lexicon sizes and dataset proportions, to systematically assess their impact on alignment quality.
Performance is measured using Spearman’s rank correlation coefficient on standard semantic similarity benchmarks, including SimLex-999 and WordSim-353. This enables a direct comparison of cross-lingual embedding quality across different projection methods, supervision levels, and language pair configurations. This is ran through eval_muse.sh, and eval_vecmap.sh
Since the effects of cross-lingual transfer are not always directly observable from raw evaluation scores, we introduce a dedicated visualization pipeline implemented in transfer_gain_analyses.py. This script computes and visualizes transfer gain across different language pairs, projection methods, and experimental conditions.
The analysis highlights both positive and negative transfer relative to monolingual baselines, enabling a clearer interpretation of which languages benefit from cross-lingual alignment and under what conditions these gains or degradations occur. This provides a more intuitive and comparative view of transfer dynamics beyond tabulated evaluation results.
The Jaccard_similarity.py script is used to measure lexical overlap between corpora using the Jaccard similarity coefficient. This provides a simple but effective measure of similarity between datasets by comparing the intersection and union of unique token sets across languages or corpora.
Formally, it quantifies the proportion of shared vocabulary between two corpora, offering insight into lexical similarity and potential transferability between languages. This analysis is used to support the interpretation of cross-lingual transfer results by relating embedding performance to underlying corpus overlap.