Write-up: https://dudeperf3ct.github.io/projects/train_llm_part0/
uv
- Install the dependencies:
uv sync-
Configure the settings in
config.yamlas needed. -
Run the dataset parsing script from root of the repo. It downloads the dataset on the first run if not already present at
data/raw_dataset/<name-of-dataset>path.
cd ..
python -m codellm_data.main parse- Run the dataset downloading script from root of the repo.
cd ..
python -m codellm_data.main download
