LoKr Training on AMD RX 7900 XT (ROCm 7.2, Windows 11) — Working Setup + the factor=8 instability nobody's documented

[tuckeranglemyer-pixel]

follow-up to my earlier AMD setup guide (the RX 7900 XT / ROCm 7.2 one over on the old repo, Discussion #404. this covers LoKr training specifically on AMD, which i couldn't find documented anywhere. i ran it a bunch this weekend, here's what actually works and one real gotcha that cost me two full runs so it doesn't cost you yours.

short version: a full LoKr trained on my 7900 XT in 16 minutes (vs ~1hr for a standard LoRA), but only with one specific config. the "correct" textbook LoKr config crashes on ROCm. details below.

my setup: RX 7900 XT (20GB, RDNA3), Windows 11, ROCm 7.2, Python 3.12, PyTorch 2.9.1+rocmsdk. if you got inference + standard LoRA working off my #404 guide you already have this. trainer is Side-Step (the standalone CLI).

1. Install LyCORIS

LoKr support comes from LyCORIS, not in the default install:

pip install lycoris-lora

2. Preprocess (same as standard LoRA)

python -m acestep.training_v2.cli.train_fixed --preprocess --audio-dir YOUR_AUDIO_FOLDER --tensor-output preprocessed\my_dataset --checkpoint-dir checkpoints --model-variant turbo --max-duration 240 --device auto --precision fp32

you'll get a wall of MIOpen(HIP) Warning [IsEnoughWorkspace] lines. harmless — it's MIOpen trying a fast conv kernel, not getting scratch memory, falling back to a slower one. the workspace numbers cycling up/down just means it's hitting different track lengths. look for [OK] Preprocessing complete: Processed: N/N. ~4 min/track on my card.

3. Train — the config that WORKS on AMD

python -m acestep.training_v2.cli.train_fixed --dataset-dir preprocessed\my_dataset --output-dir lora_output\my_lokr --checkpoint-dir checkpoints --model-variant turbo --device auto --precision fp32 --adapter-type lokr --lokr-linear-dim 64 --lokr-linear-alpha 128 --epochs 20 --batch-size 1 --gradient-accumulation 4 --offload-encoder --save-every 5 --sample-every-n-epochs 0 --yes

result on my 7900 XT: loss 1.46 -> 1.13, 16m33s, peak VRAM 8.9GB, checkpoints at epochs 5/10/15/20 + final. loads and generates fine.

AMD-specific notes:

• --precision fp32 is the ROCm-safe one (same precision that works for inference on these cards). even with fp32 passed, the trainer runs bf16-mixed under the hood — that's fine, bf16 has fp32's dynamic range, it's stable, not the NaN-prone fp16 kind.
• --offload-encoder moves VAE/encoder to CPU after setup. peak was under 9GB so an 8GB card should manage.
• --save-every 5 because the lowest-loss epoch isn't always the best-sounding. A/B them.

THE GOTCHA: --lokr-factor 8 (true Kronecker decomposition) crashes on ROCm

here's the thing nobody's documented. the config above uses the default --lokr-factor -1, and LyCORIS logs this at load:

[LyCORIS]-WARNING: lora_dim 64 is too large for dim=2048 and factor=-1, using full matrix mode.

so with the default, you're not really getting the compact Kronecker-factored LoKr — it falls back to full matrix mode. it trains fine and works, but it's not the "true" lean LoKr.

so i tried to force the real decomposition the way LyCORIS wants:

--lokr-factor 8 --lokr-linear-dim 10000 --lokr-linear-alpha 10000

(huge dim numbers are intentional — that's how you let factor drive the decomposition instead of tripping the fallback. and it worked — the "full matrix mode" warning disappeared, confirming the real Kronecker path engaged.)

but it crashes every time, consistently at epoch 5, with:

[FAIL] Training failed: The total norm of order 2.0 for gradients from `parameters` is non-finite, so it cannot be clipped.

loss is totally healthy right up to the crash (1.00, 1.09, 1.11...), then a gradient goes non-finite (NaN/inf) at the clip step. i tried the obvious fixes:

• halved the learning rate (--lr 5e-5) — same crash, same epoch 5
• tightened gradient clipping (--max-grad-norm 0.5) — same crash, same epoch 5

so it's not a hyperparameter issue. it looks structural to running the true Kronecker-factored LoKr on the ROCm backprop path — the full-matrix fallback (factor=-1) is numerically stable, the real decomposition produces smaller/more sensitive update matrices and one of them blows up regardless of LR or clip threshold.

bottom line for AMD users: use the default --lokr-factor -1 (full matrix mode). it completes, loads, and generates. avoid forcing --lokr-factor 8 until someone figures out the instability.

Open question for the community

has anyone gotten the true Kronecker-factored LoKr (--lokr-factor 8 or similar) to train stable on AMD ROCm? curious if it's:

• a ROCm fp32/bf16 backward-pass thing,
• something fixable by patching error_if_nonfinite=False in the trainer (haven't tried, didn't want to silently scale by a non-finite norm),
• or just inherent to the decomposition on this backend.

if you're on a different AMD card (RDNA4 9070, integrated, etc) and either config behaves differently, drop your setup below, trying to get all the AMD configs documented in one place like we did on #404.

and here's a track i made with the working config, deep house, all instrumental from the model: https://soundcloud.com/tuckerq/the-nerve/s-FFnVaxRwXVE?si=ba37c5886cc7438d8477c80faa99d47b&utm_source=clipboard&utm_medium=text&utm_campaign=social_sharing#t=58