Transfer learning, decoded_

LoRA at Scale: Why 0.1% of Parameters Do 80% of the Work

Last week, three papers dropped that collectively reframe how we think about parameter-efficient fine-tuning. The short version: rank matters more than we thought, and the right initialization can cut your training time in half.

§1 — The Rank Selection Problem

Choosing LoRA rank is still mostly vibes. r=8 is the default because someone wrote it in the original paper. But new ablations from the Mistral team show rank-4 often matches rank-64 on domain adaptation — if you initialize from the top singular vectors of the target-domain gradient.

# Initialize LoRA from SVD of domain gradient
U, S, Vh = torch.linalg.svd(domain_grad, full_matrices=False)
lora_A = U[:, :rank] * S[:rank].sqrt()
lora_B = Vh[:rank] * S[:rank].sqrt().unsqueeze(-1)

↑ This init cuts convergence from ~800 steps to ~200 on MedQA. The gradient signal is already pointing at what the base model needs to unlearn.

§2 — Vision Transformers on Niche Datasets

Satellite imagery fine-tuning remains a pain point. The spectral distribution shift between ImageNet and multispectral data breaks the first few attention layers. The fix is surprisingly simple: freeze the patch embedding, replace the positional encoding, and let the rest adapt.

§3 — This Week's Reading List