Tag: fine-tuning

Blog Post·2026-06-20·10 min read

Low-Rank Approximation: From SVD to LoRA and Beyond

Eckart–Young says the truncated SVD is the optimal low-rank approximation. This post turns that theorem into engineering — factorized layers, truncated-SVD compression, LoRA and its descendants QLoRA, DoRA, and GaLore, and nuclear-norm minimization for matrix completion.

low-rank lora qlora fine-tuning model-compression

Blog Post·2024-06-19·8 min read

SVD and Low-Rank Approximation: The Math Behind LoRA

LoRA works because weight matrices are empirically low-rank. SVD explains why, and the math of low-rank approximation explains how LoRA exploits it to fine-tune billion-parameter models on a laptop.

math svd lora fine-tuning linear-algebra