Blog Post··5 min read
What Actually Happens to Padding Tokens During LLM Pretraining
Padding wastes GPU compute. Sequence packing eliminates it — but introduces cross-document attention contamination unless you explicitly mask it. Here's what the attention mask actually looks like.