Tag: information-theory

Blog Post·2024-06-19·18 min read

From Likelihood to Perplexity: One Unified Reference

Probability, likelihood, log-likelihood, NLL, expectation, entropy, cross-entropy, KL divergence, and perplexity are constantly confused for one another. They are not the same thing — but they are all the same thing seen from different angles. This is the definitive deep-dive that names each one precisely and shows exactly how they connect for LLMs.

math probability information-theory perplexity llm-training

Blog Post·2024-06-19·6 min read

Information Theory for LLMs: Mutual Information, Entropy, and What Models Learn

Information theory gives precise answers to questions like: how much does the context tell you about the next token? What information is preserved in a representation? Why does compression and prediction point to the same objective?

math information-theory entropy representation-learning

Blog Post·2024-06-19·5 min read

Probability for LLMs: Distributions, Entropy, and KL Divergence

A language model is a probability distribution over sequences. Training it means pushing that distribution toward the data distribution. The math of how you measure and minimize that gap is what this post covers.

math probability information-theory llm-training