Divergence

Definition

In statistics and information theory, divergence is a measure of how one probability distribution differs from another. The Kullback–Leibler (KL) divergence is the most common variant, quantifying the information lost when one distribution is used to approximate another. Unlike a true distance metric, KL divergence is not symmetric.

Why It Matters

Divergence provides a principled way to assess how much two probability distributions disagree, which is central to model selection, Bayesian updating, and machine learning. In variational inference, minimising KL divergence between an approximate posterior and the true posterior is the core objective. In finance, divergence measures help detect regime changes in asset return distributions. Because KL divergence is asymmetric, practitioners must be careful about which direction they measure — the divergence from P to Q is generally not the same as from Q to P.

Example

A risk analyst models daily stock returns using a Normal distribution, but the true distribution has heavier tails (a Student's t with low degrees of freedom). The KL divergence from the true distribution to the Normal approximation quantifies how much information is lost by using the simpler model. A large divergence signals that the Normal model is a poor approximation and could underestimate tail risk.

Related Terms

• Maximum Entropy
• Maximum Likelihood Estimation (MLE)
• Bayesian Inference

Software Notes

• SPSS: No built-in command; compute manually using log-likelihoods or use R integration via SPSS Statistics
• R: KLdiv() from the diveMove package or KL() from the philentropy package for various divergence measures
• Stata: No built-in command; compute manually: gen kl = p * log(p / q) and then summarize kl

Contact Us for Support → /contact/