Rényi Entropy — Generalized Information Measures

Rényi Entropy H_α

The Rényi entropy of order α of distribution p is: H_α(p) = (1/(1−α)) log(Σ p_i^α). Special cases: H₀ = log(support size), H₁ = Shannon entropy (−Σ p log p, the limit α→1), H₂ = collision entropy = −log(Σ p_i²), H_∞ = min-entropy = −log(max p_i).

Rényi entropy is monotone decreasing in α: H_α ≥ H_β for α ≤ β. H₂ governs quantum entanglement via the purity tr(ρ²). H_∞ (min-entropy) controls extractable randomness in cryptography. The right panel shows H_α vs α — a signature curve that reveals the distribution's heterogeneity.