Entropy Rate — Hidden Markov Model
HMM entropy rate via chain rule H = lim H(Xₙ|X₁…Xₙ₋₁). Convergence of conditional entropy estimates.
HMM Parameters
Hidden states K:
3
Alphabet |A|:
4
Transition mixing ε:
0.20
Emission noise σ:
0.30
Sequence length T:
2000
▶ Estimate
↺ New HMM
True H(hidden):
—
Est. H(obs):
—
H(obs|hidden):
—
Convergence Δ:
—
HMM entropy rate:
H = lim_{n→∞} H(Xₙ|X₁…Xₙ₋₁)
= H(X,Z) − H(Z|X) − H(Z)
Estimated via forward algorithm:
H_n = −log P(xₙ|x₁…xₙ₋₁)
Chain rule: H = avg as n→∞.
Green = running avg H̄ₙ
Orange = true Markov entropy