Information Bottleneck — Compression vs. Relevance

The IB method finds optimal compressed representation T of X that maximally preserves information about Y. The tradeoff is controlled by β.

Lagrange multiplier β 2.0

Joint distribution

Clusters |T| 4

IB objective:
min I(T;X) − β·I(T;Y)

IB equations (self-consistent):
p(t|x) ∝ p(t)·exp(−β·D_KL[p(y|x)‖p(y|t)])

β=0: maximum compression, T independent of Y
β→∞: T=X, no compression

Successive refinement: as β increases, T undergoes phase transitions adding structure (Tishby, Pereira, Bialek 1999).

I(T;X): — bits
I(T;Y): — bits
I(X;Y): — bits
Efficiency: —

IB Plane: I(T;X) vs I(T;Y) — the IB bound curve

Joint distribution p(x,y)

Cluster assignment p(t|x) at current β