Conditional Entropy & Mutual Information

Chain Rule · Venn Diagram · Joint Distribution · Information Decomposition

P(X=0) — marginal X 0.50

P(Y=0|X=0) — channel noise 0.80

P(Y=0|X=1) — cross-over prob 0.20

Variable Z correlation 0.50

H(X,Y) = H(X) + H(Y|X) = H(Y) + H(X|Y)

I(X;Y) = H(X) − H(X|Y) = H(Y) − H(Y|X)

I(X;Y) = H(X) + H(Y) − H(X,Y)

Chain: H(X,Y,Z) = H(X) + H(Y|X) + H(Z|X,Y)