Interactive Binary Entropy Explorer

Overview

This demonstration lets you explore the binary entropy function $H(p) = -p \log_2 p - (1-p) \log_2(1-p)$ interactively. See how entropy changes with probability, understand why it’s maximized at $p = 0.5$, and visualize the relationship between uncertainty and information.

Demonstrates Concepts

• [[Shannon Entropy]]
• [[Binary Entropy Function]]
• [[Maximum Entropy Principle]]

Interactive Elements

Control	Range	Default	Effect
Probability $p$	0.01 – 0.99	0.5	Sets the bias of the coin
Show tangent	On/Off	On	Displays derivative at current point
Show area	On/Off	Off	Shades area under curve up to $p$
Zoom level	1x – 4x	1x	Magnifies region around current $p$

Demo

How to Use

1. Drag the probability slider to change the coin’s bias. Watch the entropy value update.
2. Observe the symmetry: Notice that $H(0.2) = H(0.8)$. A coin biased toward heads has the same entropy as one equally biased toward tails.
3. Find the maximum: The peak at $p = 0.5$ shows that maximum uncertainty (1 bit) occurs for a fair coin.
4. Enable the tangent line: See how the derivative is positive for $p < 0.5$ and negative for $p > 0.5$.
5. Explore the edges: As $p \to 0$ or $p \to 1$, entropy approaches 0 (certainty).

What to Notice

[!tip] Key Observations

• At $p = 0.5$: Entropy equals exactly 1 bit. This is the “hardest” coin to predict.
• At $p = 0.1$ and $p = 0.9$: Entropy is about 0.47 bits. You need less than half a question on average.
• The curve is concave: Mixing two distributions always increases entropy (Jensen’s inequality).
• Derivative at $p = 0.5$: The tangent is horizontal—this is the maximum.

Mathematical Background

The binary entropy function:

$$H(p) = -p \log_2 p - (1-p) \log_2(1-p)$$

Properties visualized:

• Domain: $p \in (0, 1)$, with $H(0) = H(1) = 0$ by convention
• Range: $H(p) \in [0, 1]$ bits
• Symmetry: $H(p) = H(1-p)$
• Maximum: $H(0.5) = 1$
• Concavity: $H''(p) < 0$ everywhere

The derivative:

$$\frac{dH}{dp} = \log_2 \frac{1-p}{p}$$

This is positive when $p < 0.5$ and negative when $p > 0.5$.

Code

Manipulate[
  Module[{h, dh, tangentLine, pRange, hRange},
    (* Binary entropy function *)
    h[x_] := If[x == 0 || x == 1, 0, -x Log2[x] - (1 - x) Log2[1 - x]];
    dh[x_] := Log2[(1 - x)/x];

    (* Tangent line at current point *)
    tangentLine[x_] := h[p] + dh[p] (x - p);

    (* Zoom handling *)
    pRange = If[zoom > 1, {Max[0.01, p - 0.5/zoom], Min[0.99, p + 0.5/zoom]}, {0.01, 0.99}];
    hRange = If[zoom > 1, {Max[0, h[p] - 0.5/zoom], Min[1.1, h[p] + 0.5/zoom]}, {0, 1.1}];

    Plot[
      {h[x], If[showTangent, tangentLine[x], Nothing]},
      {x, pRange[[1]], pRange[[2]]},
      PlotRange -> {pRange, hRange},
      PlotStyle -> {{Blue, Thickness[0.003]}, {Red, Dashed}},
      AxesLabel -> {"p", "H(p)"},
      PlotLabel -> Style[
        StringForm["H(``) = `` bits",
          NumberForm[p, {2, 2}],
          NumberForm[h[p], {2, 4}]
        ], 14, Bold],
      Filling -> If[showArea, {1 -> {Axis, Opacity[0.2, Blue]}}, None],
      GridLines -> {{0.5}, {1}},
      GridLinesStyle -> Directive[Gray, Dashed],
      Epilog -> {
        PointSize[Large], Red, Point[{p, h[p]}],
        If[showTangent, {Thin, Red, Line[{{pRange[[1]], tangentLine[pRange[[1]]]},
                                          {pRange[[2]], tangentLine[pRange[[2]]]}}]}, {}]
      },
      ImageSize -> 500
    ]
  ],
  {{p, 0.5, "Probability p"}, 0.01, 0.99, Appearance -> "Labeled"},
  {{showTangent, True, "Show tangent"}, {True, False}},
  {{showArea, False, "Show area"}, {True, False}},
  {{zoom, 1, "Zoom"}, {1 -> "1x", 2 -> "2x", 4 -> "4x"}},
  ControlPlacement -> Top,
  TrackedSymbols :> {p, showTangent, showArea, zoom}
]

Extensions

• Add comparison with ternary entropy (3 outcomes)
• Show relationship to mutual information
• Animate through $p$ values to show the full curve being traced
• Add information about what “1 bit” means in practical terms

Related Demonstrations

• [[Rate-Distortion Explorer]]
• [[Channel Capacity Visualizer]]
• [[Entropy of English Text]]