Boltzmann Entropy as Shannon Entropy

The Connection

[!abstract] Core Insight [[Boltzmann Entropy]] (physics) and [[Shannon Entropy]] (information theory) are the same mathematical quantity measuring the same conceptual thing: uncertainty about the microstate given the macrostate.

They differ only in:

1. Units ($k_B$ vs. bits)
2. Historical context
3. Typical applications

Thermodynamics Perspective

Context

In statistical mechanics, we have a system with many particles. The macrostate (temperature, pressure, volume) is what we measure. The microstate (exact positions and momenta of all particles) is unknowable.

Formulation

Boltzmann entropy:

$$S = k_B \ln \Omega$$

where:

• $k_B = 1.38 \times 10^{-23}$ J/K (Boltzmann’s constant)
• $\Omega$ = number of microstates compatible with the macrostate

For a probability distribution over microstates:

$$S = -k_B \sum_i p_i \ln p_i$$

Key Properties

1. Extensive: $S(A+B) = S(A) + S(B)$ for independent systems
2. Maximum at equilibrium (uniform distribution over accessible states)
3. Second Law: $\Delta S \geq 0$ for isolated systems

Information Theory Perspective

Context

We have a random variable $X$ representing some uncertain outcome. We want to quantify how much we don’t know before observing it.

Formulation

Shannon entropy:

$$H(X) = -\sum_i p_i \log_2 p_i$$

where $p_i$ is the probability of outcome $i$.

Key Properties

1. Additive: $H(X,Y) = H(X) + H(Y)$ for independent $X, Y$
2. Maximum for uniform distribution
3. Non-negative, zero only for deterministic outcomes

The Bridge

Formal Correspondence

Thermodynamics	Information Theory
Microstate	Outcome / Message
Macrostate	Constraint / What we know
$\Omega$ (multiplicity)	$2^{H}$ (effective number of outcomes)
$k_B$	Conversion factor to physical units
Equilibrium	Maximum entropy distribution
Heat bath	Noisy channel
Temperature	Inverse of Lagrange multiplier

Mathematical Translation

The key equation:

$$S_{\text{Boltzmann}} = k_B \ln 2 \cdot H_{\text{Shannon}}$$

Or equivalently:

$$H = \frac{S}{k_B \ln 2}$$

When physicists use natural logs and information theorists use $\log_2$:

$$S = k_B H_{\text{nats}} = k_B \ln 2 \cdot H_{\text{bits}}$$

Why This Works

Both entropies answer the same question: “How many yes/no questions do I need to specify the exact state?”

• In physics: given macroscopic measurements, how many bits to specify the microstate?
• In information theory: given the probability distribution, how many bits on average to identify the outcome?

The underlying structure is identical: a probability distribution over states, and a desire to quantify the “spread” or “uncertainty” of that distribution.

Implications

For Thermodynamics

• The Second Law is about information: isolated systems evolve toward states we can’t distinguish (maximum ignorance about microstates).
• Entropy increase = losing track of microscopic details.
• Maxwell’s demon is defeated by Landauer’s principle: erasing information costs $k_B T \ln 2$ per bit.

For Information Theory

• There’s a thermodynamic cost to computation (especially erasure).
• Channel capacity has a physical interpretation.
• Compression is fighting the natural tendency toward maximum entropy.

New Insights

• Jaynes’ MaxEnt: Use maximum entropy as an inference principle—it’s not just physics, it’s rational belief formation.
• Landauer’s bound: $k_B T \ln 2 \approx 2.9 \times 10^{-21}$ J at room temperature is the minimum energy to erase one bit.
• Reversible computation: In principle, computation needn’t cost energy; only erasure does.

Historical Development

1. 1865: Clausius introduces entropy as $dS = \delta Q / T$
2. 1877: Boltzmann connects to microstates: $S = k \ln W$
3. 1929: Szilard analyzes Maxwell’s demon information-theoretically
4. 1948: Shannon defines information entropy (reportedly suggested by von Neumann to “call it entropy—no one knows what it means”)
5. 1957: Jaynes unifies via MaxEnt principle
6. 1961: Landauer proves erasure costs energy

The connection was suspected early (von Neumann’s quip) but not rigorously established until Jaynes.

Limitations

• Negative temperatures: Thermodynamic entropy can handle population inversions; Shannon entropy is always positive.
• Continuous distributions: Differential entropy has different properties (can be negative, not invariant under coordinate change).
• Quantum: von Neumann entropy $S = -\text{Tr}(\rho \ln \rho)$ extends both, but has additional subtleties (entanglement entropy).

Sources

• Jaynes, E.T. (1957). “Information Theory and Statistical Mechanics”
• Landauer, R. (1961). “Irreversibility and Heat Generation in the Computing Process”
• Bennett, C. (1982). “The Thermodynamics of Computation—A Review”
• Cover & Thomas, Elements of Information Theory, Chapter 2