Chapter 3 — Principle of Maximum Entropy
Section outline
-

The Principle of Maximum Entropy is a method for reasoning under uncertainty that says: when choosing among probability distributions that are consistent with what you know, you should pick the one that is as spread out (least informative) as possible — the one with maximum entropy.
Core idea:
-
You have some constraints (e.g., known averages or facts).
-
Many distributions could satisfy them.
-
Don't assume anything you don't actually know.
-
Choose the distribution that is maximally noncommittal beyond the constraints.
Why entropy? Entropy measures uncertainty or "evenness." Maximizing it means avoiding unjustified bias — you stay as open-minded as the constraints allow.
Simple example: If all you know is that a die has 6 faces, the maximum-entropy distribution is uniform: each face has probability 1/6. If you also know the average roll is 4.5, max entropy gives a different (biased) distribution — but only as biased as that constraint requires.
Origins & uses:
-
Formalized by E.T. Jaynes (1957) in statistical mechanics and Bayesian inference.
-
Used in statistical physics, machine learning (e.g., maximum entropy models, logistic regression), natural language processing, and information theory.
-