Introduction to Abstract Entropy (www.lesswrong.com)

This post introduces a synthetic, domain‑agnostic view of entropy: the number of bits needed to uniquely distinguish a system’s state. Rather than tying entropy to thermodynamics or any particular physics, the author defines it abstractly via labeling strategies — most usefully by assigning finite binary strings to microstates (shorter strings for “simpler” or more important states). Key technical points: there are 2^k binary strings of length k, so low‑entropy (short‑string) states are intrinsically rare; average entropy for N states is tightly lower‑bounded and grows ≈ log2(N) (with small, slowly growing correction terms). A concrete example: a Rubik’s Cube has ≈ 4.3×10^19 states, so a random state needs about 65 bits to specify. An equivalent operational view is yes/no partitioning (Guess Who?), where optimal questions halve the remaining possibilities — the usual information‑theoretic/decision‑tree perspective. For the AI/ML community this matters because it cleanly separates entropy as a combinatorial/representational concept from notions of “order” and dynamical second‑law behavior. It clarifies why compression, labeling choices, and model priors matter (labeling is partly subjective, but efficient labelings yield objective lower bounds), links to Kolmogorov complexity when “order” is considered, and shows how entropy informs optimal querying, coding, and complexity accounting in learning and optimization.