When Models Manipulate Manifolds: The Geometry of a Counting Task (transformer-circuits.pub)

Researchers reverse-engineered how Claude 3.5 Haiku solves a ubiquitous perceptual task in text: fixed-width linebreaking. By analyzing residual-stream activations, attention heads, and an attribution graph, they show the model builds explicit, one-dimensional “count” representations (character-in-line, line width, chars remaining, next-word length) as curved manifolds embedded in a low-dimensional subspace (top 6 PCs capture ≈95% variance). These manifolds are simultaneously interpretable as families of discrete dictionary features (≈10 smoothly tuned features that tile counts like spline knots) and as continuous, high-curvature curves (visualized as helix-like trajectories). Attention QK matrices act geometrically: certain heads rotate one counting manifold to align with another at chosen offsets, producing large inner products when a boundary is near; multiple heads with different offsets produce precise sensing, and near-orthogonal placement of counts vs. next-word length makes the final newline decision linearly separable. The work is significant because it reveals how language models acquire spatial/perceptual capabilities from token streams and exposes a duality between discrete feature attributions and continuous geometric circuits—useful for mechanistic interpretability. Key technical validations include linear probes predicting counts, PCA/manifold visualizations, targeted ablations and interventions that shift perceived counts, and “visual illusions” that hijack attention. Broader implications: many ordinal concepts may be represented as manipulable manifolds, distributed across heads, imposing a complexity tax on purely feature-based interpretation but offering a clearer geometric lens when variables can be parameterized.