Tiny Recursive Model (TRM) vs. Hierarchical Reasoning Model (HRM) (gonzoml.substack.com)

Tiny Recursive Model (TRM), a new paper by Alexia Jolicoeur-Martineau, revisits and dramatically simplifies the Hierarchical Reasoning Model (HRM). TRM replaces HRM’s two-network hierarchical recursion with one tiny transformer encoder (2 layers after ablation) that iteratively refines a latent z and a solution embedding y passed alongside the input x (all summed at the block input). With 5–19M parameters (vs ~27M for HRM), TRM uses deep recursion and deep supervision instead of fixed‑point theory or separate H/L modules, and reports stronger performance on tasks like Sudoku‑Extreme (87.4% vs HRM’s ~82.4%). The authors open-source the code and structure their paper as a systematic dissection of HRM’s components and assumptions. Key technical points: a single transformer block ([self‑attention, norm, MLP, norm]) is run in an inner latent_recursion loop (n=6 updates of z, then one y update) and an outer deep_recursion loop (T=3, with first T−1 runs detached for efficiency, gradients through the last n+1=7 calls). Up to Nsup=16 supervised steps yield an effective depth ≈42 per step (≈672 total), letting a tiny network match or exceed very deep models. Ablations show two features (y: current answer embedding; z: chain‑of‑thought latent) work best; adding layers caused overfitting, while EMA (0.999), a simplified ACT halting, and an MLP‑Mixer variant (helpful for short contexts) improved results. TRM underlines that careful algorithmic unrolling and supervision can outperform naive scaling—but it’s a different model class than large autoregressive LLMs, so comparisons should be made cautiously.