Reverse Engineering and Tracing internal thoughts of LLM (medium.com)

Security researcher Harish SG ran a suite of mechanistic-interpretability experiments on small LLMs (Meta Llama 3 1B and a tiny custom model) to trace exactly how a model predicts tokens, where “knowledge” lives, and which neurons/dimensions drive outputs. Using the prompt “The capital of France is”, he tracked token-by-token logits and hidden-state statistics: “Paris” was selected at step 1 with 39.2% probability, while grammatical tokens like “is” were chosen with higher confidence (63%). A surprising find: the special <|begin_of_text|> token exerted the largest single influence (~71% contribution) on predictions, yet ablation experiments showed that removing content words like “capital” and “France” each drops the probability of “Paris” by ~39%, revealing attention-weight magnitudes do not equal causal importance. Layer- and weight-level analyses show the answer only “crystallizes” in the final transformer layer (layer 15) where the logit for “Paris” jumps (final logit ≈ 12.56), while earlier layers gradually build supporting evidence. Output-projection weights concentrate importance on a handful of vector dimensions (top dims listed), indicating knowledge is distributed across dimensions and neurons rather than stored as a single token map. Implications: these tracing tools (token traces, ablations, layer emergence, weight/dimension inspection, token→neuron mapping) provide practical ways to debug, audit and align models, but results may vary with model scale and architecture; the author notes he’s not an ML professional and these are personal findings.