Embedding Spaces – Transformer Token Vectors Are Not Points in Space (www.alignmentforum.org)

A recent deep dive into transformer embeddings clarifies a common misconception: transformer token vectors are often thought of as fixed points in a Euclidean space, but this perspective is misleading. Instead, token embeddings are better understood as directions on a high-dimensional hypersphere combined with a magnitude component. This insight stems from the way transformers use cosine similarity in the softmax layer to calculate token probabilities, emphasizing vector direction over Euclidean distance in the embedding space. The post walks through the mechanics of tokenization and embedding, explaining how language models convert tokens into vectors within a limited dimensional space (e.g., 768 dimensions for smaller models). It highlights that embedding and unembedding matrices map token IDs to vectors and back, though with noise due to compressing vast vocabularies into smaller vector spaces. Crucially, layer normalization normalizes hidden states so transformers primarily process vector directions rather than absolute positions. This reframing has significant implications for interpreting how transformers understand and generate language, encouraging researchers to focus on angular relationships in embedding space rather than pointwise distances, refining theoretical and practical approaches in AI/ML model analysis and development.