How big are our embeddings now and why? (vickiboykis.com)

Embedding dimensionality in AI has grown substantially from the early norm of 200-300 dimensions, driven by advances in model architectures, training data scale, and hardware capabilities. Early embedding methods like Word2Vec used around 300 dimensions, balancing expressiveness and computational efficiency. However, the introduction of Transformer-based models such as BERT in 2018 raised embedding sizes to 768 dimensions to enable parallel processing across multiple attention heads, improving representation quality but increasing computational costs. This size became a de facto standard, adopted by many following models for both text and multimodal embeddings. The landscape shifted further with large language models like GPT-3, pushing embeddings to 1536 dimensions and beyond, powered by massive datasets and combined training optimizations. Open-source platforms like HuggingFace and benchmarking resources like MTEB have standardized access to various embedding sizes, ranging now from 768 to 4096 dimensions or more, reflecting greater architectural complexity and richer latent spaces. As embeddings became commoditized via APIs, engineering trade-offs around inference latency, storage, and retrieval effectiveness have become central to embedding design. Recent innovations, such as OpenAI’s matryoshka representation learning, focus on hierarchical embeddings that capture essential semantic information in lower dimensions while permitting incremental refinement, addressing efficiency without sacrificing performance. This ongoing evolution highlights the balance between increasing embedding dimensionality to capture nuanced knowledge and the practical constraints of hardware and application requirements. For the AI/ML community, understanding these trends is critical for selecting or designing embeddings that optimally support downstream tasks like semantic search, recommendation, and retrieval-augmented generation in an era where embeddings are both fundamental and broadly accessible.