What Are Tokens in LLMs? (bearisland.dev)

Recent discussions have shed light on tokens and their crucial role in Large Language Models (LLMs) like GPT-4 and Llama 3. Tokens are the smallest units of input that a model can understand, forming a unique vocabulary tailored to each model during training. This means that while two models can process the same text, they may generate different token sequences based on their distinct vocabularies. For instance, "strawberry" is tokenized differently by GPT-4 and Llama 3, reflecting how each model interprets input without directly seeing the text. The significance of understanding tokens lies in their foundational role in model architecture. The commonly used Byte Pair Encoding (BPE) algorithm creates these vocabularies without requiring complex machine learning techniques. Instead, it combines frequently co-occurring character pairs into new tokens, thereby optimizing the encoding of frequent substrings into single tokens. This process directly influences computational efficiency, relationship to memory, and model performance since vocabulary size (V) impacts the embedding matrix and consequently the parameter budget of the model. Balancing vocabulary size with model dimensions ensures efficiency while maximizing the amount of input text processed, making comprehension of tokenization key for advancing AI/ML technologies.