A tiny proof for a tiny LLM (mteoh.com)

A recent exploration into the Tiny LLM course revealed a critical insight regarding the use of sampling APIs in language models. The author proved that using log-probabilities instead of logits in sampling functions, specifically Apple’s mlx.core.random.categorical, yields the same results, which is particularly useful for developers encountering API documentation discrepancies. The proof demonstrated that both greedy decoding and sampling produce equivalent outcomes when either logits or log-probs are used, reinforcing the flexibility of input formats within model implementations. This discovery is significant for the AI/ML community as it enhances understanding of how language models operate internally, especially in regards to sampling methods. By clarifying that logits and log-probs can be interchanged without impacting performance, developers can streamline their processes and avoid confusion when integrating various model components. The author’s journey through Tiny LLM serves as a reminder of the importance of experimentation and deep learning in mastering the workings of language models.