Alea iacta non est: rerolling the LLM dice (rolisz.ro)

A new tool called "reroll" has been developed to assess the consistency of responses from large language models (LLMs) by generating multiple answers for the same query and analyzing their similarities. This app generates five distinct replies to a single input and uses Haiku, a summarization tool, to categorize the responses based on their agreement and interpretation. This approach reveals how different LLMs interpret user prompts, emphasizing the variability in results that can arise from both underspecified questions and the inherent non-determinism of these models. The significance of this tool lies in its ability to illuminate the often hidden complexities of LLM behavior. By highlighting the variance in answers, "reroll" fosters a better understanding of how LLMs navigate latent spaces and form conclusions based on user input. This has implications for users seeking reliable information, as it underscores the importance of context in eliciting precise responses. Furthermore, the tool reveals that even well-defined frameworks can lead to interpretational divergences, reminding the AI/ML community of the nuanced nature of model responses. Despite its cost, "reroll" serves as a practical resource for exploring LLM dynamics and enhancing user interaction reliability.