Jankmarking: Janky Benchmarking (www.williamangel.net)

A new benchmarking initiative dubbed "Jankmarking" has emerged in the AI community, focusing on assessing local large language model (LLM) performance, notably using the Ollama model on an M5 Pro. The creator is seeking a rapid evaluation of performance versus processing speed without incurring the lengthy wait times associated with cloud solutions. This informal approach involves a series of unconventional benchmark tasks, including crafting haikus, performing basic arithmetic reasoning, and implementing code tasks, albeit with tongue-in-cheek acknowledgment of their “janky” nature. While the results may not adhere to traditional standards, the findings indicate that better models tend to cluster on performance scores, providing some directional insights. However, the creator warns that for critical applications, relying solely on these "jankmarks" can lead to misleading conclusions. This highlights a common challenge in the AI/ML realm—balancing speed and accuracy when evaluating model capabilities in local environments. As the community seeks to improve assessment methods, "Jankmarking" serves as a reminder of the importance of rigor in benchmarking practices.