Can VLMs recognize famous videos from just their colors? (loganbolton.github.io)

Logan Bolton has created an intriguing benchmark dubbed "VideoColorBench," which tests the capability of vision-language models (VLMs) in identifying popular movies and YouTube videos solely based on averaged color representations of their frames. This challenge involves presenting models with a simplified color barcode for each video, alongside ten potential answers, to ensure that the task isn't trivial. Notably, the benchmark showcases a mix of both notable commercial models and open-source varieties, revealing their performance when asked to guess the movies without additional web searches or context. The significance of this benchmark lies in its exploration of how well VLMs perform in a highly niche domain, emphasizing their recognition and reasoning abilities under daunting conditions. Initial results showed that models like Opus 5.5 and GPT-6.1 Sol managed around 25% accuracy, while open-source models trailed at approximately 18%. Interestingly, using auxiliary tools for web searches provided marginal improvements, with models such as GLM-5.3 Flash achieving the highest accuracy by analyzing YouTube thumbnails. This experiment highlights the ongoing advancements and limitations of AI models in visual tasks, indicating that targeted training and context utilization remain pivotal for better performance in complex recognition challenges.