Show HN: An AI eval based on a silly joke from an underrepresented language (kapuskonda.vercel.app)

A new evaluation challenge titled "kapus kondyachi goshta," derived from a popular Marathi joke, has been proposed to assess the cultural understanding of AI models, particularly those from non-Western languages. The challenge stems from the recognition that Marathi is an underrepresented language on the internet, making it a compelling litmus test for evaluating AI's capability to grasp nuances outside Western contexts. The evaluation consists of two parts: testing whether the AI can recognize the absurdity of the joke and understand it lacks a coherent narrative, and assessing its performance in engaging users appropriately during the joke-telling process. This initiative is significant for the AI/ML community as it highlights the necessity for models to demonstrate cultural sensitivity and avoid biased performance that favors Western-centric contexts. By challenging leading AI systems with a joke that emphasizes absurdity and loops, the evaluators aim to expose shortcomings in model training related to non-Western languages. This endeavor not only reinforces the importance of diverse cultural representations in AI but also contributes to the broader discussion on AI's potential to perpetuate biases if not thoroughly tested across varied linguistic landscapes.