Evaluating in Silico Creativity: An Expert Review of AI Chess Compositions (arxiv.org)

Researchers tested generative AI’s capacity for “in silico” creativity by building a system to produce chess compositions that prioritize aesthetic appeal, novelty and counter‑intuitive or unique solutions, then subjecting those outputs to human expert judgment. Instead of relying on automated metrics, the team compiled a curated booklet of AI‑generated puzzles and asked three leading authorities on chess composition and aesthetics—IM Amatzia Avni and GMs Jonathan Levitt and Matthew Sadler—to choose favorites and explain what made them appealing, focusing on creativity, challenge and design. The work is significant because it evaluates creative quality in a constrained, well‑formalized domain using domain experts rather than purely statistical scores, offering a clearer bridge between generative models and human notions of artistry. Technically, the system is tuned to optimize for nontrivial solution paths and aesthetic criteria (details are in the full paper and accompanying code/demos), and the evaluation emphasizes qualitative, expert‑driven judgement as a complement to automated benchmarks. Implications include better frameworks for assessing computational creativity, potential tools for human–AI co‑creation in composition and education, and new avenues for building benchmarks that measure novelty and aesthetic value rather than only correctness or fluency.