GenAI Image Editing Showdown (genai-showdown.specr.net)

A multi-model GenAI image-editing showdown put 14 generators through a battery of constrained, quirky prompts — from “two Prussian soldiers playing ring toss” and “a nine-pointed star” to “Alexander the Great riding a hippity‑hop toy” and “five translucent glass cubes stacked in a specific color order.” Contestants included FLUX.1 (dev), Gemini 2.5 Flash, Imagen 4, Midjourney v7, OpenAI 4o and Seedream 4. Success was measured not by aesthetic quality but by strict, objective criteria (exact counts, specific object attributes, action depiction, and even precise text on a chalkboard). Results were highly mixed: Imagen 4 and OpenAI 4o frequently nailed tricky constraints (e.g., nine‑pointed star, Schrödinger’s equation, extended characters), Midjourney v7 surprised with a first‑try star but repeatedly failed compositional prompts (64 attempts on the cube stack), and open models sometimes required workaround pipelines (ComfyUI + img2img) to pass. The test highlights concrete, technical failure modes still plaguing text-to-image models: poor compositional reasoning (ordering, exact counts), sensitivity to aspect ratio (portrait vs 1:1 dramatically affected vertical stacking), inconsistent handling of specialized objects or metaphors (hippity hop, sock puppets), hallucinated or missing textual/math symbols, and content-moderation blocks that interrupt valid creative outputs. Iteration counts — from single attempts to dozens — underscore reliance on heavy re-rolling or human-in-the-loop editing (inpainting) for strict tasks. The takeaway for ML practitioners and deployers: while models have improved, targeted benchmarks reveal a need for better training/data for hard compositional constraints, improved token/text rendering, and integrated editing tools or prompt pipelines when exact adherence is required.