Shotwell-Bench: An annotation benchmark that accepts valid alternatives (shotwell.ai)

Shotwell-Bench has been introduced as a new annotation benchmark designed to address the rigid nature of existing frameworks that penalize valid variations in model outputs. Traditional benchmarks like WGO-Bench are criticized for their inflexibility, which often leads to inaccurate assessments of model performance in video annotation tasks. Shotwell-Bench allows for multiple acceptable annotations and segmentations of egocentric videos, better capturing the nuances of action boundaries and descriptions. This innovation is crucial as it fosters improved evaluation of Vision-Language Models (VLMs) and provides a clearer understanding of their effectiveness in generating diverse yet valid outputs. Key innovations of Shotwell-Bench include a more comprehensive structure for scoring annotations, allowing a range of timestamps for segment boundaries, and permitting flexible descriptions that maintain essential details while allowing for interpretative variation. By emphasizing accuracy, level of detail, and the cost of producing annotations, Shotwell-Bench not only enhances evaluation consistency but also sheds light on the limitations of weaker models. As a testament to its efficacy, GPT-6 Astra emerged as the top performer on this benchmark, indicating its capability to navigate the complexities introduced by Shotwell-Bench while delivering comprehensive model assessments.