Show HN: Benchmarking Tangible Interface Understanding in Long-Horizon Tasks (huggingface.co)

The SWITCH benchmark has been introduced to evaluate embodied AI systems in their interactions with tangible control interfaces (TCIs) like light switches and appliance panels. Recognizing the crucial role of effective interaction with the real world for autonomous intelligence, SWITCH aims to address gaps in current benchmarks that typically ignore commonsense reasoning, causal predictions, and partial observability. The initial version, SWITCH-Basic, assesses five key abilities, including task-aware visual question answering and state-transition prediction, utilizing 351 tasks across 98 different devices. This benchmark is significant for the AI/ML community as it encourages the development of models that can better handle complex, real-world interactions, which often involve temporally delayed outcomes and diverse physical inputs. Notably, early evaluations revealed that existing large language models (LLMMs) struggled with simple interactions, highlighting an overreliance on textual cues instead of visual evidence. By providing data, code, and structured evaluation frameworks, SWITCH aims to promote reproducibility in research and spur further advancements in developing AI systems capable of nuanced engagement with everyday environments.