Playing a hardware synth through Claude Code (vaclav.synacek.com)

A hobbyist connected an LLM-driven workflow (using Claude Code) to physical Roland hardware synths to have the AI generate and play music via MIDI, producing short videos of the setup in action. The project involved a Python-based MIDI sequencer where the model writes note/beat sequences while a micro-app scans a directory of synth “patch” files, offers a menu to schedule the next patch, and performs live patch replacement. Iterations revealed practical real‑world issues — flaky recordings, timing glitches from naive process restarts, overlapping patches from a Unix‑signals approach, and the need for visual timing aids — so the author added on-screen visualizations that both look cool and help the human operator time knob tweaks and transitions. Technically notable is the intentional human–AI teaming: the Roland T-8 lacks MIDI CC control, so the LLM can drive notes and rhythm but cannot change synthesis parameters, forcing a collaborative workflow where humans physically adjust knobs while the AI sequences. The work is essentially an LLM-driven MIDI sequencer connected to hardware, but it highlights important implications for the AI/ML community: real‑time control and orchestration of physical devices requires robust scheduling, smooth transition logic, tempo-aware visual feedback, and human-in-the-loop constraints. It’s a practical, replicable demo of LLMs augmenting live performance and points to broader opportunities and challenges for integrating AI with physical systems.