I built NOOX, an open-source ESP32-S3 hardware assistant for desktop automation (github.com)

NOOX is an innovative open-source project that combines external hardware, desktop automation, and large language models (LLMs) to create a versatile automation assistant using the ESP32-S3 microcontroller. The device connects to any host via USB HID and CDC, allowing seamless cross-platform communication while executing shell commands and retrieving results through a web-based control console. Key features include its ability to autonomously perform multi-step planning based on context and command output, making it a powerful tool for developers looking to enhance their desktop automation capabilities. This development is significant for the AI/ML community as it highlights the potential of integrating LLMs with hardware for real-world applications, such as system maintenance and automation tasks. With support for OpenAI-compatible APIs and a customizable toolset for executing commands and controlling GPIO operations, NOOX provides users with advanced automation capabilities while emphasizing ease of use and portability. However, it is essential to exercise caution due to its ability to perform potentially harmful operations, underscoring the need for secure implementation in controlled environments.