Raspberry Pi 5 gets LLM smarts with AI HAT+ 2 (www.theregister.com)

Raspberry Pi has unveiled the AI HAT+ 2, a new hardware accessory for the Raspberry Pi 5, equipped with 8 GB of onboard RAM and the Hailo-10H neural network accelerator. This device delivers an impressive 40 TOPS of inference performance, primarily aimed at enhancing local AI computing capabilities, particularly for applications involving large language models (LLMs) and vision language models (VLMs). While it boasts superior specifications compared to its predecessor, offering improved support for generative AI tasks, developers will need to consider its relatively modest memory capacity in an era where AI applications are increasingly memory-intensive. The AI HAT+ 2 plugs directly into the Pi's GPIO connector and utilizes the PCIe interface, with an easy setup via the Raspberry Pi OS. Although it operates efficiently for LLM tasks, Raspberry Pi acknowledges that larger models used by cloud giants far exceed the device's capabilities, suggesting it is best suited for localized processing needs where cloud reliance isn't preferred. While computer vision tasks can be handled by the earlier model, the AI HAT+ 2 is aimed at those needing lightweight deployments of LLMs without overwhelming the base unit's memory.