PocketLLM – Run local LLMs from a USB stick (https://pocketllm-site.vercel.app/) (github.com)

PocketLLM has introduced a revolutionary USB toolkit that allows users to run local large language models (LLMs) without installation or digital traces on host machines. By simply plugging the USB drive into a Mac or Linux system and executing a single command, users can access a fully functioning local AI environment, complete with model weights, a chat interface, and conversation history. This innovative approach eliminates the need for cloud services and extensive hard drive usage, making AI more portable and private. The significance of PocketLLM lies in its zero-installation feature and ability to maintain user privacy; after unplugging, no data remains on the host system. Inference speeds from USB are comparable to SSDs, achieving up to 54 tokens per second post-initial loading. While the first model load takes longer from USB (due to I/O delays), subsequent interactions are swift, highlighting the balance between portability and performance. Additionally, it supports a variety of models compatible with the Ollama runtime, enabling users to easily transport and utilize their AI setups across different machines without sacrificing functionality.